ttarler

One Developer, 12 Workflows, and a Production ML System

A Sunday Night That Explains Everything

It's Sunday at 8 PM ET. The weekly calibration fires: 6 prescreening models retrain on SageMaker, isotonic regression recalibrates confidence scores, and Monte Carlo parameter tuning runs 500 Latin Hypercube samples against the last 28 days of trades. By 10 PM, a new parameter set clears all 5 guardrail gates and writes to the database.

Monday at 6:45 AM, the automated health check runs. It flags a P2: one prescreening endpoint is returning stale predictions — the calibration retraining job succeeded, but the endpoint didn't pick up the new model artifact. A GitHub Issue is created with the ml-pipeline and P2-medium labels, filed under the Platform Maturity milestone. I open the issue, read the diagnostics the health check already gathered, push a one-line fix, and watch CI deploy it via blue-green before market open at 9:30.

One person. No pager. No Slack channel. No standup.

ttarler

Documentation That Deploys: How We Turned Post-Mortems Into Production Safety Nets

Between January 9 and January 30, 2026, our AI-assisted trading system had seven significant incidents. The same root patterns — silent fallbacks, unverified deployments, training/inference mismatches — kept recurring despite post-mortems and a growing LESSONS_LEARNED.md. The lessons existed. The enforcement mechanism didn't.

This post is about the documentation system that emerged from those failures. Not documentation in the wiki sense — a self-reinforcing system where every incident writes the rules that prevent the next one, and those rules are embedded in the tools that AI coding agents read at the start of every session.

Post 5 ended with a tease: "I'll go deeper on the documentation system in the next post, including why we built it, how it prevents the kind of knowledge loss that contributed to our $78K incident, and how it works with AI coding agents." Here's the full story.

The Problem: Lessons That Don't S

ttarler

The Ensemble That Actually Trades: How PPO, SAC, and TD3 Became One Decisioning System

The Problem With One Model

I spent weeks training a single PPO agent to make portfolio decisions. It worked well in calm markets. Then VIX spiked from 14 to 22 and the model froze — its learned policy didn't generalize to a volatility regime it hadn't trained in.

This isn't a novel observation. Any quantitative researcher who's deployed a single model to trade across regimes has hit this wall. The issue isn't the algorithm. It's that markets have structural breaks — volatility regimes, sector rotations, macro shocks — and no single model captures all of them.

ttarler

I Built a Full-Stack AI Trading App with LLMs: Here's What I Learned

From Notebooks to Production with AI Pair Programmers

As a professional data scientist, I've spent years building ML models in Jupyter notebooks. But I wanted to go beyond research. I wanted to deploy a full-stack, production-grade AI trading system. The catch? I needed to supplement my data science expertise with full-stack development, infrastructure, and operations knowledge I didn't have.

Enter AI coding agents.

Over the past few months, I used LLMs (primarily Claude via Cursor) to build an automated trading system from scratch: FastAPI backend, Celery workers, Next.js frontend, PostgreSQL database, AWS infrastructure (ECS, SageMaker, EventBridge), and a complete ML pipeline for training and deploying models.

ttarler

Proof It Works: A Week of Paper Trading Performance (Jan 20-24, 2026)

After All the Lessons, Does It Actually Work?

Seven posts on architecture, context management, testing, deployment, ML failures, and operational assumptions.

But here's the real question: Does the system actually work?

In this final post, I'll show you a week of paper trading performance—the proof that all those lessons, fixes, and patterns resulted in a functioning trading system.

ttarler

"Scheduled ≠ Working" and Other Expensive Assumptions

The Assumption That Cost 7 Days

Celery beat logs:

[2025-11-18 09:32:00] Sending task: check_stop_loss_take_profit
[2025-11-18 09:34:00] Sending task: check_stop_loss_take_profit
[2025-11-18 09:36:00] Sending task: check_stop_loss_take_profit

ttarler

The ML Model Was 'Live' for 6 Days—It Never Made a Single Decision

"Deployed and Working"

Jan 8: "RL model deployed to SageMaker endpoint. Trading with trained model."

Jan 9: "RL model is active and making portfolio decisions."

Jan 10-13: Portfolio trades every day. System reports "working."

ttarler

CI/CD and Deployment When AI Writes the Code

The Illusion: "Committed = Running"

I committed a critical fix at 5:51 PM. Pushed to GitHub. CI passed. Green checkmark.

At 6:15 PM, the bug was still happening.

Why? The Docker containers were still running the old code.

ttarler

Training an RL Portfolio Agent: 47 Commits of "Why Won't You Learn to Sell?"

The Problem Nobody Warns You About

Every RL tutorial on the internet ends the same way: train PPO on CartPole, watch the reward curve go up, celebrate. Maybe they graduate to Atari and show you a DQN playing Breakout.

None of them prepare you for what happens when your action space has 25 continuous dimensions, your reward signal is delayed by days, and your agent discovers that the optimal policy is to buy everything and never sell.

ttarler

Building the ML Pipeline: From LSTM Experiments to Production RL

The ML System That Actually Trades

Go read any ML trading tutorial online. It's always the same: download some OHLCV data, train a model, show a backtest chart that goes up and to the right, collect Medium claps.

Nobody talks about what happens when you try to run that model every single day. When you need to retrain monthly without breaking the thing that's already trading. When the feature your model was trained on doesn't exist at inference time. When your processing job takes 8 hours and your data is stale by the time training starts.