"The best way to learn something is to build it from scratch." — Andrej Karpathy
This roadmap follows the Karpathy philosophy: understand every layer by implementing it yourself before reaching for abstractions. No framework worship. No magic. Just you, an LLM API, and progressively harder problems.
8 phases. ~16-20 weeks. Each phase builds on the last.
Every phase follows the same loop:
- Read the one essential reference (not five — one)
- Build something from scratch (no frameworks)
- Then look at how frameworks solved it (now you understand why they exist)
- Evaluate what you built (evals are not a phase — they're a habit)
You'll end up with a portfolio of projects that demonstrate real understanding, not tutorial copypasta.
You need:
- Python (comfortable with async, decorators, type hints)
- Basic understanding of what LLMs are (watch Karpathy's "Deep Dive into LLMs" first if not)
- An API key (Anthropic or OpenAI — ideally both)
- A code editor you're fast in
- ~$50-100 for API costs over the full roadmap
Watch first:
- Andrej Karpathy — "Deep Dive into LLMs like ChatGPT" (3.5hrs, 2025) — covers tokenization, training, RLHF, prompting, tool use, and reasoning. This single video replaces 10 blog posts.
- Andrej Karpathy — "Let's build GPT: from scratch, in code" — not strictly necessary for AI engineering, but gives you intuition for what's happening inside the model. Builds a small transformer from scratch.
Before agents, before tools, before everything — you need to be fluent with the raw API. Most people skip this and end up debugging framework abstractions they don't understand.
Build a CLI chat application using only the raw HTTP API (not even the SDK). Then rebuild it with the SDK.
You → HTTP POST to /v1/messages → Claude responds → You manage conversation history → Repeat
What you'll learn:
- Message format (system, user, assistant roles)
- Conversation state management (you maintain the array of messages)
- Streaming vs non-streaming responses
- Token counting and context window management
- Temperature, top_p, and how they actually affect output
- The difference between "the model remembers" (it doesn't) and "you send history" (you do)
- Raw HTTP chat loop (requests/httpx, no SDK)
- Rebuild with Anthropic Python SDK
- Rebuild with OpenAI Python SDK
- Add streaming support
- Add conversation history management with a token budget
- Add system prompt injection and see how it changes behavior
- Try the same prompts on Claude and GPT — notice the differences
- Anthropic API Docs — Messages — the actual spec, read every field
- Anthropic Prompt Engineering Guide — how to talk to these models effectively
- OpenAI API Docs — Chat Completions — same concepts, different API shape
- Anthropic Cookbook — practical code examples
- OpenAI Cookbook — same, for OpenAI
An agent is just: LLM + Loop + Tools. That's it. The model decides what to do, you execute it, feed the result back, and loop until done. Read this one blog post before writing a line of code:
Anthropic — "Building Effective Agents" (Dec 2024) The single best resource on agent architecture. Key insight: simple, composable patterns beat complex frameworks. Most successful production agents are just augmented LLMs with good tool definitions, not multi-agent orchestration systems.
Your agent follows the ReAct (Reasoning + Acting) loop:
while not done:
response = llm(messages) # Think
if response has tool_call:
result = execute(tool_call) # Act
messages.append(result) # Observe
else:
done = True # Final answer
That's it. Everything else is optimization on this loop.
Build a coding agent that can:
- Read files from your filesystem
- Write/edit files
- Run shell commands (with a safety sandbox)
- Search through code (grep/find)
No LangChain. No CrewAI. No frameworks. Raw API + your code.
The tools are just function definitions you pass to the API:
tools = [
{
"name": "read_file",
"description": "Read a file from the filesystem",
"input_schema": {
"type": "object",
"properties": {
"path": {"type": "string", "description": "Path to the file"}
},
"required": ["path"]
}
},
# ... more tools
]- Define 4-5 tools (read_file, write_file, run_command, search_code, list_directory)
- Implement the ReAct loop — LLM calls tools, you execute, feed results back
- Add a system prompt that instructs the agent on how to approach coding tasks
- Add safety: sandbox shell commands, prevent writes outside project directory
- Test it: "Add error handling to this function" or "Write a test for this module"
- Add conversation history so it can build on previous steps
- Add a simple retry mechanism for when tool execution fails
- Tool/function calling is just structured output — the model outputs JSON, you execute it
- The system prompt is your agent's "personality" and "methodology"
- Error handling matters enormously — agents fail on tool errors, not reasoning
- The model is stateless — your loop is the state machine
- ReAct Paper — the original paper (read the first 5 pages, skip the benchmarks)
- Anthropic Tool Use Docs — the implementation spec
- OpenAI Function Calling Guide — same pattern, different API
- Lilian Weng — "LLM Powered Autonomous Agents" — the canonical overview (Agent = LLM + Memory + Planning + Tool Use)
Now that you've built it raw, look at how others solved the same problems:
- Anthropic's Agent Cookbook — patterns that match what you just built
- OpenAI Agents SDK — see how they structured it
- Pydantic AI — type-safe agent framework, clean design
Raw LLM output is a string. Production systems need structured data — JSON, typed objects, enums, validated fields. This is the bridge between "cool demo" and "shippable product."
Build a system that:
- Takes unstructured text (emails, docs, logs)
- Extracts structured data into Pydantic models
- Validates and retries on failure
- Handles edge cases gracefully
class BugReport(BaseModel):
severity: Literal["critical", "high", "medium", "low"]
component: str
reproduction_steps: list[str]
expected_behavior: str
actual_behavior: str
# Your extraction function should:
# 1. Send text + schema to the LLM
# 2. Parse the response into a Pydantic model
# 3. If validation fails, send the error back and retry
# 4. Return typed, validated data- Build a raw extraction function (prompt → JSON → Pydantic model)
- Add retry logic with validation error feedback
- Handle partial/malformed outputs
- Try Anthropic's native tool use for structured output (tool definitions as schemas)
- Try OpenAI's structured outputs mode (response_format with JSON schema)
- Build a classification system (categorize support tickets, etc.)
- Build a chain: extract → validate → transform → store
- OpenAI Structured Outputs Guide
- Instructor library — after you build it raw, see how Jason Liu solved it
- Outlines — constrained generation for local models (grammar-based approach)
- Instructor — the gold standard for structured extraction
- Marvin — lightweight AI functions with type safety
- Mirascope — decorator-based typed LLM toolkit
LLMs know what they were trained on. RAG (Retrieval Augmented Generation) lets them work with your data — codebases, docs, databases, anything. This is the #1 most deployed LLM pattern in production.
The pipeline is: Query → Retrieve relevant chunks → Stuff into context → Generate answer
Build a RAG system for your own codebase:
- Chunking: Split code files into meaningful chunks (by function/class, not arbitrary line counts)
- Embedding: Convert chunks to vectors using an embedding model
- Indexing: Store in a vector database
- Retrieval: Given a question, find the most relevant chunks
- Generation: Send question + retrieved chunks to the LLM
Week 5 — Naive RAG (build it all from scratch):
- Write a chunking strategy for code (AST-based or delimiter-based)
- Generate embeddings using OpenAI's
text-embedding-3-smallor Anthropic's Voyage - Store vectors in ChromaDB (simplest local vector DB)
- Implement cosine similarity search
- Build the full query pipeline: embed question → search → rerank → generate
- Test with real questions about your codebase
Week 6 — Make it actually good:
- Add hybrid search (keyword BM25 + vector similarity)
- Implement reranking (Cohere Rerank or a cross-encoder)
- Add metadata filtering (filter by file type, directory, recency)
- Experiment with chunk sizes and overlap
- Add citation support — make the LLM reference which chunks it used
- Measure retrieval quality (are you getting the right chunks?)
Week 7 — Agentic RAG:
- Let the agent decide when to search (not every query needs RAG)
- Let the agent reformulate queries if initial retrieval is poor
- Add multi-step retrieval (search → read → search again with more context)
- Integrate RAG as a tool in your Phase 1 coding agent
- Anthropic's Contextual Retrieval — their approach to making RAG actually work
- OpenAI Embeddings Guide
- Pinecone Learning Center — excellent RAG tutorials regardless of which vector DB you use
- MTEB Leaderboard — compare embedding models
- ChromaDB (
pip install chromadb) — start here, local, simple, great for learning - pgvector — if you already use Postgres, just add vector columns
- Pinecone — managed, scales to production, good free tier
- Weaviate — open source, built-in vectorization modules
- LlamaIndex — the most comprehensive RAG framework
- LangChain Retrieval — retrieval chain patterns
Your Phase 1 agent had hardcoded tools. Real agents need a pluggable tool ecosystem — connect to APIs, databases, web browsers, file systems, and external services. This is where MCP (Model Context Protocol) comes in.
MCP is Anthropic's open standard for connecting AI models to tools and data sources. Think of it as USB-C for AI tools — a universal protocol so any model can use any tool.
Week 8 — Custom Tool Server:
- Build 3-4 tool integrations from scratch (GitHub API, database queries, web scraping, calendar)
- Create a tool registry — agents discover and select tools dynamically
- Add tool documentation that helps the LLM understand when and how to use each tool
- Handle auth (API keys, OAuth tokens) cleanly
- Add rate limiting and error handling for external APIs
Week 9 — Build an MCP Server:
- Read the MCP spec and understand the protocol (resources, tools, prompts)
- Build your own MCP server using the Python SDK
- Expose your codebase tools (from Phase 1) as MCP tools
- Connect your MCP server to Claude Desktop or Claude Code
- Build a second MCP server for a different domain (database, web search, etc.)
# An MCP server is surprisingly simple:
from mcp.server import Server
from mcp.types import Tool
server = Server("my-tools")
@server.tool()
async def search_codebase(query: str) -> str:
"""Search the codebase for relevant code."""
# Your implementation here
...- MCP Official Site — the spec and getting started guide
- MCP Python SDK — build servers in Python
- MCP TypeScript SDK — for TypeScript
- Anthropic MCP Announcement — why this exists
- Martin Fowler on Function Calling — the clearest architectural explanation of how tool use works
- MCP Reference Servers — official examples (filesystem, GitHub, Postgres, Slack, etc.)
One agent hits limits. Complex tasks benefit from multiple specialized agents working together — a planner that decomposes tasks, a coder that writes, a reviewer that critiques, a tester that validates. This is where orchestration patterns matter.
Andrew Ng's four agentic design patterns (from his DeepLearning.AI course):
- Reflection: Agent reviews its own output and improves it
- Tool Use: Agent calls external tools (you already built this)
- Planning: Agent decomposes a task into subtasks before executing
- Multi-Agent: Multiple agents with different roles collaborate
Additional orchestration patterns from Anthropic's blog:
- Prompt Chaining: Output of one step feeds into the next (pipeline)
- Routing: A classifier sends requests to specialized handlers
- Parallelization: Multiple LLM calls run simultaneously, results aggregated
- Orchestrator-Workers: A central agent delegates to and collects from worker agents
- Evaluator-Optimizer: One agent generates, another evaluates, loop until quality threshold
Week 10 — Reflection & Planning:
- Build a self-reflecting code writer: generate → critique → revise → repeat
- Build a planning agent: given a complex task, decompose into subtasks, execute each, synthesize
- Implement Chain-of-Thought (CoT) prompting as a first-class pattern
- Try extended thinking / reasoning models (Claude with extended thinking, o1/o3)
Week 11 — Multi-Agent System:
- Build a code review system with 3 agents:
- Planner: breaks the task into steps
- Coder: writes the code
- Reviewer: reviews and requests changes
- Implement agent-to-agent communication (message passing)
- Add a shared scratchpad / blackboard for inter-agent state
- Handle disagreements — what happens when the reviewer keeps rejecting?
Week 12 — Orchestration Patterns:
- Build a routing system: classify incoming requests → dispatch to specialized agents
- Build parallel execution: run multiple agents simultaneously, merge results
- Build an orchestrator-worker pattern with dynamic task delegation
- Add human-in-the-loop: pause for human approval at critical decision points
- Andrew Ng — Agentic AI Course — the four patterns explained clearly
- Chain-of-Thought Paper — Wei et al., the original
- Tree of Thoughts Paper — extends CoT with exploration
- Anthropic Extended Thinking — built-in reasoning
Now you understand why these frameworks exist:
- LangGraph — stateful agent graphs with cycles, persistence, and human-in-the-loop
- CrewAI — role-based agent teams
- AutoGen — multi-agent conversation framework from Microsoft
- OpenAI Agents SDK — handoff patterns between agents
Real agents need to manage complex, long-running tasks — decompose goals, track progress, handle failures, resume interrupted work. This is the "executive function" of your agent system.
Build a task management system for your agents:
User Goal
└── Task Decomposition (LLM breaks goal into tasks)
├── Task 1 [completed] → result stored
├── Task 2 [in_progress] → agent working
├── Task 3 [blocked] → waiting on Task 2
└── Task 4 [pending]
Week 13 — Task Decomposition & Tracking:
- Build a task graph: nodes are tasks, edges are dependencies
- LLM-powered decomposition: given a goal, generate a task plan
- Task states: pending → in_progress → completed / failed / blocked
- Dependency resolution: don't start blocked tasks until dependencies complete
- Progress reporting: agents report back on each completed subtask
- Persistence: save task state to disk so work survives restarts
Week 14 — Advanced Patterns:
- Dynamic replanning: when a task fails, the planner adjusts the remaining plan
- Parallel execution: run independent tasks simultaneously
- Checkpointing: save intermediate results so you can resume from any point
- Token budget management: track how much context each task is consuming
- Build a simple TUI (terminal UI) that shows task progress in real-time
- Integrate with your multi-agent system from Phase 5
- How Claude Code manages tasks internally (TaskCreate/TaskUpdate pattern)
- How Devin and similar coding agents decompose complex PRs
- The "plan → execute → observe → replan" cycle
- Hierarchical task networks (HTN) — classical AI planning meets LLMs
- Google Cloud — Agent Architectures — production patterns
- Microsoft — Agent Architecture Design — state management patterns
Without memory, every conversation starts from zero. Memory is what turns a chatbot into an assistant that knows your codebase, your preferences, and your project history.
Three types of memory:
- Working Memory: The current conversation context (you already manage this)
- Short-Term / Episodic: Recent interactions, what happened today/this week
- Long-Term / Semantic: Persistent knowledge — user preferences, codebase understanding, learned patterns
Week 15 — Build the Memory Layer:
- Implement a conversation summarization system (compress old messages while preserving key info)
- Build episodic memory: after each session, extract and store key facts/decisions/outcomes
- Build semantic memory: embed and index memories for retrieval (reuse your RAG skills)
- Build a memory retrieval system: given current context, fetch relevant memories
- Add memory types: facts, preferences, decisions, code patterns
- Implement forgetting: decay old memories, merge similar ones, prune irrelevant ones
Week 16 — Integrate Everything:
- Connect memory to your coding agent from Phase 1
- Agent remembers: user's coding style preferences, past bugs, project architecture decisions
- Add memory write triggers: "remember this", or auto-detect important information
- Build a memory inspection UI (what does the agent "know" about you?)
- Test memory persistence across sessions
┌─────────────────────────────────────────┐
│ Working Memory │
│ (current conversation context) │
├─────────────────────────────────────────┤
│ Episodic Memory │
│ (recent sessions, stored as summaries) │
├─────────────────────────────────────────┤
│ Semantic Memory │
│ (long-term facts, preferences, │
│ patterns — stored as embeddings) │
├─────────────────────────────────────────┤
│ Procedural Memory │
│ (learned workflows, tool usage │
│ patterns — stored as few-shot │
│ examples or fine-tuning data) │
└─────────────────────────────────────────┘
- Lilian Weng — LLM Powered Autonomous Agents — the Memory section is excellent
- Mem0 — open-source memory layer (study the architecture, then build your own)
- MemGPT / Letta — virtual context management for LLMs
Evals are not a phase — they should be running from Phase 1. But this phase is about building a proper evaluation system. Without evals, you're just vibing. Hamel Husain puts it best:
"If you aren't doing evals, you aren't doing AI engineering."
Evals tell you:
- Is your agent actually getting better when you change the prompt?
- Which model works best for your use case?
- Where does your RAG pipeline fail?
- Are you regressing when you ship changes?
Core eval types:
- Unit evals: Does the LLM produce correct output for known inputs? (like unit tests)
- Component evals: Does retrieval find the right chunks? Does the tool selector pick the right tool?
- End-to-end evals: Given a task, does the agent complete it correctly?
- LLM-as-judge: Use a stronger model to grade a weaker model's output
- Human evals: For subjective quality, nothing beats human review (but it doesn't scale)
Eval infrastructure:
- Build a test harness: define input/expected_output pairs, run them, score results
- Implement exact match, fuzzy match, and LLM-as-judge scoring
- Add cost and latency tracking to every eval run
- Build comparison: run same evals across models/prompts, generate a comparison table
- Version your prompts and track which version produced which eval scores
- Set up CI integration: evals run on every prompt change
Eval your systems:
- Eval your RAG pipeline: retrieval precision/recall, answer correctness
- Eval your coding agent: does it produce working code? Does it pass tests?
- Eval your structured extraction: schema compliance, field accuracy
- Eval your routing: does it pick the right agent/handler?
- Build a regression test suite: things that broke before should never break again
- Hamel Husain — "Your AI Product Needs Evals" — the essential post on why and how
- Hamel Husain — "Creating Domain-Specific Evals" — practical guide to LLM-as-judge
- Eugene Yan — "Patterns for Building LLM-based Systems & Products" — comprehensive patterns including eval strategies
- Anthropic Eval Guide — how Anthropic thinks about testing
- Braintrust — managed eval platform with logging, scoring, experimentation
- Promptfoo — open-source CLI for prompt testing and eval (now part of OpenAI)
- Inspect AI — UK AI Safety Institute's eval framework for agents
- SWE-bench — the standard benchmark for coding agents
Once you've built all of the above, here's what real production systems add:
- Trace every LLM call, tool execution, and agent decision
- LangSmith — tracing for LLM apps
- Braintrust — logging and monitoring
- OpenTelemetry + custom spans for your agent loops
- Input validation (prompt injection detection)
- Output validation (no PII leakage, no harmful content)
- OWASP Top 10 for LLMs — security checklist
- Guardrails AI — validation framework
- NeMo Guardrails — conversational safety
- Prompt caching (Anthropic supports this natively)
- Model routing: use cheap models for simple tasks, expensive ones for hard tasks
- Batch processing for non-realtime workloads
- Track cost-per-task, not just cost-per-token
- Vercel AI SDK — for TypeScript/Next.js apps with streaming
- Edge deployment for low-latency inference
- Queue-based architectures for long-running agent tasks
| Blog | Why |
|---|---|
| Simon Willison | Most prolific and practical LLM blogger. Annual recaps are must-reads. |
| Lilian Weng (Lil'Log) | Deep technical posts on agents, memory, and reasoning |
| Hamel Husain | The evals authority. Everything he writes about LLM systems is gold. |
| Eugene Yan | Patterns for production LLM systems |
| swyx / Latent Space | The AI Engineer community hub |
| Jason Liu (Instructor) | Structured outputs, practical LLM engineering |
| Chip Huyen | ML systems design and AI engineering |
| Podcast | Why |
|---|---|
| Latent Space | The AI Engineer podcast — interviews with builders |
| Practical AI | Grounded, practical AI discussions |
| Course | Why |
|---|---|
| DeepLearning.AI — Agentic AI | Andrew Ng's agent design patterns course |
| DeepLearning.AI Short Courses | Free, focused courses on specific topics (LangGraph, RAG, function calling) |
| Karpathy — Neural Networks: Zero to Hero | If you want deeper model understanding |
| Paper | Year | Why |
|---|---|---|
| ReAct | 2022 | Foundation of tool-using agents |
| Chain-of-Thought | 2022 | Why "think step by step" works |
| Tree of Thoughts | 2023 | Multi-path reasoning |
| Retrieval-Augmented Generation | 2020 | The RAG paper |
| Toolformer | 2023 | Models learning to use tools |
- Latent Space — 2025 AI Engineering Reading List — 50 papers across 10 fields, curated by the community
| Repo | Why |
|---|---|
| anthropic-cookbook | Official Anthropic patterns and examples |
| openai-cookbook | Official OpenAI patterns and examples |
| openai-agents-python | Clean agent SDK design to study |
| pydantic-ai | Type-safe agent framework |
| instructor | The structured output library |
| langchain | Understand the abstractions (even if you don't use them) |
| langgraph | Stateful agent orchestration |
| mem0 | Memory layer architecture |
| chroma | Simple vector DB to learn from |
| claude-code | A real production coding agent |
Things you'll learn that no tutorial teaches:
-
Prompts are code. Version them. Test them. Review them in PRs. A prompt change can break your system as badly as a code change.
-
The hard part isn't the LLM call. It's everything around it — error handling, retries, context management, cost control, eval, and the glue code between tools.
-
Start without frameworks. Then add them only when you've felt the pain they solve. If you start with LangChain, you'll never understand what LangChain is doing for you.
-
Evals are your compass. Without them, you're making changes and hoping they work. With them, you're making changes and knowing whether they work.
-
Simple beats clever. A single well-prompted LLM with good tools beats a complex multi-agent system 90% of the time. Add complexity only when simple doesn't work.
-
Models improve faster than your code. Design for model swappability. Today's complex workaround might be solved by next month's model.
-
Context window is your most precious resource. Every token you send costs money and displaces useful information. Be ruthless about what goes in.
-
Agents fail silently. Unlike traditional code that crashes, agents produce plausible-looking wrong answers. Evals catch this. Vibes don't.
| Week | Phase | What You Build |
|---|---|---|
| 1 | Phase 0 | Raw API chat loop, streaming, token management |
| 2-3 | Phase 1 | Coding agent from scratch (ReAct loop + tools) |
| 4 | Phase 2 | Structured extraction pipeline |
| 5-7 | Phase 3 | RAG system (naive → advanced → agentic) |
| 8-9 | Phase 4 | MCP server + pluggable tool ecosystem |
| 10-12 | Phase 5 | Multi-agent orchestration (reflection, planning, multi-agent) |
| 13-14 | Phase 6 | Task management & state persistence |
| 15-16 | Phase 7 | Memory system (episodic + semantic + procedural) |
| 17-18 | Phase 8 | Eval framework + eval all previous phases |
| 19-20 | Bonus | Production hardening, observability, deployment |
The AI engineering ecosystem moves fast. By the time you finish this roadmap, there will be new models, new tools, new patterns. That's fine. The point isn't to learn today's tools — it's to build the mental models that let you evaluate and adopt tomorrow's tools in minutes instead of weeks.
You're not learning LangChain. You're learning what LangChain solves. That's what lasts.
Now go build something.
Last updated: March 2026