Skip to content

Instantly share code, notes, and snippets.

@donbr

donbr/session12-production-agent-learning-journey.md

Created July 9, 2026 23:41
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save donbr/1d872eb764a257ad2248e11e255085bb to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save donbr/1d872eb764a257ad2248e11e255085bb to your computer and use it in GitHub Desktop.
Download ZIP
Session 12 — Production Agent Patterns: Learning Journey
Raw
session12-production-agent-learning-journey.md

Session 12 — Production Agent Patterns: Learning Journey (Student Version)

A companion to the three parts of this session — the two notebooks (01_Cat_Health_Agent_Guardrails.ipynb, 02_Cat_Health_Agent_Caching.ipynb) and the a2a/ mini-project. It's a run log: every part was actually executed — uv sync, both notebooks against the live API, and the full A2A server + client + delegation chain — so you have a "known-good" reference for what a healthy run looks like and where people trip. Use it to tell a real problem apart from expected run-to-run noise.

This is the student version. The engineering findings and setup gotchas are all here, but the graded reasoning is yours to do — this session has no question cells; you demonstrate understanding through your executed work (kept decision tables, cache timings, the A2A trace). Prompts to check your own understanding are near the end, without answers. No API keys appear here.

A reference environment (yours may differ): WSL2/Linux, uv, Python 3.13, run inside the session's own .venv. LangChain v1 (create_agent + middleware), openai 2.x. An OPENAI_API_KEY available when you run the notebooks or the A2A server.

Legend: ✅ verified by actually running it · ⚠️ watch this — the spot people trip · 💡 optional, do-it-better tip

The short version: does it all run?

Yes — all three parts run clean out of the box. uv sync resolves without a fight, the A2A smoke test passes with no API key, both notebooks execute top-to-bottom, and one agent really does delegate to another over the protocol. There are no broken imports and no wrong APIs. Everything that goes wrong for students is environment setup — your API key and which model you're allowed to call — and every fix is a one-line shell export, never a code edit. The notes below are the handful of places people actually get stuck.

Part-by-part: what a healthy run looks like

01_Cat_Health_Agent_Guardrails.ipynb ✅ runs clean

Executes end-to-end. When it's working you should see, in order:

  • Task 3 — the decision table: five inputs, five different verdicts — allow, two escalate (an emergency like a seizure, and "ate chocolate"), one block (an injection attempt), one rewrite (PII replaced with [REDACTED_EMAIL] / [REDACTED_PHONE]). If every input gets the same verdict, your rail precedence is off.

  • Task 4 — topic verdicts: three structured TopicVerdict objects (on_topic=True/False/False), not prose paragraphs. Structured output is the point.

  • Task 5 — output rails: one draft replaced by a safe fallback (diagnosis/dosage language), one repaired (a vet disclaimer appended), one left alone (none).

  • Task 6 — guarded runs: a normal question gets a real answer; a "seizure right now" input returns the canned emergency redirect without calling the model (that's the can_jump_to=["end"] short-circuit doing its job).

  • ⚠️ This notebook uses LangChain v1. The imports are from langchain.agents import create_agent and from langchain.agents.middleware import before_model, after_model. If you get an ImportError here, you're running an old LangChain — you're in the wrong interpreter. Fix it by selecting the uv environment, never by editing the imports.

02_Cat_Health_Agent_Caching.ipynb ✅ runs clean · ⚠️ numbers wander

Executes end-to-end. A healthy run shows:

  • Task 3 — exact cache: a big cold → warm gap (warm is ~0), then the punchline: add one character and you pay full price again.

  • Task 4 — semantic cache: a paraphrase check, and the chocolate-vs-chicken danger demo printing a similarity score.

  • Task 5 — embedding + tool caches: 1 hit / 1 miss, and a tool that shows slow lookups so far: 1 on both runs (the slow backend ran only once).

  • Task 6 — prompt cache: two calls; the second reports served from prompt cache: <a big number>.

  • ⚠️ The similarity numbers are not fixed — and that's the lesson, not a bug. On a real run the paraphrase can score below the 0.90 threshold and MISS (that happened on our validation run: 0.826), and the chocolate-vs-chicken number can land around 0.74, not some dramatic 0.95. Do not "fix" this by lowering the threshold and declaring victory — that's the exact trap the notebook is setting. The point is that two questions with opposite stakes (a poisoning emergency vs. dinner) sit dangerously close in embedding space, and no threshold cleanly separates "safe to reuse" from "dangerous to reuse." Keep the danger cell and reason about why the number is scary, whatever decimal you get.

  • 💡 For Task 6, if the first prompt-cache call shows cached_tokens: 0, that's fine — the provider hadn't warmed the prefix yet. The notebook's closing "Notebook Output Guidance" cell tells you a short wait-and-rerun is allowed.

a2a/ mini-project ✅ full flow verified

The runnable part with no notebook. Do the checks in the README's order:

  • ✅ Smoke test first — no API key needed. uv run python smoke_test.py prints five PASS lines and "All A2A protocol smoke tests passed." This is your best first move: a green smoke test proves your environment and the server plumbing work, so if a later step fails you know it's the key or the model, not the code.

  • ✅ Card discovery: curl …/.well-known/agent-card.json returns the agent's public résumé — and notice what it omits: no model name, no tool list, no framework. That omission is the whole idea.

  • ✅ Client round-trip: uv run python client.py "…" discovers the card and gets an answer over JSON-RPC.

  • ✅ Delegation: uv run python front_desk.py answers "clinic hours" itself (no [A2A] lines) but sends a cat-health question across the protocol (you'll see [front desk → A2A] … and [A2A → front desk] …). Those trace lines appearing on one question but not the other are the evidence that delegation is real.

  • ⚠️ Two terminals. server.py blocks — leave it running in terminal 1 and run client.py/front_desk.py in terminal 2. Both of those must be run from inside a2a/. If you get "address already in use," an old server is still on port 9999 — kill it or change PORT.

The five real trip-wires

  1. ⚠️ Your API key — and the .env trap. There is no load_dotenv() anywhere in this session, so copying .env.example to .env and filling it in does nothing by itself. Two things actually work: export OPENAI_API_KEY="…" in your shell before you launch Jupyter or the server, or type it at the getpass prompt when the notebook/server asks. (client.py and smoke_test.py need no key at all.)
  2. ⚠️ The default model must be one your account can call. Everything defaults to a specific OpenAI model. If your first live call dies with model_not_found / a 404, your project doesn't have access — that's not a code bug. Override it without touching any file: 
    export AIM_CHAT_MODEL="a-model-your-account-can-call"
    The notebooks and the A2A agents read that env var first.
  3. ⚠️ Python 3.13. This session requires Python 3.13 (pyproject.toml + .python-version). uv sync fetches it for you automatically — so use uv. A hand-rolled venv on Python 3.11/3.12 will fail to install.
  4. ⚠️ Semantic-cache scores drift (see Part 2). A MISS on the paraphrase or a chocolate/chicken score in the 0.7s is not a mistake — it's the material. Reason about the concept; don't chase a specific number.
  5. ⚠️ A2A wants two terminals and a free port 9999. Start with the no-key smoke_test.py to isolate any problem before you bring the key and the live server into it.

Check your own understanding (no answers here)

This session is graded on your executed work, not on filled-in blanks. If you can answer these from your own runs, you've got it. They're a method, not a key.

Guardrails (Notebook 1)

  • You have a good system prompt telling the agent to be safe. Why isn't that enough — what can a rail do that a prompt can't? Which of your rails runs before the model, and which after, and why does the order matter for cost?
  • Three inputs get three different actions: an emergency, an injection attempt, and a message containing an email address. Why is the right response to each one different — and why is redacting PII the correct move rather than blocking?
  • The deterministic rails are regex. Name one thing regex will never catch — and point to the cell that exists specifically to cover that gap.

Caching (Notebook 2)

  • The exact-match cache is free and perfect — until it isn't. What single change to a question makes it useless, and what kind of traffic does that leave it good for?
  • Look at your similarity(chocolate, chicken) number. Two questions, one word apart, opposite stakes. If you lowered the threshold until real paraphrases hit, what happens to this pair? Is there a threshold that's both safe and useful — and if not, what has to change instead?
  • Which piece of Notebook 1 is the actual fix for the semantic-cache danger? (Hint: what decides whether a query is even allowed to be cached?)
  • In Task 6, why does putting the system prompt and tools first and the user's question last save money, and what one careless line at the top of a prompt would destroy prompt caching for your whole app?

A2A (a2a/)

  • MCP connects an agent to a tool; A2A connects an agent to another agent. What's the difference between a tool and an agent on the other end of the wire — who does the thinking?
  • Open the agent card. List what it tells you and what it deliberately hides. Why is hiding the model, tools, and framework a feature of the protocol, not a gap?
  • In your front_desk.py trace, the clinic-hours question shows no [A2A] lines but the health question does. What in the code decides, per question, whether to delegate — and where did the front desk even learn the specialist exists?

References

The "known-good" sources when something disagrees with what you see.

Tools move fast. If a signature or a model name disagrees with these docs, the docs win — and remember the model is overridable via AIM_CHAT_MODEL, so a model error is almost never a reason to edit the notebook.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment