Skip to content

Instantly share code, notes, and snippets.

@donbr

donbr/session18-sdg-sgent-trajectory-evals-lj.md

Created July 30, 2026 23:37
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save donbr/62efd645b2a1098db3e9515cf8ea27f7 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save donbr/62efd645b2a1098db3e9515cf8ea27f7 to your computer and use it in GitHub Desktop.
Download ZIP
Session 18 — SDG for Agent Trajectory Evals: Learning Journey
Raw
session18-sdg-sgent-trajectory-evals-lj.md

Session 18 — SDG for Agent Trajectory Evals: Learning Journey (Student Version)

A companion to 01_SDG_Agent_Trajectory_Evals.ipynb. It's a run log: the whole notebook was actually executed against the live OpenAI API, cold, start to finish — so you have a "known-good" reference for what a healthy run looks like, what the numbers do, and where people trip. Use it to tell a real problem apart from expected run-to-run variation.

This is the student version. The setup gotchas and the run log are all here. The graded reasoning is yours: this session has four questions and two activities, and none of them are answered in this document. Where our run turned up something surprising, you'll find the observation and a place to look — not the diagnosis. Working that out is the assignment. No API keys appear here.

A reference environment (yours may differ): WSL2/Linux, uv, Python 3.13, run inside the session's own .venv. openai 2.x, pandas 3.x, numpy 2.x. Defaults: AGENT_MODEL=gpt-4.1-mini, JUDGE_MODEL=gpt-4.1-nano, EMBED_MODEL=text-embedding-3-small — all three resolved fine on a standard OpenAI account.

Legend: ✅ verified by actually running it · ⚠️ watch this — where people trip · 🔎 something our run turned up that's worth your attention · 💡 optional tip

The short version: does it all run?

Yes — the notebook runs clean, end to end, exit 0, in about 3.5 minutes cold. No broken imports, no wrong APIs, no missing data. The corpus is complete (all 113 Articles), all three model defaults work out of the box, and both harness runs and both charts render.

But the interesting part is what the numbers did. Our run and the run whose outputs ship in the notebook produced the exact same overall pass rate — 68%, dropping to 52% after Task 7's regression, Δ −16% in both — while four of the five capability bars differed by 20 to 40 points:

Capability Notebook's shipped run Our run
tool_selection 80% 100%
tool_chaining 40% 20%
out_of_scope 80% 100%
adversarial 100% 100%
edge_case 40% 20%
overall 68% 68%

Same headline number. Almost entirely different agent. That is the whole point of Task 6 and Task 7, and you got it for free just by running the notebook twice on different machines. Expect your numbers to differ from both columns. Nobody's run is wrong.

Triage: "something looks wrong" → what's actually going on

You won't recognise these by name — you'll recognise them by what's on your screen. Find your symptom here first.

⛔ = a real defect in the harness that fails silently. Not your fault, and you will not get an error message. Fix it and move on. 🔎 = not a defect — this is the thing the assignment wants you to investigate. Don't let anyone hand you the answer.

What you see What's going on Where
⛔ I added an atom in Activity #1 and its category is stuck at 0% Your fact substring is probably uppercase, or it doesn't appear in the part of the article the tool actually returns Trip-wires 1 and 7
⛔ My Activity #2 delta table is +0% across every row, or my numbers didn't budge after I clearly changed the agent You re-ran under a label that was already cached — the new agent never actually ran Trip-wire 2
⛔ I improved the prompt three times and the table matches my first attempt Same cause: every run after the first returned the first run's cached rows Trip-wire 2
⚠️ My category numbers don't match the notebook's shipped outputs, or my classmate's Expected. Live models vary, and your task suite is regenerated differently on each kernel restart Trip-wire 3 + the top of this doc
⚠️ I restarted the kernel and my tasks changed Expected — and it's why tasks.jsonl is cached. Don't delete it mid-assignment Trip-wire 3
⚠️ A cell finished instantly and printed loaded … cached The cache did its job. Three separate things cache here: embeddings, tasks, and eval rows Trip-wires 2 and 3
⚠️ model_not_found / 404 on the first live call Your account can't call that model. No code edit needed Trip-wire 6
⚠️ uv sync or the install fails You're not on Python 3.13 Trip-wire 5
🔎 tool_chaining is red but the answers look complete and correct to me Real observation. Something in verify() is rejecting a good answer — find out what "Two categories will probably look bad"
🔎 edge_case is red but the answers look helpful to me Real observation. Ask what is actually doing the scoring for that category "Two categories will probably look bad"
🔎 edge_case went up when Task 7 unplugged the tools Real, and it happens in every run we've seen. An agent with no tools got better at something — that's the question "Two categories will probably look bad"
🔎 A category came back at 100% Could be a good agent; could be an eval that stopped measuring anything "Check your own understanding"
🔎 The Task 4 demo made more than the three tool calls its prompt allowed Real. A prompt asks; it doesn't enforce Task 4 note above

The one habit that resolves most of this: open artifacts/eval_baseline.jsonl and read the actual rows — the answer, the tool count, the turn count — before you believe any bar in the chart. That is also literally step 1 of Activity #2, and it is the whole skill this session teaches.

What a healthy run looks like, task by task

Breakout Room #1

  • ✅ Task 1 — setup. uv sync resolves cleanly, then you should see three model names printed and judge online. If judge online prints, your key, your endpoint, and your judge model are all good.

  • ✅ Task 2 — the agent. 113 articles in the corpus (that number should be exact — Articles 1 through 113 with no gaps). Then a cold embedding pass: embedded 333 chunks -> artifacts/…npy, about 25 seconds. On every later run it says loaded 333 cached chunk embeddings and is instant — that's the cache working, not a skipped step. The smoke test should answer the fine question with EUR 35,000,000 / 7%, Article 99 and a tool trace of get_penalties.

  • ✅ Task 3 — composing tasks. Three JSON specs print, each with a success block. Then two sample openings — one chained question, and one adversarial question with a canned injection glued on the end.

  • ✅ Task 4 — the user simulator. The demo runs a genuine two-turn conversation: vague opening → the agent asks what the system does → user_sim answers with the resume-screening scenario → the agent chains tool calls and lands on a correct Annex III / Article 6(3) answer. This is the one place you will actually watch a multi-turn trajectory happen. (Hold that thought for the investigation prompts below.)

    💡 The agent in this demo often makes more than three tool calls even though SYSTEM_CLARIFY says "make at most three." That's not a bug in your code, and it's worth a moment of thought — a prompt asks; it doesn't enforce.

Breakout Room #2

  • ✅ Task 5 — scoring. 25 tasks, each printing pass= and a time in seconds. Ours took 1–5 seconds each.
  • ✅ Task 6 — the capability report. A Markdown table plus a bar chart, with the report also written to artifacts/capability_report.md.
  • ✅ Task 7 — the regression. The punchline lands reliably: with the tools unwired, every tool-dependent category goes to literal 0% while the overall average only sags by around 15 points. Both our run and the shipped run gave exactly 68% → 52%. This is the most reproducible moment in the session.

⏱️ It's faster than the warning says

The README and the notebook both warn "~5 minutes each" for Tasks 5 and 7. What we actually measured, cold, on a normal account:

Phase Observed
Cold corpus embedding 26 s
Generating the 25 task openings 27 s
Baseline harness (25 trajectories) 69 s
Tools-disconnected harness (25 trajectories) 45 s
Whole notebook, cold ~3.5 min

Treat 5 minutes as a safe ceiling, not a forecast. Our advice: leave SEEDS_PER_CATEGORY = 5. At 2 seeds each category bar moves in 50-point jumps, which makes Activity #2 genuinely harder to reason about — you won't be able to tell a real improvement from one lucky sample.

🔎 Two categories will probably look bad. Read this before you "fix" them.

In both our run and the shipped one, tool_chaining and edge_case came back red (20–40%). If yours do too, you have not broken anything — and, more importantly, do not immediately assume your agent is the problem.

Activity #2 tells you to read the failing rows in artifacts/eval_baseline.jsonl before you change anything. That instruction is doing more work than it looks like. When we did it, the reasons those two bars were red turned out to be more interesting than "the agent is bad at chaining."

We're not going to tell you what we found — that's the activity. But here is exactly where to look:

For tool_chaining:

  • Open a failing row and check its n_tools count. Then open verify() and read the tools_and_fact branch character by character. What does len(set(tools)) actually count — the number of tool calls, or something else? Now imagine the most natural correct trajectory for a question spanning Article 5 and Article 50. Does that trajectory pass?
  • Find a failing row where the answer looks complete and correct to you. Check each required facts substring against the answer by hand. If they're all there, the failure came from somewhere else in verify() — where?
  • One of our chaining tasks demanded the substring "35" for a question about breaching Article 50 transparency obligations. Go read Article 99 in data/eu_ai_act.md — specifically, which fine tier does it assign to Article 50 breaches, and which one is the 35,000,000 tier for? Our agent answered the question correctly and the harness marked it FAIL. Who was wrong — the agent, or the task the generator composed? That question is the heart of Question #1, and the obvious answer is not the whole answer.

For edge_case:

  • Read the failing answers. Do they honestly look unhelpful to you? If not, the agent isn't what's being measured.
  • edge_case and adversarial are the two categories scored by JUDGE_MODEL — which defaults to the smallest, cheapest model in the notebook. Try running the same task suite past a bigger judge: 
    # in your .env, then restart the kernel
JUDGE_MODEL=gpt-4.1-mini
    ⚠️ You have to re-run the harness for this — run_harness(…, "baseline", force=True) under a new label. You cannot faithfully re-judge from artifacts/eval_*.jsonl, because those rows store only the first 300 characters of each answer. (Worth noticing on its own: what else does the cached row throw away that you might have wanted?) Keep tasks.jsonl so you're comparing on the same exam. Does the bar move? If it moves a lot, what does that tell you about what that bar was measuring all along?
  • Now look at the Task 7 table again. edge_case almost certainly went up when you unplugged the tools. An agent with no tools got better at something. Sit with that for a second — what quality is that category actually rewarding, and why does removing the tools not damage it?

⚠️ Both of these are legitimate Activity #2 answers if you show your work. Activity #2 says to change the agent, and changing the agent is the default path — but if you can demonstrate with evidence that a red bar was caused by the eval rather than the agent, and you fix that and re-run to prove nothing else moved, that's a stronger submission than a system-prompt tweak. Show the before, the change, and the after.

The real trip-wires

The first two will cost you an activity if you hit them, and neither one prints an error. Read these before you start Activity #1 or #2.

1. ⛔ Your new fact substring must be lowercase (Activity #1)

verify() lowercases the agent's answer before checking it — but it does not lowercase your required fact. So a fact of "General-Purpose" is compared against an answer that has already been flattened to "…general-purpose…", and it will never match.

TARGETS["gpai"] = {"desc": "…", "fact": "General-Purpose"}   # ❌ can never pass
TARGETS["gpai"] = {"desc": "…", "fact": "general-purpose"}   # ✅

The four facts that ship with the notebook happen to be lowercase already, so you'll never see this until you add your own. There's no error and no warning — just a category stuck at 0% and a lot of wasted time. If your new atom scores 0%, check the case of your fact before you touch anything else.

2. ⛔ Re-running Activity #2 under a label you already used does nothing

run_harness caches its results to artifacts/eval_<label>.jsonl, and the label is the only thing it checks. It does not notice that you changed the system prompt, the tools, or the tasks:

run_harness(SYSTEM_V2, TASKS, "baseline")   # ❌ returns the OLD baseline rows
run_harness(SYSTEM_V2, TASKS, "v2")         # ✅ actually runs your new agent

Activity #2 step 3 tells you to use a new label for exactly this reason. If you iterate — tweak the prompt, re-run, tweak again, re-run — under one label, every run after the first silently returns the first run's numbers, and you'll report a table that belongs to a prompt you already threw away.

The tell: the line loaded N cached results from … (set force=True to re-run). If you see that when you meant to run something new, you didn't run it. Either pick a fresh label or pass force=True.

⚠️ This compounds with trip-wire #3 below. If you do the Activity #2 stretch goal — raise SEEDS_PER_CATEGORY, delete tasks.jsonl, re-run — and your v2 label is still cached, you'll compare a fresh 40-task baseline against 25 stale v2 rows from a different exam. Delete every eval_*.jsonl whenever you delete tasks.jsonl.

3. ⚠️ Your task suite changes every time you restart the kernel

compose_task seeds its randomness with Python's built-in hash(). Python deliberately randomizes hash() for strings on every new process, so the targets — and therefore the required facts — that tool_chaining-0 gets are different in every fresh kernel session. We saw tool_selection-0 land on three different targets across three runs.

The notebook handles this correctly by caching the whole suite to artifacts/tasks.jsonl and measuring every agent version against that one file. But it means:

  • If you delete artifacts/tasks.jsonl, you have changed the test. Delete every artifacts/eval_*.jsonl at the same time and re-run the baseline, or your baseline→V2 delta table is comparing two different exams.
  • Don't compare your category numbers to the notebook's shipped outputs, or to a classmate's. Only the deltas inside your own notebook mean anything. That's not a limitation to apologize for in your write-up — it's the point.

4. ⚠️ Your API key — and here .env really does work

Unlike some earlier sessions, this notebook does call load_dotenv(), so all three paths work:

cp .env.example .env    # then fill in OPENAI_API_KEY  ← the documented path

…or export OPENAI_API_KEY="…" before launching Jupyter, or just type it at the getpass prompt when the first cell asks. .env is gitignored — never commit it.

5. ⚠️ Use uv, not a hand-rolled venv

pyproject.toml requires Python 3.13 (>=3.13,<3.14) and .python-version pins it. uv sync fetches 3.13 for you automatically. A manual venv on 3.11 or 3.12 will fail to install. After uv sync, select the uv environment as your Jupyter kernel in Cursor/VS Code.

6. ⚠️ If a model 404s, it's your account — not the code

All three defaults (gpt-4.1-mini, gpt-4.1-nano, text-embedding-3-small) worked for us. If one returns model_not_found, your project doesn't have access. Override it in .env — no notebook edit needed:

AGENT_MODEL=a-model-your-account-can-call
JUDGE_MODEL=a-model-your-account-can-call

Changing EMBED_MODEL re-embeds the corpus automatically (the cache filename carries the model name), so that's safe too.

7. 💡 Adding an atom (Activity #1)? Your fact also has to be findable

get_article() returns only the first 1,600 characters of an article — for Article 5 that's about 14% of it. So a fact substring that lives deep in an article can only be reached through search_act, and your task may be unpassable. Before you commit to a new atom, check your substring actually appears near the top of the article you're targeting. (If you go with Article 53 as the notebook suggests: some obvious-sounding candidates are in that first 1,600 characters and some are not. Check, don't assume.)

Check your own understanding (no answers here)

The four questions and two activities are graded on your reasoning. These are extra prompts to test whether your run actually taught you something — a method, not a key.

On the harness

  • Every task carries a success block that a plain Python function can check. What does that buy you that an LLM judge doesn't — and, harder, what can a programmatic check get wrong that a judge wouldn't? (Our run found a concrete example. Go find yours in the failing rows.)
  • Two different models are in play: one being tested, one doing the generating and judging. What specifically goes wrong if they're the same model — not just "bias," but the mechanism? What happens to your eval over many iterations?
  • verify() returns True, False, or None. Why three values and not two, and what happens downstream when it returns None?
  • Look at judge_pass. It asks the judge for one word and then tests whether that word appears in the reply. What would happen if you swapped in a judge that likes to explain itself before answering? Try writing down three replies a chatty judge might give, and trace what judge_pass returns for each.

On what your run actually did

  • Check the user_turns column across all 25 rows of artifacts/eval_baseline.jsonl. Compare that to the multi-turn conversation you watched in Task 4. Are they the same? If not — look at which system prompt the harness passes to run_trajectory versus which one the Task 4 demo passes, and read _wants_clarification carefully. What has to be true for the simulated user to speak a second time?
  • Your out_of_scope category probably looks stable across the regression. Read _declined()'s keyword list and ask whether every word in it is a safe signal — "outside" is an ordinary word in legal writing. Then ask what the decline check does in the tools-disconnected run, where there are never any tool calls. How much do you trust that row?
  • adversarial likely scored 100% in both arms. Before you celebrate: is the agent genuinely robust, or is the category too easy? Look at the three canned injections. What kind of attack is not represented there?

On the report itself

  • Your overall pass rate and a classmate's might match while your bars look nothing alike (that happened to us — see the top of this doc). What does that do to the sentence "our agent is 68% accurate"?
  • A category at 100% — is your agent good, or has the eval stopped telling you anything? What's the cheapest thing this harness lets you do to find out?
  • You changed the agent and one bar went up. Name every reason that bar might have moved that has nothing to do with your change. How many of them can you rule out with what's already saved in artifacts/?

References

The known-good sources when something disagrees with what you see.

Models and libraries move fast. Our reference run used Python 3.13.2, openai 2.51.0, pandas 3.0.5. Every model is overridable from .env, so a model error is almost never a reason to edit the notebook. And if a number here disagrees with your run — that's expected. Read your own cached rows in artifacts/ before assuming either of you is wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment