PROMPT: let's iteratively build a prompt to help drive the generation of more effective and modular code based on good architecture and design for effective agent harnesses and guardrails... along with python code.
All four views rendered. Here they are, then the answer to the real question.
View 1 — The seam thesis 🎯 Agent Seam Eval Framework
🎨 Open/Edit diagram
mermaid
flowchart LR
subgraph PROD["1 - What already exists in production"]
direction TB
APP["App / API route"]
LOOP["Agent loop
prompt + model + tools"]
TOOLS["Tools: retrieval, DB, APIs"]
APP --> LOOP
LOOP <--> TOOLS
end
subgraph SEAM["2 - The seam you insert once"]
direction TB
DRIVE["AgentUnderTest
run to Turn"]
OBS["TraceSink
records ToolCall events"]
end
subgraph ONLINE["3a - Guardrails: synchronous, in the request path"]
direction TB
IN["input rails
block / escalate / redact"]
POL["policy rails
budgets, tool allowlist"]
OUT["output rails
replace / repair"]
end
subgraph OFFLINE["3b - Eval harness: same seam, driven by a simulator"]
direction TB
TASKS["TaskSource"]
SIM["UserSimulator"]
SCORE["Scorers"]
GATE["Report + regression gate"]
end
LOOP -.->|"implement once"| DRIVE
TOOLS -.->|"emit events"| OBS
DRIVE --> IN
IN --> POL
POL --> OUT
OBS --> POL
TASKS --> SIM
SIM <--> DRIVE
OBS --> SIM
SIM -->|Trajectory| SCORE
SCORE --> GATE
style SEAM fill:#fff3cd,stroke:#d39e00
style PROD fill:#e7f1ff,stroke:#0d6efd
style ONLINE fill:#e6f4ea,stroke:#198754
style OFFLINE fill:#fce8ec,stroke:#d63384
This is the whole argument. Guardrails and eval harnesses are the same insertion, differing only in who drives and whether you can mutate. A rail is synchronous and driven by a real user; an eval is offline and driven by a simulator. Build the seam once, get both. That's why your two topics belong in one prompt rather than two.
View 2 — Why you can't insert anything today 🎯 Notebook Coupling Model
🎨 Open/Edit diagram — generated from the AST scan above, not by eye.
Every function reaches into module globals. Two things worth naming:
- _declined is wired across two layers — it's a scoring predicate (verify) and a control-flow predicate (_wants_clarification). Widen the decline vocabulary to fix scoring and you silently change when conversations terminate.
- run_harness → ARTIFACTS is Trap A's architectural root. The cache is keyed by a human-chosen label because there's no object whose identity could key it.
The notebook isn't badly written — it's a script, and scripts fuse layers by design. The lesson is that the fusion is what makes it un-insertable.
View 3 — Target architecture 🎯 Pluggable Eval Architecture
🎨 Open/Edit diagram
One rule: arrows point inward. core/ imports nothing. Swapping OpenAI for LangGraph, or JSONL for Postgres, is a new file in adapters/ — core/ and policy/ never change.
View 4 — Runtime 🎯 Trajectory Scoring Flow
🎨 Open/Edit diagram · View 5 — Type model 🎯 Trajectory Eval Type Model 🎨 Open/Edit
"Outside a notebook, how do I define this?"
The honest answer: you don't start by building a harness. You start by writing ~15 lines of adapter, capturing the one trajectory that broke, and turning it into a test. The harness is what you have three weeks later.
Step 1 — The only code that knows about your agent
class MyProdAgent: agent_id = fingerprint_config(MODEL, SYSTEM_PROMPT, TOOL_SCHEMA) # NOT a human label
def run(self, messages: Sequence[Message]) -> Turn:
reply, raw_calls = my_existing_agent.invoke(messages) # your code, untouched
return Turn(reply, tuple(ToolCall(c.name, c.args) for c in raw_calls))
agent_id being a content hash of the agent's configuration is the load-bearing detail. Change the system prompt → new id → new fingerprint → the cache actually re-runs. That's Trap A fixed structurally rather than by remembering to pass force=True.
Step 2 — The ports, in full
class AgentUnderTest(Protocol): agent_id: str def run(self, messages: Sequence[Message]) -> Turn: ...
class Scorer(Protocol): scorer_id: str def handles(self, spec: TaskSpec) -> bool: ... def score(self, traj: Trajectory) -> Verdict | None: ... # None = abstain
None = abstain is the design. The notebook's score_task switches on category — the scorer knows the taxonomy, so adding a capability means editing the runner. Here, scorers declare what they handle and the chain takes the first non-None. "Programmatic first, judge only if undecided" becomes ordering, not a conditional. Adding multi_hop touches zero existing files.
Step 3 — Fix the class of bug, not the bug
The lowercase patch I applied to the notebook was a one-liner. The architectural version makes it unrepresentable:
@dataclass(frozen=True, slots=True) class ToolsAndFacts(SuccessCriterion): facts: frozenset[NormalizedText] def post_init(self): object.setattr(self, "facts", frozenset(norm(f) for f in self.facts))
Normalize at construction, store normalized, and there is no un-normalized value left to compare. Same for min_calls: name it min_distinct_tools vs min_tool_calls and Finding 1 stops being ambiguous — the bug was that one field name meant two things.
Step 4 — Rails are the same shape
class Rail(Protocol): rail_id: str def check(self, ctx: RailContext) -> Allow | Block | Rewrite | Escalate: ...
A Rail is a Scorer that runs online and may mutate. Because both sit on the same seam, your eval harness can test your guardrails — which is the thing neither Session 12 nor Session 18 does on its own.
Where I'd take the prompt
The prompt shouldn't say "write modular code." It should encode the failure modes we actually measured, as checks a generator must pass — every one traceable to evidence from this session:
┌──────────────────────────────────────────────────┬───────────────────────┐ │ Rule the prompt enforces │ Evidence it came from │ ├──────────────────────────────────────────────────┼───────────────────────┤ │ Cache keys are content hashes of inputs, never │ Trap A │ │ human labels │ │ ├──────────────────────────────────────────────────┼───────────────────────┤ │ Normalize at construction; never compare raw to │ Trap B │ │ normalized │ │ ├──────────────────────────────────────────────────┼───────────────────────┤ │ One name = one meaning (min_distinct_tools) │ Finding 1 │ ├──────────────────────────────────────────────────┼───────────────────────┤ │ Scorers abstain; the runner never switches on a │ score_task dispatch │ │ taxonomy │ │ ├──────────────────────────────────────────────────┼───────────────────────┤ │ No predicate serves both scoring and control │ _declined │ │ flow │ │ ├──────────────────────────────────────────────────┼───────────────────────┤ │ Seeds are strings, never hash() │ Finding 3 │ ├──────────────────────────────────────────────────┼───────────────────────┤ │ Judged checks report the raw verdict string │ "IGNORED" in "NOT │ │ │ IGNORED" │ └──────────────────────────────────────────────────┴───────────────────────┘
Before I draft it, one thing decides the shape:
Is this prompt for you driving Claude Code in a fresh repo, or is it a reusable skill you'd ship into .claude/skills/ for the cohort? The first wants a dense one-shot spec; the second wants a SKILL.md with progressive disclosure, a checklist, and a reference implementation — and that changes what I write next.