An agent that runs a single task is tractable. An agent that fans out into ten parallel subtasks — each with its own model call, its own filesystem state, its own failure modes — is a distributed system. And distributed systems fail in ways that single-process code does not: a node dies mid-fan-out, a subagent completes but its result gets lost, a retry re-executes work that already had side effects. The workflow system exists to make that distributed fan-out safe. It does so by making orchestration durable and deterministic: the control flow is code, not model-driven improvisation, and every node in the graph has a stable, replayable identity.
You have already seen, in Chapters 6 and 7, how the journal gives a single turn its durability — a monotonically advancing phase marker that tells recovery exactly how far the turn progressed before the process died. This chapter extends that idea upward. Where the journal durability operates at the boundary of a single model call, workflow durability operates at the boundary of an entire orchestration graph. The mechanisms are analogous: phase markers become node identifiers, the commit-before-deliver guarantee becomes a structural dispatch id derived from position rather than chance, and replay becomes re-walking the same code path to re-derive the same ids. The coordination primitive that makes all of this work is the coordinate path.
Before touching any workflow code, you need to resolve a naming collision that will otherwise cause real confusion. The codebase uses the word "task" for three distinct, unrelated concepts.
A Todo task (T1) is a structured planning ledger: it lives in the task database, carries a dependency DAG, and is how the model tracks what it intends to do. It is not running anything — it is a list with state. A Process task (T2) is a running background OS process or long-running agent, identified by an opaque handle like wd1re9un1. It is a process registry entry, not a logical unit of work. A Workflow agent task (T3) is a subagent node inside a deterministic orchestration graph — a durable unit of computation whose identity is derived from its structural position in a script. The DSL primitives (agent(), parallel(), pipeline()) live only in T3. T1 and T2 share nothing with T3 except vocabulary.
The distinction matters because the three live in different stores, serve different purposes, and have entirely different durability models. T1 is written and read by the model as part of planning. T2 is managed by the process supervisor. T3 is managed by the workflow executor and the Coordinator. When you see "task" in a stack trace or a table name, the first question is always: which kind?
A workflow script is not an invocation of a framework. It is a plain async Python module body that runs against a set of ambient globals injected by the runtime at execution time. The DSL surface is deliberately small:
async def agent(prompt: str, *, label: str | None = None, model: str | None = None,
effort: str | None = None, schema: dict | None = None) -> Any: ...
async def parallel(thunks: list[Callable[[], Awaitable[Any]]]) -> list[Any | None]: ... # BARRIER: raise → None element
async def pipeline(items: list[Any], *stages: Callable) -> list[Any]: ... # NO barrier: per-item streaming
def phase(title: str) -> None: ...
def log(message: str) -> None: ...The shape of this surface was derived empirically from 94 real corpus scripts — not designed upfront from a spec. The first structural fact the corpus reveals: there is no async def run(ctx): ... wrapper. Zero of 94 scripts use it. The primitives are free globals, not ctx.agent(). The required first statement is meta = {'name': ..., 'description': ..., 'phases': [...]} as a pure dict literal; after that, the file is a bare module body with top-level await and an early return. This means the shape of a workflow is closer to a configuration file with executable steps than to a framework callback.
The design choice to use ambient globals rather than a context object is not accidental. It keeps the script body structurally simple — no destructuring, no dependency injection, no constructor — and it keeps the structural position of each agent() call unambiguous. When agent() is a free global, its call site is exactly where it appears in the source; it cannot be wrapped, deferred, or aliased in a way that confuses the structural identity machinery. You will see why that matters in the next section.
The central problem in a durable fan-out graph is this: when the process crashes and a new attempt resumes, how does the executor know which nodes have already completed and which have not? The naive answer — assign random ids at call time — fails on resume: a new execution generates new random ids, which don't match the ids stored in the durable record, so the executor cannot tell "this node was already settled" from "this is a new node." The correct answer is to derive each node's id from its structural position in the execution tree, not from when it happens to run.
The runtime maintains a WorkflowScope per run, carried through contextvars.ContextVar:
import contextvars
from dataclasses import dataclass
class Counter:
def __init__(self) -> None:
self.n = 0
@dataclass(frozen=True)
class WorkflowScope:
workflow_name: str
script_version: str
run_id: str
coordinate_path: tuple[str, ...]
agent_counter: Counter
pipeline_counter: Counter
depth: int
_workflow_scope: contextvars.ContextVar[WorkflowScope] = contextvars.ContextVar('workflow_scope')frozen=True prevents reassigning the scope's fields, but the Counter instances are mutable by design — scope.agent_counter.n += 1 is intentional, because the deterministic-ID scheme depends on each branch accumulating a count. On fan-out, each parallel branch gets a fresh Counter so the IDs never collide across siblings.
contextvars.ContextVar propagates automatically into asyncio.create_task() — each task inherits a copy of the current context at the moment it is created. This is a structural improvement over AsyncLocalStorage in the Node.js runtime: in-process fan-out via asyncio.gather or create_task requires no explicit bridging. Cross-process boundaries (the subprocess isolation approach covered in Chapter 11) do not propagate context — explicit token passing is required there, and the mechanism is straightforward: serialize the scope fields into the subprocess invocation arguments.
The coordinate_path is a tuple of position strings accumulated as the executor descends into parallel branches and pipeline items. An agent() call at the top level of the script gets the path () plus a within-scope counter: agent-0001. An agent() call inside the second branch of a parallel, inside the first pipeline item, gets a path like ('pipe-1-item-0', 'agent-0001'). The full structural dispatch id is then:
wf:{workflow_name}@{script_version}:{run_id}:{coordinate_path/.../leaf}
A concrete example shows the full shape:
wf:deep-research@sha256:3a9f1c2b4d7e8f01:k4j2r8z:agent-0001
wf:deep-research@sha256:3a9f1c2b4d7e8f01:k4j2r8z:parallel-branch-0001/agent-0001
wf:deep-research@sha256:3a9f1c2b4d7e8f01:k4j2r8z:parallel-branch-0002/agent-0001
wf:deep-research@sha256:3a9f1c2b4d7e8f01:k4j2r8z:pipe-1-item-0-stage-s0/agent-0001
The id is computed SYNCHRONOUSLY at the call site — before any await — so it is fixed regardless of which branch settles first. This is the critical Step-3a requirement: a node's id must be stable across crash and resume. The ASCII picture makes the derivation concrete:
withWorkflowScope("deep-research", "sha256:3a9f...", "k4j2r8z")
│
├── agent() → path=[], counter=1
│ id = wf:deep-research@sha256:3a9f...:k4j2r8z:agent-0001
│
└── parallel([thunkA, thunkB])
│
├── withCoordinate("parallel-branch-0001") ← fresh agent_counter
│ └── agent() → counter=1
│ id = wf:...:parallel-branch-0001/agent-0001
│
└── withCoordinate("parallel-branch-0002") ← fresh agent_counter
└── agent() → counter=1
id = wf:...:parallel-branch-0002/agent-0001
Notice that with_coordinate resets the agent_counter to zero for the nested scope. This means the first agent() inside any branch is always agent-0001, independent of what other branches do. Sibling branches do not share counter state. This is what makes the id a pure function of the execution structure — you can replay the script and re-derive every id without any external state.
Both script_version and run_id are folded into the id, and both are load-bearing. The script_version is sha256:{first 16 hex chars of the source hash}. If the script source changes, the version changes, and every id in the new run occupies a disjoint namespace from the old run's settled nodes. A crashed run that resumes after a script change never accidentally matches its old nodes — the ids simply don't overlap. The run_id is a djb2 hash of the run anchor (name + args hash), so two runs of the same script with different arguments also get disjoint namespaces. Same-args replay reuses the same run_id, making resume idempotent: re-reaching a node that already settled is a no-op, because the id matches and the Coordinator short-circuits to the frozen result.
_BASE36 = "0123456789abcdefghijklmnopqrstuvwxyz"
def stable_run_id(anchor: str) -> str:
h = 5381
for ch in anchor:
h = ((h << 5) + h + ord(ch)) & 0xFFFFFFFF
digits: list[str] = []
while h:
digits.append(_BASE36[h % 36])
h //= 36
return "".join(reversed(digits)) or "0"load_workflow_source separates parsing from execution. At load time, the source is parsed, hashed, and validated. Then an eager compile-check runs against a throwaway subprocess — Python's compile() builtin run in a fresh process with no access to any durable state:
# EAGER compile-check: compile the WRAPPED source in a throwaway subprocess.
# The loader wraps the body in `async def __workflow_main__(): ...` so that
# bare `return` and top-level `await` are both valid Python.
import subprocess
import sys
import textwrap
SANDBOX_ENV = {'PYTHONHASHSEED': '0', 'LC_ALL': 'C', 'PYTHONIOENCODING': 'utf-8'}
try:
wrapped = f"async def __workflow_main__():\n" + textwrap.indent(
open(source_path).read(), " "
)
result = subprocess.run(
[sys.executable, "-c", f"compile({wrapped!r}, {source_path!r}, 'exec')"],
capture_output=True, text=True, timeout=COMPILE_CHECK_TIMEOUT_S,
env=SANDBOX_ENV,
)
if result.returncode != 0:
raise WorkflowSandboxError(
f"[workflow] workflow body failed to compile: {result.stderr}"
)
except subprocess.TimeoutExpired as exc:
raise WorkflowSandboxError("[workflow] compile-check timed out") from excThe loader wraps the body in async def __workflow_main__() before compilation, so both bare return and top-level await are valid Python — no special compiler flags needed. The compile-check must compile the wrapped source, not the raw body, or a valid workflow with return fails at load time. cmd[0] must be sys.executable (an absolute path), not a bare "python" — a stripped env has no PATH, so a bare name would either raise FileNotFoundError or resolve to the system Python instead of the virtualenv interpreter that loaded the module.
The word "eager" here carries weight. A syntax error in the script body is caught at load time — before any agent() calls have been dispatched, before any subagents have been spun up, before any durable state has been written. The failure is clean and early. Without this check, a script with a typo on line 200 might run successfully through lines 1–199 (dispatching subagents, writing records) before dying at the syntax error. Recovery would then be trying to resume a run whose control flow was never valid — a confusing and unrecoverable situation.
The WorkflowDef that emerges from loading has the shape WorkflowDef(name, script_version, run). At run time, the executor spawns a fresh subprocess, injects the DSL globals as module-level names, and calls run. When the run completes, the process exits. There is no shared state between runs — not even between two concurrent runs of the same WorkflowDef. Each run gets its own interpreter heap, its own scope, its own counter state. This is what makes the fresh-process contract meaningful: you cannot accidentally share a global variable between runs, because there is no shared process.
The subprocess is also spawned with env={'PYTHONHASHSEED': '0'} — a detail whose importance is easy to underestimate. CPython randomizes hash seeds per process by default, which means set and frozenset iteration order vary across runs. (Dict iteration is insertion-ordered since Python 3.7 and is unaffected by the hash seed.) A workflow body that iterates a set of agent names and dispatches them in iteration order would produce different coordinate paths on different runs — breaking the deterministic replay that makes resume safe. Pinning the hash seed to a fixed value makes iteration order deterministic across process boundaries.
The next chapter covers what the subprocess boundary actually enforces: it is not just a performance boundary but a security and determinism boundary. Workflow bodies are written by the model, which means they are untrusted code. The subprocess runs with a restricted import set — subprocess spawning, filesystem access, and the host environment are blocked by providing a constrained sys.modules and a stripped __builtins__. The compile-check uses a throwaway subprocess for the same reason: it should never be able to affect any durable state.
parallel looks like a fan-out helper but it is a coordination primitive with specific, load-bearing semantics. It takes a list of zero-argument callables that return awaitables, launches all of them concurrently, waits for all to settle (a barrier), and maps raised exceptions to None elements rather than propagating them. The result preserves the order of the input array regardless of completion order.
The contrast with raw async fan-out is sharp. If you write:
a, b, c = await asyncio.gather(agent_a(), agent_b(), agent_c())a raise from agent_b propagates as an exception out of asyncio.gather. Your caller has to catch it or the run terminates. parallel internalizes the error boundary:
a, b, c = await parallel([lambda: agent_a(), lambda: agent_b(), lambda: agent_c()])
# b is None if agent_b raised; a and c are their resultsThe caller gets the results it can use and None for the ones it cannot, without the whole block failing. This design keeps the orchestration graph from cascading: one subagent dying does not abort the parallel block. The caller decides what to do with the None slot — retry, skip, log, fail at a higher level.
The pervasive [r for r in results if r is not None] pattern in the 94 corpus scripts is the ergonomic consequence of this contract. Scripts that call parallel almost universally follow it with a filter to drop None elements before processing results. This is not a smell — it is the correct response to a system that chose partial failure over cascading failure. The trade-off is that you must be disciplined about checking for None: a caller that ignores None results will silently process incomplete data. The system can tell you a node failed; it cannot make you care.
pipeline processes a list of items through a sequence of stages, streaming each item through independently. Where parallel has a barrier — all thunks must settle before the caller gets results — pipeline has no barrier between items. Each item moves through the stages on its own, and the result of one item's stage does not block another item's stage from starting.
parallel vs pipeline
parallel([a, b, c]) pipeline([x, y, z], stage1, stage2)
┌────────────────────────┐ ┌──────────────────────────────┐
│ │ │ x ──▶ stage1 ──▶ stage2 │
│ a() ─────▶ result_a │ │ y ──▶ stage1 ──▶ stage2 │
│ b() ─────▶ result_b │ │ z ──▶ stage1 ──▶ stage2 │
│ c() ─────▶ None (err) │ │ │
│ ▼ │ │ No barrier between stages — │
│ ┌─ BARRIER ─┐ │ │ x can be in stage2 while │
│ │ wait ALL │ │ │ z is still in stage1 │
│ └───────────┘ │ │ │
│ returns [a, b, None] │ │ Wall-clock = slowest item, │
│ │ │ not slowest stage │
└────────────────────────┘ └──────────────────────────────┘
Use parallel when you need Use pipeline when items are
ALL results before moving on independent and stages chain
The stages are passed variadically as bare functions, not as a list. The signature (prev, item, index) threads the prior stage's output as prev and the original list element as item, so stages can both transform the result and recover per-item context from the original data. A stage that returns None drops that item from the downstream result, which is consumed via the same [r for r in results if r is not None] pattern.
The choice between parallel and pipeline is a design statement about what kind of failure you can tolerate. parallel groups thunks that are conceptually independent — each thunk is a complete unit of work, and None in the result means "this unit failed." pipeline groups items that move through a transformation — None in the result means "this item was dropped." In the corpus, parallel is used when you want all-or-nothing per node (audit an entity, research a claim), while pipeline is used when you want per-item progress (enrich a list, review a batch). The wrong choice does not break correctness but it produces confusing diagnostic signals — a None element in a pipeline result means something different than a None element in a parallel result.
The DSL type file declares a budget global with total, spent(), and remaining(). In 94 corpus scripts, zero scripts call budget.spent() or budget.remaining() — every budget grep hit is prompt-string text or a comment. This is not an oversight; there is a structural reason the primitive is unexercised.
spent() returns a value that varies by model call latency and prompt size. If a script uses budget.spent() to decide whether to continue a loop, the script's control flow becomes environment-dependent: the same script with the same inputs might take a different branch depending on how fast the model responded to the first batch. That violates the determinism contract that makes resume safe. A replay of the same run must walk the identical graph and re-derive the identical ids. If the branch condition depends on elapsed cost, replay can diverge from the original run, producing a graph with different nodes — which means the Coordinator would re-dispatch nodes that already settled, or skip nodes that should resume.
Scripts that need size or cost ceilings hand-roll them with plain Python. MAX_FETCH, a manual fetch_slots counter, [:N] for prompt truncation — these are all deterministic in the replay sense: they depend only on the input data and the script's constants, not on runtime state. The budget field exists in the type signature and the globals interface; the runtime will wire enforcement at some point. But until the enforcement mechanism is itself deterministic — meaning it produces the same decision on replay that it produced on the original run — any script that relied on it would be unsafe to resume.
The workflow() primitive allows one workflow to call another inline. Two distinct depth limits govern the execution.
The first is the inter-workflow nesting limit: a workflow may call another workflow, but only one level deep — workflow() inside a nested workflow throws. Inter-workflow recovery and result propagation across the boundary are not yet corpus-validated, so the one-level cap is pragmatic rather than fundamental.
The second is MAX_WORKFLOW_DEPTH = 8, which bounds the agent/fan-out depth within a single workflow's coordinate path. This limits how deeply parallel and pipeline can nest agents within one workflow, not how many workflows can call each other.
In the current corpus, zero scripts call workflow() inline. The primitive exists in the type signature, confirmed by doc comments in the source, but it is unexercised. This is a recurring pattern in the DSL: the shape is defined before the use cases exist. workflow() appears in deep-research.py as a header comment documenting the form Workflow(name='deep-research', args='<question>') — but that is a comment describing the primitive to the model, not a call from the script itself.
The practical implication: if you are building a workflow that needs to invoke a sub-workflow, the mechanism exists but has no production-validated semantics yet. The coordinate path machinery supports it — with_coordinate can push a workflow-name segment onto the path before entering the child workflow's scope — but the executor's handling of depth errors, recovery across the workflow boundary, and inter-workflow result propagation are all in the "structural shape exists, behavior unconfirmed by corpus" category.
The workflow system is how the runtime escapes single-agent limitations without abandoning the durability guarantees that Chapters 6 and 7 established for single turns. Control flow is code — deterministic, replayable, testable — not model-driven improvisation. The coordinate path gives every agent() call a stable id derived from its structural position in the script, not from when it happens to run. The eager compile-check catches structural errors before any durable state is written. The fresh-process-per-run guarantee means no shared state leaks between runs. And the parallel barrier keeps node failures local rather than propagating them upward.
What remains open is the budget enforcement (the primitive fights determinism until spent() can be made replay-safe) and the async fan-out shape. Today, agent() is await-synchronous: when you call agent(), you hand control to the subagent and wait. There is no way to detach a subagent, get a handle back, do other work, and rejoin later. The coordinate path can support that shape — the id is derived at call time, before the first await — but the executor, the Coordinator, and the resume machinery would all need to be reshaped around async handles rather than sequential fan-out. When that reshape comes, it will be the largest single change to the workflow layer since its design.
The next chapter addresses the other open question this chapter raised: what exactly does the subprocess boundary enforce, and why is a same-process exec() not sufficient for untrusted workflow bodies?
Files read for this chapter:
apps/runtime/src/workflow/scope.ts— theWorkflowScopeinterface,with_coordinate,dispatchIdFor, andstableRunIdimplementations; the canonical source for the coordinate-path and structural-id machinery.apps/runtime/src/workflow/dsl.d.ts— the empirically derived DSL type surface (94 corpus scripts);AgentFn,ParallelFn,PipelineFn,WorkflowBudget, andWorkflowGlobals; the comments carry the corpus findings that drive design decisions.apps/runtime/src/workflow/sandbox.ts— the eager compile-check implementation,WorkflowSandboxError, and the full sandbox design rationale including the isolation primitive and determinism enforcement.