Every workflow script the model authors is a potential attack surface. Chapter 10 introduced workflow scripts as agent-authored code: a string of Python that the runtime compiles and executes on each run. That script can contain arbitrary logic. It can loop forever. It can try to allocate a gigabyte of memory. If it finds a path to the host process, it can read the environment variables where the LiteLLM API keys live. Running that code in the same process as the runtime — even behind restricted builtins — is not sufficient without process-level isolation.
But isolation has costs. Every boundary you draw is a round trip, a serialization format, and a new class of failure mode. The question is not whether to isolate; it is which threats you are isolating against, what you are willing to pay, and where you draw the line. Chapter 9 drew one isolation boundary around tool execution — the SandboxApi seam that puts filesystem access on a separate Fly machine per tenant. This chapter draws a second, orthogonal boundary — between the workflow script and the runtime's own host process — and traces the mechanisms that make it work: a DoS budget, a scope bridge, and Python's native context propagation. The two boundaries defend against different threats. Neither is sufficient without the other.
The runtime draws two separate isolation boundaries, and conflating them produces confusion about what each one is actually defending.
The first boundary is code isolation: workflow scripts execute inside a sandboxed subprocess. This boundary defends against agent-authored code. Scripts the model wrote can contain infinite loops, while True hangs, memory bombs, and host-escape attempts. The subprocess buys two things: the script cannot read or write the host process's memory, and the runtime can cap its resources with setrlimit. It does not make the script's namespace safe — exec inside the subprocess still runs against a full CPython interpreter, and a determined script can reach the stdlib through any live function's __globals__ or through getattr with a constructed dunder name. Restricted builtins and AST validation raise the cost of that traversal; they are defense-in-depth, not a hermetic seal. The real containment is the OS: the subprocess runs with a sanitized environment (no inherited API keys, deterministic hash seed), and the targeted hardening — dropped privileges, a seccomp-bpf syscall filter, and a network namespace — would ensure that even a full escape reaches nothing of value. Today the primary containment is the sanitized environment; the OS-level hardening is the planned next layer.
The second boundary is sandbox isolation: tool execution — filesystem reads, shell commands, process spawns — runs on a separate Fly machine per tenant. This boundary defends against side effects. Even a fully contained subprocess calling agent() needs to reach a shell and a filesystem. That reach goes through the SandboxApi seam to the Fly machine, not to the host filesystem. Chapter 9 traced how SandboxApi abstracts this surface; this chapter is concerned with the code isolation layer that sits above it.
workflow script (agent-authored)
│
[subprocess] ← code isolation (memory isolation + sanitized env + defense-in-depth builtins)
│ __import__ blocked; os/sys reachable only via escape
DSL globals (agent, parallel, pipeline)
(bridged via JSON over stdin/stdout)
│
SandboxApi seam
│
[Fly machine] ← sandbox isolation (per-tenant VM)
│
/workspace (shell, filesystem)
Neither boundary alone is sufficient. Code isolation without sandbox isolation lets the script reach the host filesystem through DSL calls — agent() calls execute_process, which calls exec, which spawns a process on the host. Sandbox isolation without code isolation lets the script run infinite loops or allocate unbounded memory directly in the runtime's host process. You need both.
The workflow sandbox spawns a fresh subprocess for every run() call. The subprocess's lifecycle maps exactly to the run:
@dataclass
class SandboxProcess:
process: subprocess.Popen
guard: RunGuard
@classmethod
def spawn(cls, cmd: list[str], guard: RunGuard) -> 'SandboxProcess':
return cls(
process=subprocess.Popen(
cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
env={'PYTHONHASHSEED': '0', 'LC_ALL': 'C', 'PYTHONIOENCODING': 'utf-8'},
),
guard=guard,
)
def dispose(self) -> None:
self.process.terminate()
self.process.wait()The env dict is minimal and explicit: PYTHONHASHSEED pins set iteration order, LC_ALL pins locale-dependent string operations, and PYTHONIOENCODING ensures consistent IO encoding. No PATH is inherited, so cmd[0] must be an absolute path — callers pass sys.executable to guarantee the same interpreter that loaded the module. A fresh subprocess is created at the start of each run(). When the run completes — success, raised exception, or abort — the subprocess is terminated and joined. This is not a pool. There is no reuse. The cost of fresh construction is accepted in exchange for a guarantee: two runs of the same WorkflowDef cannot share memory, closures, or global state. If a buggy script corrupts its own namespace, the damage ends when the subprocess exits.
Before the script body executes, a restricted execution environment is set up. This is the sandbox setup phase: it installs the DSL globals, enforces determinism restrictions, and blocks forbidden constructs. Two mechanisms work in combination. First, AST validation runs before execution: the script source is parsed with ast.parse and a custom NodeVisitor rejects dangerous nodes — Import, ImportFrom, Global, and attribute access on known forbidden paths. Second, the subprocess itself starts with a restricted __builtins__ dict that replaces __import__ with a function that always raises:
RESTRICTED_BUILTINS = {k: v for k, v in vars(builtins).items()
if k not in ('__import__', 'open', 'exec', 'eval', 'compile')}
def _block_import(*a, **kw):
raise ImportError("imports are not allowed in workflow scripts")
RESTRICTED_BUILTINS['__import__'] = _block_importThe script body is written to a temp file and executed via exec(source, {'__builtins__': RESTRICTED_BUILTINS, **DSL_GLOBALS}) inside the subprocess. Top-level await is handled by wrapping in an async function and running with asyncio.run. The host process tracks the subprocess via IPC over stdin/stdout; when the subprocess exits, the run is complete.
The runtime has a guarantee to uphold that goes beyond security: deterministic replay. Chapter 7 traced how the executor can resume a run from a checkpoint by replaying the structural IDs. For replay to work, a re-executed script must produce the same sequence of DSL calls as the original. A script that calls random.random() for a branch condition, or datetime.datetime.now() to construct a prompt, breaks this guarantee.
The sandbox enforces determinism by replacing the nondeterminism APIs in the execution namespace with stubs that raise:
def _blocked(name: str) -> None:
raise RuntimeError(
f"{name} is not allowed in workflow scripts — "
"workflow execution must be deterministic"
)
DETERMINISM_SHADOWS = {
'datetime': type('datetime', (), {
'datetime': type('datetime', (), {
'now': staticmethod(lambda *a: _blocked('datetime.datetime.now()')),
'utcnow': staticmethod(lambda *a: _blocked('datetime.datetime.utcnow()')),
})
})(),
'random': type('random', (), {
'random': staticmethod(lambda: _blocked('random.random()')),
'choice': staticmethod(lambda seq: _blocked('random.choice()')),
'randint': staticmethod(lambda a, b: _blocked('random.randint()')),
})(),
'time': type('time', (), {
'time': staticmethod(lambda: _blocked('time.time()')),
'sleep': staticmethod(lambda s: _blocked('time.sleep()')),
})(),
}os.urandom is blocked at the __import__ level — the restricted namespace simply has no os. uuid.uuid4() is similarly blocked. The AST validator additionally rejects any use of import random, import datetime, or from time import * at the source level, so the script cannot bypass the namespace restrictions by importing fresh copies. The same __globals__ path that can reach os for a security escape can reach the real time, random, or datetime for a determinism escape. Determinism enforcement, like the security boundary, is defense-in-depth — it raises the bar against accidental nondeterminism, not deliberate circumvention.
The compile() built-in is removed from RESTRICTED_BUILTINS. This closes the determinism-bypass path where a clever script compiles new Python at runtime to reach shadow-free behavior. eval and exec are similarly removed. The AST validator catches ast.Attribute accesses targeting __class__, __bases__, or __subclasses__ — the Python equivalent of constructor-chain host escapes. (In Python 3, exec is a builtin function, not a statement — there is no ast.Exec node. The sandbox blocks exec by removing it from RESTRICTED_BUILTINS; the AST validator optionally rejects ast.Call forms whose callee resolves to the exec name.)
Security isolation stops the script from escaping the process. DoS isolation stops the script from consuming the process. The RunGuard is the mechanism:
@dataclass
class RunGuard:
node_count: int = 0
aborted: bool = False
abort_reason: str = ""Defense-in-Depth: Four Layers
Layer 1: AST Validation (before execution)
┌─────────────────────────────────────────────────┐
│ Rejects: Import, ImportFrom, Global, │
│ __class__/__bases__/__subclasses__ │
│ Bypassable: string concatenation, __globals__ │
└──────────────────────┬──────────────────────────┘
▼
Layer 2: Restricted Builtins (at execution)
┌─────────────────────────────────────────────────┐
│ Removed: __import__, open, exec, eval, compile │
│ Shadows: datetime, random, time (raise on call) │
│ Bypassable: __globals__ on any DSL function │
└──────────────────────┬──────────────────────────┘
▼
Layer 3: Sanitized Environment (at spawn)
┌─────────────────────────────────────────────────┐
│ env = {PYTHONHASHSEED, LC_ALL, PYTHONIOENCODING}│
│ No API keys, no PATH, no inherited secrets │
│ cmd[0] = sys.executable (absolute path) │
└──────────────────────┬──────────────────────────┘
▼
Layer 4: OS Hardening (targeted, not yet deployed)
┌─────────────────────────────────────────────────┐
│ Planned: seccomp-bpf, network namespace, │
│ dropped privileges │
│ Effect: even a full escape reaches nothing │
└─────────────────────────────────────────────────┘
Each layer raises the cost of escape.
The runtime relies on Layer 3 today.
Layers 1-2 are defense-in-depth, not hermetic.
Three independent budgets defend against three independent resource-abuse patterns. The first is a wall-clock timeout: the subprocess is run under signal.alarm (Unix) or a watchdog thread, and terminated if it exceeds the budget (default 5 seconds). A synchronous while True consumes CPU continuously; the watchdog detects this and sends SIGTERM. Five seconds is a 50× margin over typical orchestration — synchronous DSL dispatch takes single-digit milliseconds — so the budget catches genuine hangs without firing on legitimate computation.
The second budget is the node count (max_nodes, default 10,000): every agent() dispatch increments guard.node_count. If the count exceeds the limit, the run aborts:
guard.node_count += 1
if guard.node_count > limits.max_nodes:
reason = (
f"[workflow] workflow exceeded its node budget "
f"({limits.max_nodes} agent dispatches) — possible unbounded loop."
)
abort(reason)
return json.dumps({"ok": False, "error": reason})The third budget is a memory limit, enforced via resource.setrlimit(resource.RLIMIT_AS, (limit_bytes, limit_bytes)) in the subprocess before the script runs. A script that allocates one large list per iteration burns no measurable CPU and dispatches zero agent nodes — it slips past both other budgets — but trips the memory cap and receives MemoryError, which the sandbox catches and records as a clean diagnostic.
The design insight here is that wall-clock timeout is the wrong primitive for workflows. A workflow waiting on 50 parallel agent turns may legitimately take 10 minutes of wall clock. You do not want to kill it. What you want to kill is a while True: await agent('...') loop — a hang that consumes zero subprocess CPU (each iteration suspends at await), so the CPU watchdog never fires. The node count catches it: after 10,000 dispatches, the run aborts with a recorded diagnostic. The run() caller surfaces that diagnostic as a clean error rather than a raw subprocess.TimeoutExpired.
Chapter 10 traced how the executor uses contextvars.ContextVar to carry a WorkflowScope implicitly through every async call chain. Context variables are how the structural ID system knows where in the workflow tree a given agent() call lives. The challenge: context variables do not propagate across a subprocess boundary. When the subprocess calls back to the host via IPC, it exits one Python interpreter and communicates with another. The context from the run's with_workflow_scope frame is in the host process, not the subprocess.
The solution is a token-keyed scope registry. Before the run starts, the executor's active WorkflowScope is captured and stored under a fixed root token ("r"). The subprocess is initialized with that root token as its current scope context. When the subprocess sends a request over the IPC channel with {"op": "agent", "ctx": token, "payload": {...}}, the host looks up the token, re-enters the executor's context for that scope, and dispatches the agent call inside it.
class ScopeRegistry:
def __init__(self) -> None:
self._scopes: dict[str, WorkflowScope] = {}
self._seq = 0
def set_root(self, scope: WorkflowScope) -> None:
self._scopes["r"] = scope
def mint(self) -> str:
scope = get_workflow_scope()
token = f"s{self._seq}"
self._seq += 1
self._scopes[token] = scope
return token
def get(self, token: str) -> WorkflowScope:
scope = self._scopes.get(token)
if scope is None:
raise WorkflowSandboxError(f'scope bridge: unknown token "{token}"')
return scopemint() is called synchronously inside the executor's with_coordinate callback — at the moment when the child scope is active. It captures that scope under a fresh token and returns the token to the subprocess. The subprocess then uses that token for any IPC calls made inside that branch. The structural ID system on the host side runs exactly as it does for a native workflow: it sees with_coordinate frames in the right nesting order, derives deterministic IDs, and the fact that the calls crossed a process boundary is invisible to it.
The flow for a parallel block looks like this:
subprocess (Python) host (Python)
─────────────────────── ──────────────────────────
parallel([...]) call ─────────► {"op": "parallel", "ctx": parent_token, "count": N}
registry.get(parent_token) → parent scope
dsl.parallel(thunks)
with_coordinate('par-0', ...) {
child_token = registry.mint() ← captures branch scope
◄─ {"child_ctx": child_token}
agent("prompt") call ◄───────── subprocess sets ctx = child_token
─────────► {"op": "agent", "ctx": child_token, ...}
registry.get(child_token) → branch scope
run_in_workflow_scope(scope, lambda: dsl.agent(...))
◄───────── {"ok": true, "value": result}
Inside the subprocess, after a native await resumes, which scope token is current? In the original TypeScript implementation, this was a problem: V8 has no built-in analog of contextvars, so the implementation had to manually snapshot and restore __ctx via a context-preserving thenable — an object whose .then method captures the current token synchronously and restores it before the continuation fires.
Python's contextvars solves this natively. When asyncio.create_task() is called, it automatically copies the current Context snapshot into the new task. When a branch suspends at await and another branch resumes, each branch retains its own context variables without any manual snapshot or restore:
async def run_branch(scope_token: str) -> Any:
_current_scope_token.set(scope_token)
return await agent_call(...)
# asyncio.create_task copies the current Context — no manual thenable needed
task_a = asyncio.create_task(run_branch("s1"))
task_b = asyncio.create_task(run_branch("s2"))Branch A sets _current_scope_token to "s1" and task creation snapshots that context. Branch B sets it to "s2" in its own context copy. When branch A's continuation resumes after an await, it reads "s1" — not "s2", even if branch B ran in the interim. This is a genuine advantage of Python's async model: the context propagation is provided by the runtime rather than requiring manual threading through every then handler.
Cross-process scope — communicating the token from subprocess to host — still requires explicit IPC (JSON over stdin/stdout), the same concept as the V8 bridge's JSON serialization. What Python eliminates is the intra-process snapshot-restore pattern that the TypeScript implementation needed.
Compare the code isolation layer with the sandbox isolation layer from Chapter 9. The subprocess boundary is about what the script can do: it cannot import arbitrary modules, cannot loop forever, cannot use nondeterminism. The Fly machine boundary is about what the agent can do: it can read and write files, but only in /workspace, and only on the tenant's machine, not the host.
FlySandboxEnv implements this by routing every filesystem operation through a shell command via ControlServerClient:
readFile(path) → cat '<path>'
writeFile(path) → printf %s '<b64>' | base64 -d > '<path>'
stat(path) → stat '<path>'
readdir(path) → ls -1A '<path>'
There is no direct filesystem RPC. Every op is a shell one-liner. The cost is round-trip latency per call — each file read is one exec to the Fly machine and back. The gain is a minimal server-side surface: the control server only needs to implement exec. Adding a native readdir RPC would cut latency for directory listings but would complicate the server contract. The current design accepts the latency.
The Fly machine owns a tenant-shared /workspace volume. The runtime then scopes each session to /workspace/sessions/<stable segment> via create_scoped_session_env and per_session_sandbox_cwd, so normal tool and file operations are per-session even though the backend connection and volume are per-tenant. The session_id rides each exec as a trace tag in the environment. Isolation at the Fly level is per-tenant; isolation at the runtime level is per-session through the SessionEnv wrapper that Chapter 9 described.
In Chapter 9, the trust boundary was between the Coordinator and the sandbox: the SandboxApi seam. Crossing it was intentional and explicit — the tool called exec, the sandbox ran the command. In this chapter, the trust boundary is between the workflow script and its own DSL. The script calls agent('prompt'), and that call silently crosses a process boundary.
The cost is that all data must be serialized, not passed by reference. Arguments are serialized when they leave the subprocess; results are serialized when they return. Functions cannot cross the boundary. You cannot pass a callback to agent(). The DSL is deliberately value-typed: prompts are strings, results are JSON-serializable objects. This keeps the serialization cost bounded — a prompt is a few hundred bytes, a result is a JSON object — but it constrains what the DSL can express.
The schema option on agent() is the runtime's concession to this constraint. Instead of passing a validation function, you pass a JSON Schema; the executor applies it on the host side before returning the result to the subprocess. This moves expressiveness from the script into the schema language — which is serializable — and keeps the boundary clean.
As the workflow system grows, this serialization cost will push the design toward coarser-grained primitives. The current DSL is fine-grained: agent() for a single call, parallel() for a fan-out, pipeline() for a map across stages. As workflows grow more complex, the temptation will be to add richer in-subprocess computation — filtering, transforming, conditional routing — that reduces the number of cross-boundary calls. That evolution is not a bug; it is the natural pressure of having drawn a boundary with a real cost attached.
A second honest gap is the language-level containment. The AST denylist and restricted builtins raise the cost of namespace escape but do not eliminate it. A script needs nothing more than a plain getattr((), '__class__') — getattr is not in the removed set, and the AST validator only inspects ast.Attribute nodes, not getattr calls — to reach CPython's full class hierarchy through any live object. It can reach os through a DSL function's __globals__. The defense the runtime actually relies on is not the denylist but the OS layer: the subprocess runs with an empty environment (no inherited API keys, deterministic hash seed). The targeted next layer — dropped privileges, a seccomp-bpf syscall filter, and a network namespace — would ensure that a script that escapes its restricted namespace reaches a process with no credentials and no network. Today the sanitized environment is the primary containment; a script that bypasses the denylist can still make syscalls and reach the network. The escape is real, and the blast radius is bounded by the sanitized env today, with OS hardening as the planned seal.
The sandbox design reveals the runtime's actual threat model. It is not abstract: it is specifically about agent-authored scripts with unbounded fan-out running inside a production process that holds API keys in its environment. The two-layer response addresses different parts of that threat. The subprocess sandbox keeps agent-authored code from touching the host process memory. The Fly machine keeps agent side effects from touching the host filesystem. The node-count DoS guard catches resource abuse that the CPU watchdog misses. The scope bridge makes the process boundary transparent to the executor's structural ID system. Python's contextvars makes it transparent to the script author — without the manual thenable machinery the TypeScript version required.
What the design cannot make transparent is the serialization cost. Every call from script-space to DSL-space is a crossing. That is the price of the boundary, and it is deliberate. Chapter 12 extends this model to the fleet level: what happens when the Fly machine is itself a distributed system, and the sandbox isolation boundary becomes a network boundary between machines.
apps/runtime/src/workflow/sandbox.ts— subprocess sandbox lifecycle (SandboxProcess), restricted builtins, determinism enforcement,ScopeRegistry,RunGuard, DoS abort path, JSON IPC bridgeapps/runtime/src/sandbox/fly/control-sandbox.ts—FlySandboxEnv: exec-only Fly machine filesystem adapter, shell command mapping for eachSandboxApimethoddocs/the-rewrite/chapters/09-the-sandbox-boundary.md—SandboxApicontract,SessionEnv, local vs. remote sandbox implementations (context for comparison in §7)docs/the-rewrite/chapters/10-workflows.md— workflow DSL introduction, agent-authored scripts, structural ID system (context for §1 and §5)