The journal from Chapter 6 answers one question precisely: where did the turn stop? It records a phase, a stream key, a tool request, a transcript leaf pointer. What it does not do is decide what happens next. That is the question this chapter answers.
When a turn resumes after a crash, the system faces a branching problem: the submission's journal says how far the turn got, but "how far" admits multiple interpretations. Did the model finish streaming? Were tool calls partially executed? Were some results already committed to the transcript and some not? Each combination demands a different recovery strategy — and the wrong strategy either re-runs a side-effecting tool or skips a step the model still needs. classify_submission_state is the pure function that reads the durable record and names which case you are in. This chapter is about that classifier and what each classification requires.
This is also the chapter that reveals why the journal was worth building. A classifier that reads a well-structured durable record can be a pure function. A pure function can be tested with a table of inputs. A testable classifier is a correct one. The design decision in Chapter 6 — externalizing phase into the store — is what makes recovery auditable, not just operational.
A naive approach to crash recovery is full replay: restore the last known state and re-run from the beginning. This works for pure computations. A sorting algorithm replayed from the same input produces the same sorted output. A statistical model replayed from the same training data converges to the same weights. Replay is safe when the computation is free of side effects.
Agent turns are not free of side effects. An agent turn calls tools — external actions that touch the world. Re-running a turn that already dispatched a tool call re-dispatches it. Re-running a turn that already sent an email sends it again. Re-running a turn that already created a Jira ticket creates a duplicate. Replay is not just wrong here; it's actively harmful. The model had no way to know the first attempt had already fired.
Recovery has to be continuation from a specific point. Which means the system must first answer a harder question: which point? The transcript might show a completed assistant message. It might show an assistant message followed by some tool results but not others. It might show nothing at all after the input — the model call never started. Each of these situations demands a different action: settle the turn, synthesize the missing results, or start fresh. The classifier's job is to look at what the store actually contains and name the situation precisely, so the recovery path can proceed without guessing.
classify_submission_state takes two arguments: the active-path history entries that follow a persisted submission input (list[SessionEntry] | None), and the current model's context window size for overflow detection. It returns a SubmissionState. It performs no I/O. It touches no database. It is a pure function.
@dataclass(frozen=True)
class Absent:
kind: Literal['absent'] = 'absent'
@dataclass(frozen=True)
class AdvancedPastInput:
kind: Literal['advanced_past_input'] = 'advanced_past_input'
@dataclass(frozen=True)
class Completed:
assistant: AssistantMessage
overflow: bool
kind: Literal['completed'] = 'completed'
@dataclass(frozen=True)
class ToolUseUnresolved:
assistant: AssistantMessage
kind: Literal['tool_use_unresolved'] = 'tool_use_unresolved'
@dataclass(frozen=True)
class TerminalError:
reason: str
kind: Literal['terminal_error'] = 'terminal_error'
@dataclass(frozen=True)
class Pending:
kind: Literal['pending'] = 'pending'
@dataclass(frozen=True)
class Resume:
mode: SubmissionResumeMode
assistant: AssistantMessage
consecutive_retryable_errors: int
kind: Literal['resume'] = 'resume'
type SubmissionState = Absent | AdvancedPastInput | Completed | ToolUseUnresolved | TerminalError | Pending | ResumeThe resume branch carries the assistant message the interrupted attempt produced — it is already in the store — and the mode that tells recovery exactly how to continue. The other branches handle terminal outcomes: completed means the turn finished and just needs settling; terminal_error means a non-retryable failure that recovery cannot fix; absent means the input entry itself was not found.
Why a pure function? Because the classifier is shared. The reconciler calls it when it wakes up and inspects an expired lease. The new attempt calls it when it boots and needs to know what it inherited. If the classifier touched the database — if it ran a query to decide what to do — then two callers against slightly different snapshots could reach different conclusions. You would be debugging recovery failures that only occur under concurrent reconciler runs. Making the classifier pure eliminates that class of bug entirely. The same durable state always produces the same answer, regardless of who asks or when.
The resume kind carries one of seven modes, each naming a distinct interruption point:
input_only — input applied, model not yet called
stream_continuation — mid-stream crash; captured segments can be replayed
tool_results — model finished, all tool results present but not committed
tool_results_partial — model finished, some tool results present, some missing
transient_retry — a retryable error; restart the model call
overflow — context window exhausted; compaction needed before retry
aborted_partial — a partial tool batch from an aborted turn
Think of these as a decision tree over the assistant message's stop_reason. If there is no assistant message at all, the mode is input_only — the model was never called. If the stop reason is tool_use and tool results are present in the history, the question becomes whether all of them are there, which produces either tool_results or tool_results_partial. If the stop reason is error but the error message matches a retryable pattern (overloaded, rate-limited, timed out), the mode is transient_retry. If the stop reason is aborted and captured stream segments exist, the mode is stream_continuation.
The distinction between tool_results and tool_results_partial is the most consequential. If all tool results are present, recovery can commit the transcript delta and settle the turn without calling any tool again — no repair step needed. If some are missing, recovery must synthesize interrupted markers for the unresolved calls before the model can continue. Getting this wrong in either direction breaks the invariant: marking a complete batch as partial would inject phantom errors; failing to detect a partial batch would leave the model with an incomplete view of what happened.
The classifier delegates the partial-vs-complete question to find_trailing_partial_tool_batch. You should understand what this function actually does, because the logic is subtle.
The function receives the full following slice — every history entry after the submission input. It first rules out one case immediately: if the history contains a stream_continued signal, there is a recovered stream continuation in progress and no partial batch to detect. Then it checks whether the last entry in the slice is an aborted assistant message, and if so, excludes it from consideration — that aborted partial belongs to the next turn, not the tool batch being examined.
From that adjusted endpoint, the function walks backward. It collects toolResult entries into a set of resolved call IDs. When it hits a non-toolResult entry, it expects an assistant message with stop_reason == 'tool_use'. It then compares that assistant's tool calls against the resolved set. If every call has a result, the function returns None — the batch is complete. If any call is missing a result, it returns a TrailingPartialToolBatch describing the incomplete assistant entry, its tool calls in original call order, and the entry ID of the assistant message. The classifier uses the non-None return to select tool_results_partial over tool_results.
This function is isolated from the rest of the classifier for a reason: it is the most complex piece of logic, and it needs to be independently testable. You can write a table of (history entries → expected return) pairs and verify every branch without instantiating a full recovery session.
tool_results_partial is the most dangerous recovery case. The interrupted attempt executed some tools and not others. The completed tools may have had side effects. They cannot be re-run. The incomplete tools need to be resolved before the model can continue. The challenge is that you cannot simply skip the unresolved calls — the model knows it issued them and will be confused if they vanish from the transcript.
The runtime guarantees exactly-once result delivery, not exactly-once side effects. A tool whose side effect fired but whose result was not committed before a crash is reclassified as interrupted and may be retried by the model. The journal records tool intent at tool_request_recorded, not tool completion, so it cannot distinguish "not yet run" from "ran but result lost." Tools with externally-visible side effects must be idempotent — idempotency key, dedup at the target — because the window between side effect and result-commit is at-least-once.
The repair strategy is to write synthetic results. For each unresolved tool call in the partial batch, repairInterruptedToolCalls inserts an "interrupted" error result into the transcript. The model sees a clean record — every tool call it issued has a response — and can decide how to proceed. It may retry the tool via a follow-up message. It may report the failure to the user. What it will not do is re-execute a tool that already ran and whose result was committed.
Interrupted turn state:
assistant → [tool_call_A: result ✓] [tool_call_B: result ✗]
After repair:
assistant → [tool_call_A: result ✓] [tool_call_B: "interrupted" synthetic result]
─────────────── first-write-wins ────────────────────────────────────
result already in transcript is preserved; unresolved calls receive
a synthetic "interrupted" marker
The critical implementation detail is first-write-wins. When repairInterruptedToolCalls attempts to write a synthetic result for a tool call, it checks whether a result already exists for that call ID. If it does — because the original attempt managed to commit the result before crashing — the existing result is preserved. The synthetic write is a no-op. This means repair is safe to call even in ambiguous situations: it never overwrites real results, only fills gaps.
Now you can see how the journal phase from Chapter 6 and the classifier from this chapter connect. The reconciler wakes up, finds submissions with expired leases, and orchestrates recovery. The sequence depends on where in the journal the interrupted attempt stopped.
reconciler wakes
│
├─ submission.input_applied_at is None? → requeue
│
└─ classifySubmissionState(history entries since input)
│
├─ 'pending' → start fresh turn
├─ 'completed' → settle, done
└─ 'resume' → inspect journal.phase
│
├─ before_provider / provider_started
│ → replace_turn_journal_attempt → new attempt
│
└─ tool_request_recorded
→ repair_interrupted_tool_calls
→ update_turn_journal_phase → before_provider
→ replace_turn_journal_attempt → new attempt
Three cases from the journal phase determine the repair path. If journal.phase is before_provider or provider_started and the attempt was not committed, replace_turn_journal_attempt is called with no repair — the previous attempt either never reached the provider or was interrupted before recording a tool request, so a fresh attempt at the same submission is safe. If journal.phase == 'tool_request_recorded' and journal.tool_request exists, the attempt reached the point of issuing tool calls before crashing; repair_interrupted_tool_calls fills in synthetic interrupted results for any unresolved calls, update_turn_journal_phase resets the journal back to before_provider, and then replace_turn_journal_attempt hands off to a new attempt that now has a complete tool result batch to work from.
The third case handles the edge before any of this: if submission.input_applied_at is None, the submission was admitted to the queue but the input was never applied to the conversation. Nothing happened. The submission goes back to queued via a requeue operation, and the next worker picks it up as though it arrived fresh.
In v1, there was no classifier, no journal, no structured repair step. Recovery was implicit: when the session loaded, it reconstructed its history from a flat array of persisted entries and re-ran the turn loop from the current position. The loop inspected in-memory state and made ad-hoc decisions about what to re-run based on what it found.
This approach has a deep problem: it conflates recovery with normal execution. The same code path that handles a fresh model response also handles a resumed response. When a tool had already run and left side effects, those effects ran again if the history was in the wrong state. There was no first-write-wins protection because there was no deliberate write at all — the transcript simply accumulated whatever the loop produced.
More importantly, v1 recovery was untestable in isolation. There was no function you could call with a slice of history and ask "what state is this?" You had to run the full session loop, feed it a crafted history, and observe behavior. This made edge-case verification impractical. What happens if the process dies between tool dispatch and result storage? You could reason about it, but you could not test it with a unit test. Extracting classify_submission_state into a pure function is not a refactor of convenience. It is a prerequisite for correctness. A classifier you cannot test independently is a classifier you cannot trust in production.
The deepest point about classify_submission_state is its determinism guarantee. Given the same following slice, it always returns the same SubmissionState. No matter which process calls it, no matter how much time has passed since the crash, no matter whether the reconciler or the new attempt is asking.
This matters more than it first appears. Consider what happens without this guarantee. The reconciler classifies an interrupted submission as resume/tool_results_partial and begins repair. Simultaneously — because leases have race conditions — a second reconciler wakes up against the same submission. If the classifier were non-deterministic, the second reconciler might classify the same history as resume/tool_results and attempt to settle the turn without repair. You now have two recovery processes taking conflicting paths against the same submission. The outcome is undefined.
Determinism closes this race. Both reconcilers see the same history and reach the same classification. One of them will win the attempt-replacement write (the operation is transactional); the other will find the lease already taken and back off. The classifier does not prevent the race — durable writes and leases do that — but it ensures that any two processes that do run simultaneously will at least agree on what they are trying to do.
The practical consequence is that you can audit recovery behavior offline. Take a snapshot of the history for a specific submission, pass it to classify_submission_state, and you get the exact same classification the live system produced. You can write a regression suite of (history, expected mode) pairs and run it as a unit test. You can reproduce reported bugs without access to production infrastructure. The pure function is not just an architectural nicety — it is what makes recovery debuggable.
Parts I and II together answer what it means for an agent turn to be durable. The turn has a shared inner loop (drive_run), an outer envelope that journals each phase, and a recovery classifier that reads that journal and names the next action precisely. The primitive is established; the durability is layered on top without touching the primitive.
Part III shifts focus from time to space. Durability asks "what happens when the process dies?" The sandbox asks "what can the agent touch, and from where?" The tools that the model calls are the agent's reach into the world. How those tools are constructed — and what they can and cannot access — is the question Chapters 8 and 9 take up.
Files read for this chapter:
-
docs/the-rewrite/chapters/07-resume-and-repair.md— Chapter outline, covering eight sections: the recovery classification problem, the pure classifier design, seven resume modes, partial batch detection, repair strategy, recovery sequence, v1 comparison, and the determinism invariant. -
apps/runtime/src/durable/submission-state.ts— Full implementation ofclassify_submission_stateandfind_trailing_partial_tool_batch, including the completeSubmissionStatetype union, the reverse-scan helper for ES2022 compatibility, the retryable-error regex pattern, and thecount_consecutive_retryable_model_errorshelper.