A Claude Code Workflow pattern for tasks that take days, must not run in parallel, and must not stop.
Battle-tested on a real run: 500+ generated documents, 8 batches, ~20M subagent tokens, 3 days of continuous operation, one rate-limit outage survived. Every lesson below comes from something that actually broke.
Use this pattern when all of these hold:
- The work is a long list of similar items (hundreds to thousands)
- Each item needs real reasoning, not a script — so you want a subagent per item
- An external constraint forces serialization — rate limits, bot detection, a licensed API, a device that can only do one thing at a time
- The run is too long for one sitting, so it must survive context compaction, notification gaps, and outages
Do not use it when items are independent and parallelizable. Then you want parallel() or pipeline() and you should stop reading.
The counterintuitive part: Workflow's value here is not parallelism. It's deterministic control flow — a loop that lives in a script instead of in the model's judgment each turn.
The stop points are the fragile part, not the work.
The first 55 items of the real run were done one at a time, each needing a human "go on". The bottleneck looked like per-item quality. It wasn't. Every pause is a chance for the chain to break — the human gets distracted, the session closes, the assistant decides to summarize instead of continue.
Going from 1 item per turn → 40 items per batch removed 97% of the stop points and the pipeline immediately became stable.
If a user has to say "continue" twice, the loop design is wrong. Redesign the loop; don't do one more item.
┌─ Main loop (the assistant) ────────────────────┐
│ launch batch → read report → update pointer │◄──┐
└───────────────────┬────────────────────────────┘ │
│ │
┌───────────▼──────────────┐ │
│ Workflow (background) │ │
│ for await (item of N): │ ← serialized │
│ agent(item) │ │
│ retry failures in-run │ │
│ return {next, retries} │──── notify ──────┘
└──────────────────────────┘ ▲
│
┌─────────────────────────────┐ │
│ Watchdog cron (every 2-3h) │─── resume ────┘
│ probe API → only then run │
└─────────────────────────────┘
Three layers of self-binding, strongest first:
| Layer | Mechanism | Survives |
|---|---|---|
| Structure | The endpoint is encoded in a data structure that cannot generate work past it | Everything |
| Memory | A persistent memory file stating the rule and the goal | Context compaction |
| Watchdog | A cron that probes and resumes | Notification gaps |
Put the goal in a data structure, not in a comment or a memory. In the real run, the year list [112, 111, 110, ...105] drives the ID generator — after the last item it physically cannot produce another ID. Forgetting the endpoint is not possible.
export const meta = {
name: 'sequential-pipeline',
description: 'One agent per item, strictly serialized, rate-limit aware, auto-retry',
whenToUse: 'args = { startId, count, retryIds?, prevId? } or { ids: [...] }',
phases: [
{ title: 'Main', detail: 'One agent per item: pace → fetch → work → write back' },
{ title: 'Retry', detail: 'Re-run this batch\'s failures once, with rephrased prompt' },
],
}
const WORKDIR = '/abs/path/to/your/workdir'
// ── GOTCHA #1 ────────────────────────────────────────────────
// args sometimes arrives as a JSON *string*. Without this line,
// `args.ids` is undefined and the workflow silently runs 0 agents.
const A = typeof args === 'string' ? JSON.parse(args) : (args || {})
// ── The endpoint lives HERE. Past the last group, no IDs exist. ──
const GROUPS = [112, 111, 110, 109, 108, 107, 106, 105]
function nextIds(startId, n) {
const m = /^(\d+)-(\d+)$/.exec(startId)
if (!m) return []
let gi = GROUPS.indexOf(Number(m[1]))
let num = Number(m[2])
if (gi < 0) return []
const out = []
while (out.length < n && gi < GROUPS.length) {
out.push(`${GROUPS[gi]}-${String(num).padStart(3, '0')}`)
num += 1
if (num > 100) { num = 1; gi += 1 } // roll over to next group
}
return out
}
const DERIVED = (A.ids && A.ids.length) ? A.ids : nextIds(A.startId, A.count || 20)
// Failures from the previous batch jump the queue — no hand-built lists.
const RETRY_IN = (A.retryIds || []).filter(q => DERIVED.indexOf(q) < 0)
const IDS = RETRY_IN.concat(DERIVED)
if (!IDS.length) return { error: 'need args.ids, or args.startId + args.count' }
const RESULT = {
type: 'object',
properties: {
id: { type: 'string' },
ok: { type: 'boolean', description: 'true ONLY if the work was verifiably written' },
output: { type: 'string' },
angle: { type: 'string', description: 'what made this item distinct from prior ones' },
caveat: { type: 'string', description: 'honest flags: bad source data, conflicting evidence' },
note: { type: 'string', description: 'if failed, which step it died on' },
},
required: ['id', 'ok'],
}
// ── GOTCHA #2 ────────────────────────────────────────────────
// agent() returns null when the subagent dies. Normalize it, so a
// failure ALWAYS has a shape and can never vanish silently.
async function runOne(id, prevId, isRetry) {
const r = await agent(buildPrompt(id, prevId, isRetry), {
label: isRetry ? `${id}·retry` : id,
phase: isRetry ? 'Retry' : 'Main',
schema: RESULT,
})
return r || { id, ok: false, note: 'agent died or was skipped' }
}
phase('Main')
const done = []
for (let i = 0; i < IDS.length; i++) {
// SERIAL ON PURPOSE. parallel() here would trip the rate limiter.
const prevId = i === 0 ? (A.prevId || IDS[0]) : IDS[i - 1]
const r = await runOne(IDS[i], prevId, false)
log(r.ok ? `${IDS[i]} ✅` : `${IDS[i]} ✗ ${r.note || '?'}`)
done.push(r)
}
// Most failures (content filters, upstream hangs) clear on a reworded retry.
// Do it in-run so a human never has to assemble a retry list by hand.
let failed = done.filter(d => !d.ok).map(d => d.id)
if (failed.length) {
phase('Retry')
let prev = IDS[IDS.length - 1]
for (const id of failed) {
const r = await runOne(id, prev, true)
const idx = done.findIndex(d => d.id === id)
if (idx >= 0) done[idx] = r
prev = id
}
failed = done.filter(d => !d.ok).map(d => d.id)
}
// nextStart uses DERIVED, not IDS — otherwise queue-jumping retries
// would rewind the pointer.
const nextStart = nextIds(DERIVED[DERIVED.length - 1], 2)[1] || null
return {
total: IDS.length,
ok: done.filter(d => d.ok).length,
retryIds: failed, // pass straight back in as args.retryIds next time
nextStart, // pass straight back in as args.startId next time
results: done,
}The return value is the whole point. nextStart + retryIds mean relaunching the next batch requires zero re-derivation. Lowering the cost of continuing is what keeps the chain alive.
Four things matter far more than prose quality:
## Hard rules (violating any = failure)
1. **You must actually call <the external service>.** Only a real
response ID counts as success. Never fabricate content to fill the gap.
2. **Pacing first.** Step 1 is ALWAYS `./pace.sh <prev-id>`.
Do not improvise with sleep. Do not skip it.
3. **Never reuse an angle already covered.** See step 3.
This is the single highest-leverage instruction in the whole pattern.
### Step 3: find an angle that hasn't been used
grep -i -E '<keyword for this item>' LEDGER.md | tail -15
LEDGER.md lines are `- <id> | ✅ | <ref> | <summary>`. The summary IS the
angle that item already claimed.
- Topic never seen → use the standard treatment.
- Topic seen before → you MUST change axis. Options:
· why this *test* discriminates, rather than the mechanism again
· why the study's *endpoint design* failed or is over-cited
· how anatomy/location decides whether the treatment helps at all
· how the benefit and the risk are the same mechanism seen twice
· what the population/genotype difference means locally
· which *direction* the wrong answer got backwards
Then — and this is the part that actually works:
### Step 4: submit
Include a **"scope" section** in the request that lists, verbatim, the
angles you found in step 3, and says: "I already have notes covering X, Y,
Z — do not repeat or expand those."
In the real run this kept 18 consecutive items on the same disease from ever colliding.
Do not write "please check carefully." Write:
### Step 6: inspect the output before writing
Two failures that have ACTUALLY happened — check for both:
- **Mismatched labels** (item 112-029): the service analyzed option (C) as
one thing when (C) was something else. Verify every label maps correctly.
- **Wrong figure** (item 112-052): a figure was inserted whose caption
didn't match the surrounding section. Cross-check against the figure list.
Naming two real precedents turned "checking" from an attitude into an action. In the real run, agents then went on to independently catch six other categories of error nobody had told them about — including source documents citing markers that didn't exist in their own tables, and a mechanism described backwards in the reference material.
Return per the schema. `ok` is true ONLY if you obtained a response ID
AND the write-back verifiably succeeded. Make the ledger summary specific —
later items depend on it to avoid collisions.
A cron that probes before resuming. A watchdog that blindly relaunches into a rate-limited API is worse than none.
Every 2-3 hours:
1. Is it still running?
find <workflow-dir> -name 'agent-*.jsonl' -printf '%T@ %p\n' | sort -rn | head -1
- Written within 20 min → reply one line, "running", stop.
2. Not running → PROBE FIRST.
Send one tiny request to the external service.
- 429 / throttled → reply "still throttled (waited Nh)", launch nothing, stop.
- Success → continue.
3. Throttle cleared → relaunch with a REDUCED count (20, not 40).
Confirm the limit doesn't immediately recur before returning to full size.
4. Three consecutive throttled probes → say plainly that a human must check
the account. Do not retry forever.
| Signal | Cadence | Verdict |
|---|---|---|
journal.jsonl |
once per item (~10 min) | ❌ too coarse — reads as "stalled" constantly |
agent-*.jsonl |
seconds | ✅ this is the real heartbeat |
And beware ls -t. On a machine with eza aliased to ls, -t means something else entirely and the sort silently lies. Always find -printf '%T@'.
Late in the run, everything looked healthy: heartbeat fresh, agents writing, no errors surfaced. But three hours had produced one item, and the output count hadn't moved.
The tell was not the heartbeat. It was the divergence between "items completed" and "artifacts produced".
Digging in: the upstream service had started returning 429 {"code":"throttled"} and an agent had faithfully logged "step 4 returned 429 seven times in a row."
A heartbeat tells you something is moving. It does not tell you anything is progressing. Monitor both.
Also worth knowing: the error body identified the culprit. {"type":"client_error","errors":[{"code":"throttled",...}]} is Django REST Framework + drf-standardized-errors — an application-level quota. Bot-detection services return 403 with a challenge page, not structured JSON. The pacing regime had been defending against the wrong threat model for the entire run: request density was never the constraint; cumulative volume was.
Read the actual error body before designing around a guess.
| Failure | Symptom | Fix |
|---|---|---|
args arrives as a JSON string |
args.ids undefined, 0 agents run |
typeof args === 'string' ? JSON.parse(args) : args |
| Silent no-op step | A sed/sd replace whose target never existed; exit code 0 |
Give every write a verifiable echo; put verification in the schema |
| Upstream request hangs | Status running >20 min, empty output |
Declare it dead, resubmit a slightly trimmed prompt |
| Tool call exceeds harness timeout | Call gets backgrounded | Use timeout_sec under the limit + re-poll after a wait |
| Content filter blocks output | Output blocked by content filtering policy |
Retry with reworded prompt — keep the depth, change the framing |
| Rate limit / quota | 429 throttled |
Stop the run. Probe on a schedule. Resume smaller. |
Chained sleep rejected |
Harness blocks sleep N; cmd |
Put waiting in a script (pace.sh) |
#!/usr/bin/env bash
A=${1:-10}; ID=${2:-000}
NUM=${ID##*-}; NUM=$((10#${NUM:-1}))
if [ $((NUM % 10)) -eq 0 ]; then T=$((480 + RANDOM % 91)); TAG="long rest"
elif [ "$A" -ge 12 ]; then T=$((210 + RANDOM % 91)); TAG="backing off ($A)"
elif [ "$A" -le 8 ]; then T=$((130 + RANDOM % 51)); TAG="relaxed ($A)"
else T=$((150 + RANDOM % 61)); TAG="standard ($A)"
fi
echo "--- ${TAG}: waiting ${T}s ---"
sleep $TPass the previous item's ID, so the every-10th long rest lands automatically with no extra logic. The randomness matters — perfectly regular intervals are themselves a signature.
| Metric | Value |
|---|---|
| Per item | ~9.6 min wall-clock, ~73k subagent tokens |
| Per 40-item batch | ~6.4 hours, ~2.9M tokens |
| 1000 items (projected) | ~160 hours, ~73M tokens |
Budget for this before starting. Note that with a cumulative quota, slowing down buys you nothing — it just stretches the same volume over more days.
- Encode the endpoint in a data structure that can't generate work past it
- Write the "never stop" rule to persistent memory — it must outlive compaction
- Start with large batches (40+); don't ramp up from small
- Every write step gets a verifiable echo, declared in the return schema
- Choose the heartbeat signal and the stall threshold before launching
- Monitor progress separately from liveness — they fail differently
- Make dedup a grep-then-paste-into-prompt step, never an exhortation
- Name real past failures in the prompt, with IDs
- Retry in-run, and return leftovers as a ready-to-pass array
- Probe before resuming after any outage
- Verify long-lived external auth won't expire mid-run
- Cron jobs are session-only and expire after 7 days. The watchdog covers a broken notification chain; it does not survive the session closing. For a multi-week run you need an external scheduler.
- The chain still needs the assistant to be re-invoked. Background work continues, but nothing launches the next batch on its own if the session is gone.
- Interactively-authenticated MCP servers may be unavailable in headless or cron contexts.
- Delegating to subagents costs quality unless the verification loop is concrete. It worked here because the checks were mechanical and named real precedents — not because subagents are inherently careful.
Derived from a production run building 1000 contextual reference documents from a medical board-exam question bank, August 2026. Every gotcha listed is one that actually cost time.