Iris v4-2048 zombie-worker analysis (issue #4724)

v4_2048_analysis.md

Zombie workers from a reserved TPU slice whose QR was abandoned but not cancelled

Cluster: marin (config lib/iris/examples/marin.yaml) Slice: marin-tpu-v4-reserved-2048-us-central2-b-20260414-2123-fd07e934 Symptom: Provider sync: 660 workers, 56 failed, 13015ms failed=[... 33× v4-2048 workers ...] at 22:51 UTC on 2026-04-14.

Timeline

From controller logs filtered by fd07e934:

21:23:22  Create queued resource marin-tpu-v4-reserved-2048-...-fd07e934
21:23:29  Slice registered with autoscaler; QR state = WAITING_FOR_RESOURCES
21:24–33  QR stays WAITING_FOR_RESOURCES (polled every 60s)
21:33:30  Bootstrap FAILED — "Queued resource did not become ACTIVE within 600.0s"
          ← hits cloud_ready_timeout in _run_tpu_slice_bootstrap (workers.py:714)

...70 minutes pass, no log entries for this slice...

22:43:26  Worker-124 registers at 10.130.0.189:10001
22:43:27-35  32 more workers register (worker-{248,227,214,109,117,13,35,88,
             157,230,166,27,198,23,177,29,242,62,179,48,173,113,236,169,4,46,
             205,103,40,246,159}) — exactly the 33 from the "failed=" list

22:51:50  Provider sync lists these 33 as failed — 10 consecutive missed
          heartbeats × 5s interval = ~50s unreachability

22:58:22  Controller force-deletes the queued resource

Root cause

The v4-2048 queued resource never became ACTIVE within the 10-minute cloud_ready_timeout (hardcoded in _run_tpu_slice_bootstrap at lib/iris/src/iris/cluster/providers/gcp/workers.py:684). The bootstrap thread set handle._bootstrap_state = FAILED and gave up, but the queued resource was not cancelled in GCP.

~70 minutes later GCP fulfilled the reservation (normal behaviour for v4 reserved capacity — QRs can sit in WAITING_FOR_RESOURCES for hours). VMs booted, ran the iris startup script, and the worker daemons phoned home to the controller via the registration RPC. Only 33 of the 512 expected VMs (v4-2048 / 4-chips-per-VM) ever registered — the rest either failed startup or were never provisioned.

Those 33 landed in the workers table as orphans. The controller-side slice had already been marked failed in the autoscaler, so there was no active slices row backing them. They missed heartbeats, crossed the heartbeat_failure_threshold (10) all in the same sync round, and got logged together. The 13s elapsed sync fits: 33 simultaneous RPC failures to unreachable VMs will blow the sync ThreadPool latency.

Why this bypassed the failure path

With the controller code prior to PR #4727:

• Bootstrap thread sets _bootstrap_state = FAILED on timeout.
• _composite_slice_state(cloud_state=CREATING, bootstrap_state=FAILED) returns CREATING — bootstrap failure is hidden.
• refresh() only handles READY / FAILED / UNKNOWN. CREATING has no timeout path. Slice stays alive in autoscaler bookkeeping.
• QR remains alive in GCP. Eventually fulfilled → zombie workers.

PR #4727 (still OPEN as of 2026-04-14) partly fixes this:

• _composite_slice_state now returns FAILED when bootstrap is FAILED.
• refresh() adds a 15-minute unresolvable_timeout for CREATING / BOOTSTRAPPING / REPAIRING.

But 15 minutes is too aggressive for reserved TPUs — GCP can legitimately keep a QR in WAITING_FOR_RESOURCES for hours. A one-size-fits-all timeout will trigger false failures on normal reservation queue time.

Proposed design (for discussion)

Split the single cloud_ready_timeout into three purpose-specific deadlines for queued TPU slices:

Phase	GCP state transition	Default timeout
Phase 0a: Assign	QR submitted → PROVISIONING (GCP agrees to allocate)	4h
Phase 0b: Provision	PROVISIONING → ACTIVE (VMs exist)	1h
Phase 1: Cloud READY	ACTIVE → all VMs have IPs	10 min (current)
Phase 2: Worker health	IPs → all /health = 200	10 min (current)

Non-queued (preemptible, on-demand) TPUs keep the current single 10-min cloud_ready_timeout — they have no queue phase.

Phase 0a is the phase that currently misbehaves: _run_tpu_slice_bootstrap treats WAITING_FOR_RESOURCES as one amorphous blob with a 10-min deadline, which is wrong twice — too long to fail a genuinely broken QR submission, too short to tolerate normal reservation queueing.

Critical: cancel the QR on bootstrap timeout

Whatever timeouts we pick, the fix must also call queued_resource_delete() when the bootstrap thread raises InfraError. This can live either:

• In the bootstrap thread's except handler (cleanest — same place that sets _bootstrap_state = FAILED), or
• In refresh() when the slice transitions to FAILED (via handle.terminate(), which already calls queued_resource_delete for QR-backed slices per handles.py:372).

The handle.terminate() path works if the slice reaches FAILED in the autoscaler — which requires PR #4727 to land. Belt-and-braces: do it in both places.