pu/Incus cluster saturation — overnight 2026-07-05→06 (kolu #1204)

pu-saturation-gist.md

pu / Incus cluster saturation — overnight 2026-07-05→06 (W4 CI run)

For the pu admin. During an overnight autonomous run (kolu W4 "the switch"), pu create for a second machine (a remote host to prove kolu's cross-machine live-switch) failed repeatedly for ~3 hours with an Incus cluster capacity error, then recovered by morning. Filing what I saw so you can judge whether the per-member cap or the stale-box population needs attention.

Symptom

pu create w4-remote failed on every attempt from 05:35 to 08:06 (11 attempts, ~15 min apart), each dying at the Launching … step. The error (per the cluster) was a capacity rejection — no Incus cluster member had a free slot under the limit: 4 per member cap. A retry loop logged:

05:35 attempt 1: Launching w4-remote
05:50 attempt 2: Launching w4-remote
06:05 attempt 3: Launching w4-remote
06:20 attempt 4: Launching w4-remote
06:35 attempt 5: Launching w4-remote
06:50 attempt 6: Launching w4-remote
07:05 attempt 7: Launching w4-remote
07:21 attempt 8: Launching w4-remote
07:36 attempt 9: Launching w4-remote
07:51 attempt 10: Launching w4-remote
08:06 attempt 11: Launching w4-remote

Recovery

By ~morning (post-08:xx) capacity freed up: a fresh pu create succeeds now (verified — created + destroyed a w4-diag-probe box cleanly), and one of the late overnight w4-remote attempts eventually landed (it's in the list below). So this was a transient ~3-hour saturation, not a hard outage.

Current placement (pu list, this morning)

18 instances across 8 cluster members. Per-member counts show how close the pool runs to the 4 per member ceiling — dev-x86-64-linux-07 is at the cap (4):

  4  dev-x86-64-linux-07   (kolu-pr-1096-after, kolu-pr-1181, kolu-pr-1404-after, kolu-pr-1699)
  3  idliv2-02             (kolu-ci-5, kolu-flake-subterm, w4-diag-probe*)
  3  dev-x86-64-linux-08   (kolu-bot, kolu-ci-3, w4-remote)
  2  idliv2                (kolu-ci-4, kolu-flake-codetab)
  2  dev-x86-64-linux-05   (kolu-ci-8, kolu-pr-1509b)
  2  dev-x86-64-linux-04   (kolu-ci-2, kolu-ci-7)
  2  dev-x86-64-linux-03   (kolu-ci-1, kolu-ci-6)
  1  dev-x86-64-linux-09   (kolu-pr-1404-before)
     (* w4-diag-probe was destroyed right after this snapshot)

What likely held capacity

A cluster of stale-looking per-PR / flake boxes appears to be sitting on slots the CI + dev pool needs during a busy overnight window:

• kolu-pr-1096-after, kolu-pr-1181, kolu-pr-1404-after, kolu-pr-1404-before, kolu-pr-1509b, kolu-pr-1699 — per-PR boxes; PRs #1096/#1181/#1404 look long-merged.
• kolu-flake-codetab, kolu-flake-subterm — flake-repro boxes.

If those are done with, reaping them would give the 4 per member cap real headroom (the kolu-ci-1..8 pool + any pu create burst needs ~8+ slots concurrently, and 6 stale PR boxes on top of that is what tipped it over overnight).

Ask

1. Consider reaping the stale kolu-pr-* / kolu-flake-* boxes (confirm with their owners first).
2. Consider whether the limit: 4 per member cap is right for the current member count, or whether an overnight CI + dev burst legitimately needs more headroom.

No action needed from me — the W4 run recovered and its CI is green; this is capacity hygiene for the shared cluster. Tracking on kolu#1204.