Skip to content

Instantly share code, notes, and snippets.

@denislemire

denislemire/gist-k3s-system-upgrade-left-behind-2026-03-05.md

Created March 6, 2026 00:12
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save denislemire/e3401fa915c37cc42908ed6e0321c53c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save denislemire/e3401fa915c37cc42908ed6e0321c53c to your computer and use it in GitHub Desktop.
Download ZIP
K3s system-upgrade left-behind incident 2026-03-05 (EHWS)
Raw
gist-k3s-system-upgrade-left-behind-2026-03-05.md

K3s System Upgrade "Left Behind" Incident — 2026-03-05

Cluster: EHWS K3s (rick, morty, summer = control-plane; jerry = worker)
Channel: https://update.k3s.io/v1-release/channels/latest
Outcome: Two nodes (summer, jerry) stayed on v1.35.1 while rick/morty had upgraded to v1.35.2. First time the auto-upgrade failed in this way.

Findings

1. Agent-plan had failed on jerry (worker)

  • Plan status: agent-plan showed COMPLETE: False with message:
    Job system-upgrade/apply-agent-plan-on-jerry-with-43c357aaac296efdc5d4af2e23-aaf92 failed on Node jerry: DeadlineExceeded: Job was active longer than specified deadline
  • Effect: The failed job was eventually cleaned up (no jobs left in system-upgrade), but the controller does not retry failed jobs automatically. So jerry never got a new upgrade job until we forced re-evaluation.

2. Summer (control-plane) had plan label but old version

  • State: Summer had the label plan.upgrade.cattle.io/server-plan=43c357aaac296efdc5d4af2e23cc28986bf59fe657da54e592079e16, so the controller considered server-plan "complete" and did not schedule another server-plan job for summer.
  • Reality: Summer’s kubelet was still reporting v1.35.1.
  • Cause: A previous upgrade job on summer had run, compared binary checksums, found "Binary already been replaced" (same checksum), and exited 0 without restarting k3s. So the new binary was on disk but the running process was still the old version.

3. No jobs left to delete

  • When we ran the fix script, kubectl get jobs -n system-upgrade was empty. The failed jerry job had already been removed (by controller or TTL). So the "delete failed jobs" step did nothing; only the plan re-annotation triggered new jobs.

4. Plan re-annotation worked

  • Annotating both plans (e.g. k3s-upgrade.trigger=<timestamp>) caused the controller to re-evaluate. It then created:
    • A new agent-plan job for jerry (which ran to completion this time).
    • No new server-plan job for summer until we removed summer’s plan.upgrade.cattle.io/server-plan label.

5. Summer needed a k3s restart

  • After removing the server-plan label, the controller created a new server-plan job for summer. That job again saw "Binary already been replaced" and exited without restarting k3s.
  • Fix: We ran a one-off privileged pod on summer that did nsenter --target 1 --mount -- systemctl restart k3s. After the restart, summer reported v1.35.2.

What we did (remediation)

  1. Diagnose: ./scripts/diagnose-k3s-system-upgrade.sh — showed version skew and plan status (agent-plan failed, server-plan complete).
  2. Re-trigger plans: ./scripts/fix-k3s-system-upgrade-left-behind.sh --all — no jobs to delete; annotated both plans.
  3. Summer: Removed label plan.upgrade.cattle.io/server-plan from node summer so the controller scheduled a new server-plan job. That job completed but again did not restart k3s (binary already replaced). Restarted k3s on summer via one-off pod; summer then reported v1.35.2.
  4. Jerry: New agent-plan job was created after re-annotation and completed successfully (jerry → v1.35.2).
  5. Cleanup: Deleted the one-off restart pod.

Result: All four nodes on v1.35.2+k3s1.

Takeaways for next point release

  • If nodes are left behind with no failed job in the namespace:
    Annotate plans to force re-evaluation:
    kubectl annotate plan -n system-upgrade server-plan k3s-upgrade.trigger=$(date +%s) --overwrite
    (and same for agent-plan). Or use ./scripts/fix-k3s-system-upgrade-left-behind.sh --all.

  • If a control-plane node has the server-plan label but is still on the old version:
    Remove the plan label so the controller schedules a new job:
    kubectl label node <name> plan.upgrade.cattle.io/server-plan-
    If the new job completes but the node version doesn’t change, the upgrade script likely exited with "Binary already been replaced" and didn’t restart k3s. Restart k3s on that node (SSH: sudo systemctl restart k3s) or via a one-off privileged pod with nsenter + systemctl restart k3s.

  • Jerry’s DeadlineExceeded:
    First time we’ve seen an agent-plan job hit the active deadline. If it happens again, consider increasing the job’s activeDeadlineSeconds in the plan (if the controller supports it) or ensuring the prepare step doesn’t block too long.

  • Scripts added in ehws-infra:

    • scripts/diagnose-k3s-system-upgrade.sh — node versions, cordon, plans, jobs/pods.
    • scripts/fix-k3s-system-upgrade-left-behind.sh — delete failed (or all) jobs, annotate plans.
    • clusters/ehws/system-upgrade/README.md — documents plans, why nodes get left behind, and how to diagnose/fix.

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment