Window: 2026-07-22 (Code Freeze AoE; enforced 2026-07-23 12:00 UTC via test-infra#37475) → 2026-08-03
Release: v1.37
Compiled: 2026-08-03 (~13:30 UTC data). All "status now" claims verified against live prow job history, TestGrid summary JSON, k8s-triage failure_data.json (2026-08-03 11:43 UTC), and k8s-metrics (2026-08-03 12:05 UTC).
- sig-release-master-blocking has ZERO failing tabs as of 2026-08-03 — 19 passing, 4 flaky, 0 failing. All four gce-cos blocking lanes are at 100% of recent runs.
- Two multi-day outages were diagnosed and resolved:
ci-kubernetes-cross-canarywas stuck in an abort loop for 11.5 days (568 ABORTED builds, 07-21 → 08-01) — root-caused to AKS ephemeral-disk eviction, mitigated by moving to the EKS build cluster. 21 consecutive greens since.- The 2026-07-31 GCE incident (~152 builds across ~51 jobs failed cluster-up in one day) — root-caused to two active images in the COS
cos-129-ltsfamily; hardening fix merged intocluster/gce/util.shsame day.
- The AWS/EC2 board got a full overhaul: an AMI-build path red since at least May 19 is green, the EC2 alpha lanes flipped from red streaks to green streaks (verified in prow history to the hour), and the EC2 conformance flake (~11% bring-up failures since March) is fixed. EC2 conformance amd64 + arm64: 100% of recent runs.
- ~30 jobs/lanes fixed via ~60 PRs across 6 repos (kubernetes/kubernetes: 102 PRs merged in the freeze window, ~37 CI/test-related; test-infra ~30 relevant merges; provider-aws-test-infra 10/10 merged PRs were CI fixes/hardening; plus k8s.io, windows-testing, boskos).
- Per k8s-metrics, only ONE job that started failing after code freeze is still continuously failing (
ci-kubernetes-e2e-gce-agentic-sandbox-continuous-burst-100, red since 07-26, owned by the agent-sandbox team). Every other entry in the continuous-failure list predates the freeze by months. - Call to action (details in §4): a Windows presubmit blocking ALL PRs (#140900),
gce-master-scale-correctnessat 30%, the amd64 serial EC2 node lane, Windows-on-GCE jobs (4 large failure clusters), s390x integration, a kubelet DATA RACE on the kind race lane, and 3 fresh LoadBalancer ETP flake issues filed 08-03.
| Category | Count |
|---|---|
| Jobs/lanes broken → fixed & verified green | ~27 (counting the 51-job GCE incident as one line) |
| Fixes merged, verification still pending (too recent) | 2 (arm64-e2e-serial-ec2 glibc fix merged 08-03; cloud-provider-aws OOM bump merged 08-03, first post-fix run green) |
| Harness/capacity improvements merged | 15+ |
| Jobs retired to free capacity | all kubemark jobs + experimental scale jobs + 1.33 DRA/jobset jobs |
| Open failing-test/flake issues needing owners (§4) | ~30 |
| Dashboard | Failing | Flaky | Passing | Notes |
|---|---|---|---|---|
| sig-release-master-blocking | 0 | 4 | 19 | Flaky: integration-master (77.8%), ci-kubernetes-unit (90%), skew-cluster-latest-kubectl-stable2-gce (90%), kind-ipv6-master |
| sig-release-master-informing | 1 | 9 | 15 | Failing: gce-master-scale-correctness (30%) |
| amazon-ec2 | 0 | 5 | 7 | Flaky tabs are converging post-fix (see §2 receipts) |
| google-gce | 3 | 5 | 7 (+8 stale) | Failing: agentic-sandbox-burst-100 (0%), scale-correctness, gci-gce-flaky (red by design, 1722 days) |
| Job(s) | What was broken | Root cause | Fix | Status now (verified) |
|---|---|---|---|---|
| ci-kubernetes-cross-canary | 568 ABORTED builds at ~27 min cadence, 2026-07-21T10:01Z → 2026-08-01T20:22Z. One ProwJob stuck pending; plank (error_on_eviction=false) recreated the pod every cycle; horologium never scheduled fresh runs. Invisible to k8s-triage (aborted runs upload no junit). |
AKS k8s-infra-prow-build nodepools: 100GB ephemeral OS disk shared by images + emptyDirs (kubelet_disk_type=OS) while 440GiB NVMe sat unused; make release writes ~60–90GB → deterministic kubelet disk-pressure eviction ~25–29 min in |
test-infra#37598 — move job to eks-prow-build-cluster (RAID0 NVMe) — merged 08-01 | ✅ First SUCCESS 2026-08-01T20:25Z; 21 consecutive greens through 08-03T12:37Z, ~35 min runtime, 2h cadence restored. |
| All GCE kube-up e2e jobs (~152 builds / 51 jobs on 2026-07-31), incl. every gce-cos blocking lane | e2e-up.sh failed: two newline-joined image names in sourceImage → instance creation rejected |
cluster/gce/util.sh resolves boot image via gcloud compute images list --filter=family:... (returns ALL active images); cos-cloud briefly had cos-129-19506-299-60 and -61 both active (~6h, then rollback) |
k/k#141082 — sort by creation timestamp, take exactly one — merged 07-31T18:49Z (issue #141080); stopgap image pin test-infra#37595 | ✅ All 4 gce-cos blocking tabs 100% of recent runs (13212/13212 cells on default-master). |
post-kubernetes-push-e2e-*-test-images, build/pause postsubmits |
Immediate failure: builder image manifest not found | gcr.io/k8s-staging-test-infra/gcb-docker-gcloud January tag was pruned from the staging registry |
k/k#141034 repin to v20260729; rebuild chain test-infra#37568/#37571 (docker v29)/#37576; ownership → releng #37588 |
✅ merged 07-29/07-30 |
| ci-aws-ec2-janitor | Every run failed in ~9s since 07-29 16:55 UTC — nothing was being cleaned | boskos#260 added FSx cleanup; node-e2e-tests role lacked fsx:DescribeFileSystems; janitor aborts entire sweep on first error |
Manual IAM fix (k8s.io#9790, closed 07-30); robustness: boskos#262 don't abort sweep on first error | ✅ TestGrid PASSING 100% |
| prow.k8s.io spyglass | Intermittent upstream connect error crash-loop for days (k8s.io#9742) |
deck/hook/horologium memory limits too low | k8s.io#9748 — merged 07-22 | ✅ |
| Job(s) | What was broken | Root cause | Fix | Status now (verified) |
|---|---|---|---|---|
| ci-kubernetes-ec2-conformance-latest + arm64 (informing board) | ~11% bring-up flake since March: SynchronizedBeforeSuite timeout at 2/3 nodes, kubelet loses apiserver (no route to host) when cilium starts |
kubeadm podSubnet was set to the VPC CIDR on the Cilium overlay path → pod CIDRs randomly contained other nodes' instance IPs → blackhole route |
paws#576 — 10.244.0.0/16 for overlay; cilium 1.18.4→1.19.6 — merged 07-17 (k/k#137793 closed) | ✅ Conformance-EC2-master 9/9, arm64 10/10 recent runs |
| ci-kubernetes-e2e-ec2-alpha-features, ci-kubernetes-e2e-ec2-alpha-enabled-default (AMI path) | Red back to at least 2026-05-19: packer AMI build died on kubelet.sha256 S3 403 |
populate-s3.sh used cwd-relative _output (hashed the wrong tree; no .sha256 in versioned S3 tree); eks-ami Makefile forced us-west-2 while check-ami.sh reads us-east-1 |
paws#582 — merged 07-29 (k/k#141038: "we have a green ci job now") | ✅ see receipts §2 |
| ci-kubernetes-e2e-ec2-alpha-features | failed to create gzip reader: EOF — S3 NoSuchKey XML used as version string |
Job passed --test-package-url=https://dl.k8s.io/ (trailing slash → //ci/fast/latest-fast.txt). Latent for months, unmasked by the AMI fix |
test-infra#37586 — merged 07-30 | ✅ |
| ci-kubernetes-e2e-ec2-alpha-enabled-default | NFS host-cleanup tests: error reading SSH key /root/.ssh/kube_aws_rsa: no such file (13-build triage cluster) |
With EC2 Instance Connect the deployer generated per-node keys into temp files; e2e framework only reads ~/.ssh/kube_aws_rsa |
paws#583 — shared persisted key — merged 08-02 | ✅ 7 consecutive greens since 08-02T00:12Z (was 7 consecutive fails before) |
| ci-kubernetes-e2e-ec2-alpha-features | [FAILED] No ssh-able nodes (networking.go:554) starting 08-02 — regression exposed by the #583 fix (test was previously silently skipped) |
Framework tried to dial node port 22 from the prow pod outside the VPC | paws#584 — SSH via control-plane bastion (KUBE_SSH_BASTION) + SG rule — merged 08-02 |
✅ green since 08-02T23:06Z |
| ci-containerd-node-e2e-ec2 (+ serial, intermittent since ~07-27) | Instance boots, containerd never ready in window | package_upgrade: true against a 5-month-old pinned AMI = 248+ package apt upgrade at boot; bonus: Python 3.14 removed pipes used by configure.sh |
paws#578 — Ubuntu 26.04 (serial 20260722), drop package_upgrade, pipes→shlex — merged 07-29 (k/k#141013) |
✅ 14 consecutive greens since 07-31 |
| All EC2 jobs (shared userdata) | worker user data is too large ... 16498 vs AWS 16384-byte limit |
Userdata growth (cilium, DRA, CIDR fix, 26.04) | paws#580 + #581 — merged 07-29 | ✅ ( |
| ci-containerd-node-arm64-e2e-serial-ec2 | Every OOMKiller test failed since 07-29 (exit 128/StartError instead of 137/OOMKilled) — triage | glibc 2.43 (Ubuntu 26.04) enables malloc THP by default on aarch64 only; runc:[2:INIT] picks up 4–6×2MB THPs → 15Mi-limit containers OOM during runc create |
paws#585 — disable 2MB THP pre-containerd on aarch64 — merged 08-03T11:45Z | ⏳ Verification pending — no completed post-fix run yet (last check: FAILURE at 04:10, ABORTED 10:11) |
| ci-cloud-provider-aws-e2e-with-kubernetes-master | Go compiler SIGKILLed compiling aws-sdk-go-v2/CEL after k8s 1.37-beta.0 dep bump; worst tab on amazon-ec2 (33%) | 4Gi container limit too small | test-infra#37602 — 8Gi — merged 08-03T01:15Z | 🟡 First post-fix run (06:58Z) SUCCESS; needs a few more cycles |
| kops AWS grids (e.g. e2e-kops-grid-cilium-rocky9-k33-ko33) | could not find Image — latest Rocky 9/10 AMIs exist only in us-east-1; jobs pick random regions |
AMI-selection didn't check per-region availability | test-infra#37587 (rifelpet) — require AMI present in all regions — merged 07-30 | ✅ merged (job-level recovery not individually re-verified) |
| Job(s) | What was broken | Root cause | Fix | Status now |
|---|---|---|---|---|
| ci-kubernetes-e2e-capz-master-windows (informing) | Setup failure since 07-22: (K8sVersionNotSupported) ... only available for Long-Term Support |
Management AKS cluster version fell into LTS-only range | windows-testing#584 — dynamic AKS version — merged 07-23 (k/k#140869) | ✅ capz-windows-master tab PASSING 100% |
capz-windows: 3× Override hostname of Pod [NodeConformance] |
Regressed when HostnameOverride went GA (tests started running) | Windows busybox hostname.exe lacked -f/--fqdn |
k/k#139171 (new hostname.exe) → #140917 (busybox 1.38.0-1 baseimages) → #141042 (consume agnhost 2.66.1 etc.; promoted via k8s.io#9787/#9788) | ✅ #140993 closed 07-30 after greens. |
capz-windows: Projected PodCertificate ... mTLS [Conformance] — 35/35 fails since 07-21 |
Test went Conformance with PodCertificateRequest GA | Test used partial FQDN server.<ns>.svc; Windows treats dotted names as FQDN, skips search suffixes |
k/k#140921 — use ClusterDNSDomain FQDN — merged 07-28 |
✅ #140992 closed 08-03 |
| ci-kubernetes-e2enode-windows-master + presubmit: all 12 CPUManager specs since 07-17 | Empty AffinityCpus via CRI; suspected WS2022 July CU VM image |
Never definitively root-caused | Recovered by 07-30 (#140777 closed: "both have recovered!") — possibly VM image rollover or windows-testing#582 (containerd 2.3.3) | ✅ recovered; cause uncertain |
| sig-windows node e2e | Broken by officially-built containerd | Under investigation | test-infra#37506 — revert to nightly containerd — merged 07-21 | ✅ mitigated |
| Job(s) | What was broken | Root cause | Fix | Status now |
|---|---|---|---|---|
| Serial node e2e lanes (incl. both EC2 serial lanes) | Kubelet panic at startup since 07-22 | PodLevelResourcesFixDefaulting set while PodLevelResources disabled → gate-dependency validation panic |
k/k#140853 (ndixita) — merged 07-23 (#140848); related #140850/#140514 | ✅ panic gone (amd64 serial-EC2 lane has a different remaining failure — §4.1) |
| ci-kubernetes-e2e-node-canary | Suite timeouts: 1283 specs consuming 97%+ of the 75m ginkgo budget (worst 4396s/4500s) | Suite growth; pass/fail was timing noise | test-infra#37547 — 90m→120m (explicit stopgap; sharding is the durable fix) — merged 07-26 (k/k#140943) | ✅ 13+ consecutive greens through 08-03 |
| ci/pull node-e2e-containerd-alpha-features (+slow) | Round 1: unrecognized feature gate: PerPodPIDLimit (gate added to config before k/k implementation merged). Round 2: WAS/GenericWorkload gates broke lanes |
Hand-maintained named gate lists drift against k/k master | test-infra#37510 revert (07-22, #140813); #37526 (07-23, #140887); durable fix: #37529/#37537 — switch to AllAlpha=true,AllBeta=true + api/all=true |
✅ structurally prevented |
| ci-node-crio-kubelet-serial | Timeouts/instability | [Slow]/[Disruptive] tests in main serial lane |
test-infra#37527 — skip + new *-slow-disruptive lanes (07-24); containerd equivalent #37558 (07-29) |
✅ |
| Node e2e COS lanes | (proactive) cos-125 lacks containerd 2.2 CRI stats (ListPodSandboxMetrics) | — | test-infra#37596 cos-125→cos-129 master lanes; k/k#141049 GCE family → cos-129-lts | ✅ merged 07-30/07-31 |
| Job(s) | What was broken | Root cause | Fix | Status now |
|---|---|---|---|---|
| ci-kubernetes-kind-network-nftables-ipv6 | nft coredump: Assertion failed: dreg < ctx->reg_low (netlink_linearize.c:911) |
kube-proxy nftables single-endpoint DNAT rule form crashes nft on IPv6 (exposed when netlink mode went default in #137536) | k/k#140960 (adrianmoisey) — merged 07-28T11:11Z (#140951) | ✅ First green 07-28T11:24Z (13 min after merge); 7 consecutive daily greens |
| ci-kubernetes-e2e-kind-alpha-beta-features-canary (StatefulSet suites) | StatefulSet e2e failing (30-build triage cluster) | Dead upstream images: registry.k8s.io/galera-install:0.1 unpublished; docker.io/library/java:openjdk-8-jre removed |
k/k#140982 (soltysh) — delete mysql-galera + zookeeper suites — merged 07-29 (#140910) | ✅ (triage cluster is trailing-window residue) |
| Job(s) | What was broken | Root cause | Fix | Status now |
|---|---|---|---|---|
| ci-kubernetes-verify-master-canary | verify-gofmt failed only on arm64 (185 failed builds in the 2-week triage window) | go1.26.5 gofmt formats a CSILimits literal differently per arch |
k/k#141075 — arch-stable literal — merged 07-31 (#141074) | ✅ mostly green 08-03 (9 of last 10) |
| ci-kubernetes-unit-fieldsv1string | Perma-red since 07-09 — package didn't compile under the fieldsv1string build tag |
Test used FieldsV1{Raw:...}; Raw doesn't exist under that tag |
k/k#141066 — use NewFieldsV1 — merged 07-31 |
✅ green since fix (7 of last 8) |
| ci-kubernetes-unit-dependencies | TestCelCostStability failing since 07-26 |
Job tests against latest deps; cel-go v0.30.0 changed cost accounting (intentional upstream) | k/k#141063 — pin cel-go v0.29.2; real bump deferred to v1.38 (#141058, open) | ✅ 8 consecutive greens |
integration arm64/race/ppc64le: TestInterPodAffinityPreemption (blocking board) |
Flaking since 07-28 | Resource contention with neighboring podgroup tests | k/k#141048 (BenTheElder) — split into own package — merged 07-30 (#141027) | ✅ issue closed "fixed by #141048" |
pull-kubernetes-integration: TestPodGroupPreemption |
Cycle-state pollution between podgroup post-filters | — | k/k#140871 — merged 07-24 | 🟡 #140875 closed but closure contested (huww98: ordering root cause may remain) |
publishing-bot smoke: TestEventSeriesWithEventSinkImplRace |
Blocked staging-repo publishing | client-go EventBroadcaster goroutine change too risky to fix-forward near RC | Revert k/k#140990 — merged 07-28 (#140859) | ✅ |
| kubelet podcertificate unit: synctest deadlock panic | deadlock: main bubble goroutine has exited |
Goroutine leak inside synctest bubble | Revert k/k#140826 — merged 07-23; re-land tracked in #140545 (open) | ✅ mitigated |
TestComputePodActionsWithInitContainers/...RestartRules unit flake |
Shared runtime manager leaked actuatedState between subtests | — | k/k#140821 — merged 07-22 | ✅ |
| apidiff verify lane | Incompatible changes: ...SharedInformerFactory.Lifecycle: added after EvictionRequest added a whole new API group |
apidiff rules didn't allow brand-new groups | k/k#140931 — merged 07-30 | ✅ |
Downstream compile break: FakeCustomStore no longer satisfied cache.Store |
Ecosystem test code failed to compile | #134827 added Bookmark/LastStoreSyncResourceVersion w/o updating the fake |
k/k#140966 + 1.34 pick #141001 | ✅ |
Conformance: ServiceAccountIssuerDiscovery fails on ES384 clusters |
Test image hardcoded RS256/ES256 | — | k/k#141012 (liggitt) — sync full algorithm list — merged 07-29 (#141010) | ✅ |
| DRA 500-node kops scale lanes: pod startup latency up since ~07-03 | Perf regression flaking ci-kubernetes-e2e-kops-aws-500-node-dra-* |
Suspected ResourceClaim mutation-cache informer events | Revert k/k#140886 — merged 07-24 | 🟡 helped, but #141046 + #140877 remain open (§4) |
| IPVS session-affinity de-flake | affinity-timeout tests can't work on IPVS (min timeout 2min) | Test/design mismatch; IPVS deprecated | test-infra#37522 — skip on IPVS lanes, enables lower timeout to de-flake iptables/nftables | ✅ |
The claims above were re-verified today directly, not taken from PR bodies:
- ci-kubernetes-cross-canary — walked 666 builds of prow history (34 pages): healthy 2h cadence through 07-21T08:00Z (last good SUCCESS), then 568 ABORTED + 31 PENDING + 2 FAILURE, first SUCCESS 2026-08-01T20:25:12Z (build 2083650139615924224), then 21/21 SUCCESS through 08-03T12:37Z. The fix (#37598) merged 08-01T16:59Z — ~3.5h merge-to-green.
- ci-kubernetes-e2e-ec2-alpha-enabled-default — 7 consecutive FAILURE through 08-01T22:05Z start → 7 consecutive SUCCESS from 08-02T00:12Z (first run after paws#583 merged).
- ci-kubernetes-e2e-ec2-alpha-features — the fix sequence is visible in run history: red for months (AMI 403) → green 07-31→08-01 (after paws#582 + test-infra#37586) → 4 fails 08-02 (the
No ssh-able nodesregression exposed by paws#583) → green from 08-02T23:06Z (after paws#584). 3 consecutive greens at check time. - ci-containerd-node-e2e-ec2 — 14 consecutive SUCCESS since 07-31 (post paws#578).
- ci-kubernetes-e2e-node-canary — 13 consecutive SUCCESS through 08-03T12:36Z (post #37547).
- ci-kubernetes-kind-network-nftables-ipv6 — FAILURE 07-27T11:24 → SUCCESS 07-28T11:24 (fix merged 07-28T11:11Z) → 7/7 daily greens since.
- ci-kubernetes-unit-dependencies — 8/8 SUCCESS since 08-02T08:45Z. ci-kubernetes-unit-fieldsv1string — green since 07-31 fix (7 of last 8; job was red every run 07-09→07-31).
- TestGrid (summary JSON, 08-03 ~13:35 UTC): blocking board 0 failing; gce-cos-master-default 9/9 with 13212/13212 cells; Conformance-EC2 master 9/9 and arm64 10/10; capz-windows-master passing; ec2-arm64-ubuntu-master-containerd 13050/13050 cells.
- k8s-triage (
failure_data.json, 08-03 11:43 UTC): no clusters at all for ci-kubernetes-cross-canary (aborted runs upload no junit — see §5), gce-cos "Up" failure builds all date to the 07-31 window, thekube_aws_rsacluster (13 builds) stops at the 08-02 fix. - k8s-metrics (
failures-latest.json, 08-03 12:05 UTC): among 168 continuously-failing jobs, the only one with failing_days ≤ 14 is agentic-sandbox-burst-100 (8 days) — i.e., no lingering breakage from the freeze window remains in the continuous-failure list.
| Change | Why it matters | PRs |
|---|---|---|
Alpha node lanes now run --service-feature-gates=AllAlpha=true,AllBeta=true --runtime-config=api/all=true instead of hand-maintained gate lists |
Eliminates the failure class that broke alpha lanes twice in one week (PerPodPIDLimit, WAS gates) | #37529, #37537 |
Serial lanes split: [Slow]/[Disruptive] moved to dedicated *-slow-disruptive lanes (CRI-O + containerd) |
Stops timeout starvation; keeps main serial signal fast | #37527, #37558 |
gce-cos release-blocking periodics interval ladder: master 1h→2h; release branches 6h/8h/12h/24h via fork-per-release-periodic-interval |
Cost/capacity for 4 lanes × 4 branches without losing master signal | #37607, #37606 |
| All kubemark jobs deleted (2,475 lines, 1.33–1.36 + scalability) + experimental/duplicate scale jobs deleted | "Freeing up credits for more modern scale jobs" | #37559, #37549 |
| Scalability job policy rewrite: one experiment at a time, kops harness mandatory, approval process | Cloud budget guardrails | #37509 |
Spyglass build-log lens now highlights Ginkgo [FAILED] |
Faster human triage on prow.k8s.io | #37581 |
kops scale jobs: node-exporter everywhere, PROMETHEUS_SCRAPE_ETCD=true, preset refactor |
Observability for 100/5000-node runs | #37533, #37534, #37538, #37551 |
| EKS build cluster on on-demand instances (Karpenter) | Pod stability for jobs now living there (cross-canary, sig-network) | k8s.io#9765 |
| sig-network jobs → EKS cluster; SCTP coverage restored (kernel modules available) | Un-skips [Feature:SCTPConnectivity] (k/k#140489) |
#37507, #37508, #37555 |
DRA: ResourceHealthStatus exclusions lifted, canary → regular lanes |
Coverage for beta-since-1.36 gate | #37519, #37523 |
| 1.33 job cleanup (out of support): DRA + jobset jobs dropped | Capacity hygiene (#37584 tracks the rest) | #37579, #37566 |
| snoopdb image bump v20250906→v20260803 (conformance gate) | Refresh for the APISnoop pipeline (see #141065 in §4.4) | #37606 |
| VPA release-branch image builds in CI (no more laptop builds) | Supply-chain hygiene | #37546 |
| OOM right-sizing for neighbors: etcd grpcproxy 6Gi, cloud-provider-openstack bumps, kueue CPU | Fewer infra-red lanes org-wide | #37518, #37570, #37552 |
| Job / area | Evidence (2026-08-03) | Notes / suggested owner |
|---|---|---|
| pull-kubernetes-unit-windows-master failing on ALL PRs | k/k#140900 (07-24): VM provisioning can't install OpenSSH (WU 0x80244022) | Presubmit-blocking. sig-windows + sig-testing |
| gce-master-scale-correctness | 3 of 10 (30%) recent runs; only FAILING tab on the informing board; related Up-flake #140752 | sig-scalability |
| ci-containerd-node-e2e-serial-ec2 (amd64) | 10+ consecutive FAILURE through 08-03T08:52Z; triage: summary_test.go:451 Eventually timeout (18 builds) |
Distinct from the fixed arm64 THP issue; likely the PSI memory-pressure follow-up flagged in paws#585. sig-node. Also #141036 (arm64-ubuntu-serial bring-up) |
| ci-containerd-node-arm64-e2e-serial-ec2 | paws#585 merged 08-03T11:45Z — watch next runs to confirm; then fix k/k OOMKiller test limits (15Mi is inside runtime overhead margin) | dims/sig-node |
| gce-agentic-sandbox-continuous-burst-100 | 0% since 07-26 — the only post-freeze continuous failure in k8s-metrics (8 days) | Failure is inside clusterloader2 per #37548; agent-sandbox team |
| Windows-on-GCE master jobs (e2e-windows-containerd-gce-master, e2e-windows-win2022-...) | 4 big triage clusters: run-e2e.sh failures (161 builds), implicit-nonroot-uid timeouts (156), hostname -f not supported (215 across two clusters), gce-pd-csi Windows volumeID failures (148) |
The hostname -f cluster is the same busybox issue fixed on capz — the fixed images need to reach these jobs: #138975, #141085. sig-windows |
| ci-kubernetes-integration-master-s390x | 92 failed builds: TestLocalSubjectAccessReview ... etcd-client retrying of unary invoker failed |
IBM/s390x maintainers + sig-api-machinery |
| ci-kubernetes-e2e-kind-alpha-beta-features-race | Triage cluster f31edca5ed1862b4f788 (21 builds): kubelet DATA RACE in pkg/kubelet/allocation |
Real bug signal from the race lane — deserves a k/k issue (none found). sig-node |
| ci-minikube-docker-crio-linux-x86 | 10 triage clusters × ~84 builds each (TestStartStop, TestAddons, …) | minikube maintainers |
| ci-kubernetes-coverage-e2e-gci-gce | Flakiest CI job this week (11 flakes, 47.6% consistency) | sig-testing |
| Issue | What | Created |
|---|---|---|
| #141138, #141136, #141133 | LoadBalancers ExternalTrafficPolicy:Local flakes (3 issues, filed today) | 08-03 |
| #141085 | sig-windows-gce Security Context tests failing | 07-31 |
| #141073 | Restartable init container liveness probe flake (capz-windows) | 07-31 |
| #141072 | test/integration/dra.all flaking | 07-31 |
| #141059 | TestDRA OptionalNodeOperations podgroup flake | 07-30 |
| #141046 + #140877 | Pod startup latency up since 07-03; 500-node DRA lanes flaky (revert #140886 helped; new failure path in FastFillPodStart) | 07-30 |
| #140979 | cacher unit panic via leaked watch goroutine | 07-28 |
| #140942 | Ensure Credential Pulled Images still flaking (swap-ubuntu-serial) | 07-26 |
| #140924 | TestValidateEvictionStatusUpdate flaky on Windows | 07-24 |
| #140893, #140423, #140149 | WorkloadAwarePreemption / podgroup scheduling flakes | Jul |
| #140762 | TestVolumeAttachLimitExceededCleanup (unit-windows) — fix PR #140763 is open and needs lgtm/approve | 07-20 |
| #140760 | TestExternalJWTSigningAndAuth go-jose crypto error | 07-20 |
| #140701 | DRA ubuntu tests OOM, not running at all | 07-18 |
| #140644 | HugepageAwareMemoryReporting flake | 07-16 |
| #140642 | gce-master-scale-performance-5000 Up flake | 07-16 |
| #140598 | blocking board: integration-master scheduler plugins flake | 07-15 |
| #140545 | podcertificate synctest re-land (post-revert) | 07-14 |
| #140823 / #140352 | volumemanager reconciler flakes — watch whether #141062 (merged 08-03) helps | 07-22 / 07-09 |
| #140251 | csi-hostpath volume data source [Serial] | 07-06 |
| #140187 | Container restart e2e timeouts on Hyper-V Windows | 07-02 |
| Older, still open | #140070 (sig-cli client-go), #138975 (windows-gce hostname), #138973 (compat-version jobs dirty git), #138679 (TestEventSeries) | Jun–Jul |
Pre-existing, not freeze fallout — but they're the backlog behind "everything else is green":
| days | Job |
|---|---|
| 276 | cloud-provider-azure-conformance-windows-capz |
| 253 / 239 | ci-kubernetes-e2e-azure-dra-scalability (+ -with-workload) |
| 252 | periodic-cluster-api-addon-provider-helm-e2e-workload-upgrade-main |
| 244 / 243 | e2e-kops-aws-nftables-deb11, e2e-kops-grid-cilium-eni-rocky9-k33-ko33 |
| 226–241 | 15× e2e-kops-grid-{calico,flannel,kubenet}-{rhel10,rocky10}arm64-* (blocked on distros publishing AMIs to all regions — needs outreach, per #37587) |
| 1090 / 1080 / 594 | pull-kubernetes-node-arm64-e2e-containerd-{serial-,}ec2-canary, pull-kubernetes-node-kubelet-serial-containerd-alpha-features (canary/experiment lanes — decide: fix or delete) |
- AKS build-cluster root fix not filed:
infra/azure/terraform/k8s-infra-prow-build/aks.tfstill hasos_disk_size_gb=100+kubelet_disk_type="OS"— any disk-heavy job scheduled there will hit the same eviction loop cross-canary did. Needs a k8s.io PR (kubelet_disk_type="Temporary"to use the 440GiB NVMe, or bigger OS disk) + a tracking issue. - Revert test-infra#37595 (COS image pin on pull-sig-storage-local-static-provisioner-e2e) now that k/k#141082 is in
ci/latest. - paws#585 verification (arm64 THP) + the stated follow-ups: relax k/k OOMKiller 15Mi limits; upstream reports to runc (glibc tunable for init) and glibc (no per-process opt-out).
- apisnoop-conformance-gate red since 07-23 (#141065):
getLifecycleAPIGroupuntested (EvictionRequest API),pending_eligible_endpoints.yamlnot consumed; conformance-test PR #140897 was closed unmerged. Today's snoopdb bump (#37606) refreshes the pipeline — check whether the gate needs more. - test-infra#37603 (pull-kubernetes-e2e-ec2-alpha-canary presubmit — validation vehicle for kubetest2-ec2 changes) — open, needs review.
- #140875 closure contested — re-check podgroup preemption flake stats in a week.
- test-infra hygiene backlog: #37584 (finish 1.33 job removal), #37561 (SSH env-var simplification), #37560 (k8s-triage time-range filter), #37540.
- AWS account rename
k8s-infra-e2e-aws-playground-01→k8s-infra-node-e2e-tests(suggested in k8s.io#9790) and the packer instance-profile credentials oddity noted in paws#582. - Watch:
ci-cloud-provider-aws-e2e-with-kubernetes-master(needs a few more post-8Gi cycles),verify-master-canaryresidual flake, #141119 (kubelet eviction manager disables disk-pressure signals after transient failure — possibly relevant to build-cluster behavior).
- Aborted/evicted runs are invisible to our triage stack. cross-canary was down 11.5 days with zero k8s-triage signal (no junit uploaded) and no flake-metric entry; only job-history walking exposed it. Ideas: alert on N consecutive non-SUCCESS-non-FAILURE results; alert on ProwJobs pending > X hours (the stuck job had generation 869!); consider
error_on_eviction: truefor release-critical jobs so failures become visible; have crier/GCS upload a synthetic junit for aborted runs so triage can cluster them. - Named feature-gate lists in job configs drift against k/k master. Broke the alpha node lanes twice in one week. The
AllAlpha=true,AllBeta=truepattern (#37529/#37537) should be the default for every alpha/beta lane that intends "everything on". - Single-image-family assumptions are fragile.
gcloud compute images list --filter=family:returns all active images; one COS publishing hiccup took out 51 jobs for a day. k/k#141082 fixedcluster/gce/util.sh; audit other harnesses (windows GCE scripts, kops, node-e2e image config) for the same pattern — preferdescribe-from-family. - Suite growth silently eats timeout budgets. node-canary was at 97% of its ginkgo budget on green runs; CRI-O/containerd serial lanes were timing out. We now split Slow/Disruptive lanes and bumped budgets, but the durable fix is sharding + a metric/alert for "suite runtime vs budget > 85%".
- External image/tag dependencies rot. Staging registry pruning deleted the gcb-docker-gcloud tag k/k builds depended on;
java:openjdk-8-jreandgalera-installvanished upstream and broke StatefulSet suites; busybox'shostname.exesilently lacked a flag the tests needed. Ideas: pin builder images by digest, keep test-image deps on registry.k8s.io only, and a periodic "image liveness" check for every image referenced by e2e manifests. - OS/toolchain upgrades carry multi-layer landmines — canary lanes earn their keep. Ubuntu 26.04 brought Python 3.14 (
pipesremoval) and glibc 2.43 (aarch64-only malloc THP → OOMKiller test breakage); go1.26.5 gofmt output diverged by architecture;package_upgrade: trueagainst an old AMI melted boot times. Stagger upgrades through canary lanes first (this is exactly how these were caught), and pin AMIs/serials explicitly. - Byte budgets need presubmit guards. EC2 user-data is capped at 16384 bytes and we're within ~342 bytes of it after trimming — a size check in provider-aws-test-infra presubmits would turn the next overflow into a review comment instead of a red board.
- Janitors must be partition-tolerant. One missing IAM permission aborted the entire AWS sweep for a day (nothing got cleaned). boskos#262's continue-on-error is the right pattern; same review is worth doing for the GCP janitor.
- Spend capacity where the signal is. This window retired kubemark, deduplicated scale jobs, added a scalability-job approval policy, and spread release-branch periodic intervals (1h/2h master, 6h→24h ladder by branch age) — that budget belongs to lanes people actually watch.
- Triage-tooling gaps found while doing this work: Spyglass now highlights
[FAILED](#37581); k8s-triage still needs a time-range filter (#37560) — the 2-week clustering window keeps "fixed" failures looking alive for days (several §4 clusters are trailing residue); generic buckets likeerror during e2e-up.shneed splitting by event window before attribution.
- GitHub: all merged PRs in kubernetes/kubernetes 07-22→08-03 (102; ~37 CI-relevant), kubernetes/test-infra (first-parent log #37498–#37607), kubernetes/k8s.io, kubernetes-sigs/provider-aws-test-infra (#576–#585), plus issue timelines (
closedByPullRequestsReferences). - TestGrid:
https://testgrid.k8s.io/<dashboard>/summary?json=1for sig-release-master-blocking, sig-release-master-informing, amazon-ec2, google-gce (2026-08-03 ~13:35 UTC). - k8s-triage:
https://storage.googleapis.com/k8s-triage/failure_data.json(Last-Modified 2026-08-03 11:43 UTC; 5,817 clusters; ~2-week window). - k8s-metrics:
failures-latest.json,flakes-latest.json,job-health-latest.json(Last-Modified 2026-08-03 12:04–12:05 UTC) from https://storage.googleapis.com/k8s-metrics/. - Prow: job-history walks (inline
allBuildsJSON) for cross-canary (666 builds), the six EC2/node lanes, both unit jobs, nftables-ipv6, agentic-sandbox, verify-canary. - Items marked uncertain or contested are exactly that; everything else traces to a linked PR/issue/dataset.
sig-release-master-blocking · sig-release-master-informing · amazon-ec2 · google-gce · k8s-triage · k8s-metrics failures · job health · Prow