UUID: 6ec7ee26-e0a2-4bb3-a03a-3a57c15bba4e | Author: peterkay_86616 | Category: platform-eng | Difficulty: hard
Discord: https://discord.com/channels/1427397917685321919/1483839196408840232
Solid task — bug design, mock services, and decoys are all well-crafted. Three grading structure fixes needed, then re-eval on biggie. Bug set and setup are fine as-is.
When a task's setup.sh installs Helm releases with imagePullPolicy: Always in Nebula's air-gapped environment, the pods start successfully on idle machines — but under resource pressure (hosted infra, CI runners, concurrent containers), the registry pull timeout can trigger a crash-loop cascade that takes down ArgoCD's repo-server.
Discovered during review of broken-canary-gitops-migration-recovery (v22), where test-solution scored 0.0 despite the solution completing correctly.
Context: This document proves that the "StatefulSet Ordinal Disruption" task scenario (PostgreSQL replica WAL/pg_control corruption after pod eviction) cannot be hardened to appropriate difficulty for apex-arena evaluation. The diagnostic path from broken state to fix is a straight line with zero decision branching.
Method: We reproduced the exact broken state inside a running Nebula environment, then traced every diagnostic command an agent would run, capturing real outputs. Each step's output unambiguously points to exactly one next step.
When a grader.py times out during a hosted evaluation, the system records it as a successful run with score 0.0 rather than an error. This skews aggregate scores because timeout runs are included in the mean calculation instead of being excluded or flagged.
glitchtip-alert-webhook-failure (v17)e3b1049c-0aca-46d2-8c8c-0788475c22aa
The setup corrupts both format.json and xl.meta, but xl.meta corruption is detectable at runtime — so mc admin info on the running pod immediately shows data1=corrupt, data3=corrupt. Every agent (10/10) runs this command early, gets the unambiguous answer, and ignores the monitoring artifacts entirely. The diagnostic puzzle provides zero signal.
The fix has three parts, all low-effort changes to setup.sh:
format.json (not xl.meta), using valid JSON with wrong disk UUIDs (not random bytes). This blocks both mc admin info and filesystem inspection on the running pod.
Task: Cross-Service PVC Snapshot Orchestration | UUID: 59932490-21ef-4882-81c1-64a2052d8db1 | Version: 25
V25 mean score is 0.458 with grader bugs in AC2 and AC5 accounting for most failures. Once those are fixed, scores will likely rise above the 0.70 threshold. AC6 — currently 0/8 pass, also due to a grader bug — is the natural place to add genuine difficulty to compensate.
The current AC6 compares latest data timestamps across restored databases and checks they're within 30 seconds. This is nearly redundant with AC3 (which already validates snapshot timing) and fails today only because of the same MongoDB readiness bug as AC5. Once that's fixed, AC6 becomes a near-freebie, since the 30-second threshold is too generous to distinguish quiesced from unquiesced snapshots.