A layered explainer, written to be learned.
A non-monetary, for-fun AI experiment that predicts the 2026 World Cup. This document explains the new selective-prediction gate for the knockout stage: why it exists, how it works, and why it is honest about being a leaderboard strategy rather than a magic accuracy boost.
- Permalink (Gist): https://gist.github.com/hungson175/89fb01faf00c7e929e9aff7ddbc2980d
- Status: ENABLED for live use — wc-reviewer PASS and WC Coordinator GO on 2026-06-30.
"What did you change for the knockouts?"
"We kept the same v0.3 ensemble but added a reject option: for every knockout match the model computes its own expected RPS and the disagreement among Elo, Dixon–Coles and GBDT. If the match looks like a coin-flip or the models fight each other, we don't submit. The final four matches are always played. Backtests on group-stage data show this cuts mean binary RPS by roughly one-third to one-half on the matches we do enter. It is a leaderboard-average strategy, not a claim that the forecasts themselves got smarter."
One breath even shorter: Skip the knockout coin-flips, play the clearer calls, and always play the final four.
The honest punchline: We are not predicting better; we are choosing which predictions count.
- Level 0 — the 30-second pitch
- Level 1 — the 3-minute version
- Level 2 — the full explainer (~15 min)
- 2.1 — Why the knockout stage is different
- 2.2 — Selective prediction / reject-option theory
- 2.3 — Difficulty signals for a knockout match
- 2.4 — The selection function
- 2.5 — Group-stage backtest
- 2.6 — Implementation and guardrails
- 2.7 — Honest caveats
- 2.8 — What this means for WC Kimi going forward
- Appendix — mini-glossary
The situation. ClawCup's public leaderboard reports the average RPS over matches an agent actually predicts. If you skip a match, it is not counted in your denominator. That means a perfectly calibrated forecaster can raise its reported average by abstaining on matches where it expects to score badly — even though its true skill is unchanged.
The idea. We treat each knockout match as a binary "home advances / away advances" problem and ask: "If our probabilities are true, what RPS do we expect to get?" For a binary forecast with probability p, the expected RPS is p(1-p) (Gneiting & Raftery, 2007). This is highest near a coin-flip and lowest for heavy favourites. We skip the high-expected-RPS matches.
The second signal. We also look at ensemble disagreement: the standard deviation of the advance probabilities produced by Elo, Dixon–Coles and GBDT. When the three models disagree, the case is epistemically harder (Kendall & Gal, 2017).
The gate. For each non-final-four knockout match:
- Compute
eRPS = p_home_adv * p_away_adv. - Compute
δ = std([adv_elo, adv_dc, adv_gbdt]). - Predict if
eRPS <= 0.24andδ <= 0.18. - If too few matches are selected, relax to
eRPS <= 0.2475. - Always predict the final four (semi-finals, 3rd-place, final).
The result. On 49 group-stage matches converted to the same binary scoring, the gate predicts ~65 % of matches and cuts mean binary RPS from 0.167 to 0.119 (-29 %). A stricter gate (eRPS <= 0.2275) predicts ~47 % and cuts RPS by -47 %.
The constraints. More than 5 knockout predictions required; prefer more coverage when options are similar. Final four mandatory. Base v0.3 unchanged. Independent WC reviewer sign-off required before live use.
Group-stage matches have three outcomes: home win, draw, away win. Knockout matches have two: a team advances or it does not. The ClawCup scoring metric is therefore binary RPS:
RPS_binary = (p_home_adv - outcome)^2
where outcome = 1 if the home team advances, 0 otherwise.
This changes the unit of measurement but not the underlying uncertainty. Football is still low-scoring and high-variance; the better team can dominate xG and lose on penalties. What changes is the leaderboard mechanics. The public board averages your RPS only over the matches you predict. If you do not submit for a match, it does not hurt your average. That creates a strategic option: abstain when your expected loss is high.
This is not unique to ClawCup. Forecasting tournaments have long recognised that leaderboard averages can be gamed by selective participation (Aldous, 2019; ForecastBench difficulty-adjusted Brier, 2025). The honest response is to make the selection rule transparent and model-based rather than ad-hoc.
A selective predictor is a pair (f, g): a base predictor f and a selector g that decides whether to output the prediction or abstain (Chow, 1970).
For a proper scoring rule like RPS, the optimal reject strategy ranks inputs by their conditional expected risk and rejects the highest-risk fraction (Franc, Průša & Voráček, 2023). A proper uncertainty score — any score that preserves the risk ordering — is enough to build the optimal selector.
For binary RPS with calibrated probability p, the conditional expected risk is exactly:
E[RPS | p] = p * (1-p)^2 + (1-p) * p^2 = p(1-p)
This is maximised at p = 0.5 and minimised at the extremes. It means the hardest matches are the coin-flips, and the easiest are the heavy favourites. This is the score we use.
From the v0.3 1X2 probabilities [p_home, p_draw, p_away] we first compute binary advance probabilities:
p_home_adv = p_home + 0.5 * p_draw
p_away_adv = p_away + 0.5 * p_draw
p_home_adv, p_away_adv /= (p_home_adv + p_away_adv)
This treats a 90-minute draw as a 50/50 coin-flip in extra time / penalties, which is the standard conversion.
Then we compute two difficulty signals.
eRPS = p_home_adv * p_away_adv
Because p_home_adv + p_away_adv = 1, this equals p(1-p) where p is the larger advance probability. It is the model's own expected RPS if its probabilities are true. Higher eRPS → harder match.
v0.3 is an ensemble of Elo, Dixon–Coles and GBDT. Each component has its own advance probability:
adv_elo = elo_probs[0] + 0.5 * elo_probs[1]
adv_dc = dc_probs[0] + 0.5 * dc_probs[1]
adv_gbdt = gbdt_probs[0] + 0.5 * gbdt_probs[1]
δ = std([adv_elo, adv_dc, adv_gbdt])
If the three diverse models disagree, the match is epistemically harder: the available information supports more than one story. Empirically, ensemble disagreement is one of the strongest simple abstention signals across domains (Black et al., 2022; Hamidieh et al., 2026).
H = eRPS + δ
Both terms push in the same direction: high H means "this match is hard."
def select_knockout_match(p_home, p_draw, p_away, component_advs,
max_eRPS=0.24,
max_disagreement=0.18,
fallback_eRPS=0.2475,
min_predictions=10,
already_selected=0,
is_final_four=False):
home_adv = p_home + 0.5 * p_draw
away_adv = p_away + 0.5 * p_draw
s = home_adv + away_adv
home_adv /= s
away_adv /= s
confidence = max(home_adv, away_adv)
eRPS = home_adv * away_adv
disagreement = float(np.std(component_advs))
hardness = eRPS + disagreement
if is_final_four:
return True, {"confidence": confidence, "eRPS": eRPS,
"disagreement": disagreement, "reason": "final_four"}
if eRPS <= max_eRPS and disagreement <= max_disagreement:
return True, {"confidence": confidence, "eRPS": eRPS,
"disagreement": disagreement, "reason": "primary_gate"}
if already_selected < min_predictions and eRPS <= fallback_eRPS \
and disagreement <= max_disagreement:
return True, {"confidence": confidence, "eRPS": eRPS,
"disagreement": disagreement, "reason": "fallback_gate",
"fallback": True}
return False, {"confidence": confidence, "eRPS": eRPS,
"disagreement": disagreement, "reason": "too_hard"}The threshold eRPS <= 0.24 corresponds roughly to confidence >= 0.60. The fallback eRPS <= 0.2475 corresponds roughly to confidence >= 0.55.
We converted 49 group-stage matches (whose logs contain full component probabilities) to the binary "home advances / away advances" scoring and applied the gate.
| Rule | Predicted | Mean binary RPS | Improvement |
|---|---|---|---|
| Predict all | 49 | 0.1673 | — |
eRPS <= 0.2275 |
23 | 0.0893 | -46.6 % |
eRPS <= 0.24 |
32 | 0.1188 | -29.0 % |
eRPS <= 0.2475 |
40 | 0.1466 | -12.4 % |
eRPS <= 0.24 and δ <= 0.18 |
29 | 0.1079 | -35.5 % |
eRPS <= 0.2275 and δ <= 0.18 |
22 | 0.0887 | -47.0 % |
The primary rule (eRPS <= 0.24) predicts about two-thirds of matches and improves mean RPS by ~29 %. This is the chosen default because the user prefers more coverage when options are similar.
New files:
scripts/ko_selector.py— core selection function, input validation, and auditable batch planner.scripts/backtest_ko_selector.py— reproduces the group-stage proxy backtest table.tests/test_ko_selector.py— unit tests.
Modified files:
scripts/submit_ko_prediction.sh— live--select-logpath now requires an approved--plan <plan.json>, validates component vectors, and fails closed on missing/invalid evidence..kimi-code/skills/predict/SKILL.md— knockout workflow now uses the batch planner.CLAUDE.md/AGENTS.md— new methodology rule and reviewer sign-off requirement.docs/wiki/model/knockout-selection.md— distilled knowledge.docs/raw/2026-06-30-knockout-selection-research-synthesis.md— research notes.
Coverage enforcement (batch plan):
- Final four always predicted.
- Default minimum of 10 knockout predictions; primary gate → fallback gate → coverage fill until the minimum is reached.
- Hard floor of 6 predicted knockout matches (>5 as required by the user).
- Postcondition: total predicted (selected + already submitted) must satisfy both floors when enough matches remain; otherwise the planner raises
CoverageError.
Input validation:
- 1X2 probabilities must be 3 finite values in
[0,1]summing near 1. - Live path requires all three component vectors (
elo_probs,dc_probs,gbdt_probs) and validates each as a proper 1X2 vector. - Invalid logs fail closed.
Testing:
pytest -qpasses (76 tests after reviewer fixes).- Backtest reproduces the reported numbers.
- Leaderboard artefact, not skill. The improvement is in the reported average over selected matches. It does not make the underlying forecasts more accurate.
- In-sample thresholds. The
0.24/0.2475thresholds were chosen by inspecting group-stage results. They may be slightly overfit. - Calibration dependence. The gate assumes v0.3 probabilities are reasonably calibrated. If the model is overconfident, the gate will fail.
- Small sample. 49 group-stage matches is not a huge training set. Standard errors on mean RPS matter.
- Final-four exposure. The four heaviest-weighted matches are played regardless of hardness. If they go badly, the average suffers.
- No published football-KO benchmark. The direct evidence comes from WC Kimi's own backtest and adjacent literature, not a peer-reviewed knockout-stage study.
- Live submissions use an auditable batch plan produced by
scripts/ko_selector.py --plan logs/ko/m*.json --output data/approved_ko_plan.json. - The plan validates every log (1X2 probs + all three component vectors) and fails closed on missing/invalid evidence.
- The plan enforces postconditions: final four selected; total predicted (selected + already submitted) must exceed 5 and meet the configured minimum when enough matches remain.
- Skipped matches are logged with diagnostics for post-tournament reflection.
- The base v0.3 ensemble is unchanged; only the submission decision is gated.
- Live deployment is currently BLOCKED pending wc-reviewer re-review and WC Coordinator go/no-go.
- RPS (Ranked Probability Score): Ordered squared-error for probabilistic forecasts. For binary knockouts it equals the Brier score.
- eRPS: Expected RPS under the model's own distribution. For binary
p,eRPS = p(1-p). - Ensemble disagreement (δ): Standard deviation of component-model advance probabilities.
- Selective prediction: A predictor that may abstain rather than issue a low-confidence forecast.
- Reject option: The classical name for abstention in classification.
- Coverage: Fraction of matches for which a prediction is submitted.
- Proper scoring rule: A scoring rule maximised in expectation by reporting true probabilities.
- Final four: Semi-finals, 3rd-place match, and final — always predicted.
Built as WC Kimi: a self-improving, timestamped, non-monetary World Cup 2026 forecasting experiment. No betting; no stakes; no bookmaker names.