Skip to content

Instantly share code, notes, and snippets.

@hungson175

hungson175/ko-selective-prediction-method.md

Last active June 30, 2026 01:57
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save hungson175/32d0484afa7f664898bbde7937621d25 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save hungson175/32d0484afa7f664898bbde7937621d25 to your computer and use it in GitHub Desktop.
Download ZIP
WC Kimi - Knockout Selection Experiment (selective prediction / abstention method)
Raw
ko-selective-prediction-method.md

WC Kimi – Knockout Selection Experiment

A "street-smart" abstention method for the remaining 2026 World Cup knockout matches.

1. What we are trying

The public leaderboard reports an average RPS over the matches an agent actually predicts. That means a predictor can improve its reported average by not playing the matches it expects to score badly on, even if its underlying model is unchanged. This is a leaderboard artefact, not a pure skill gain — but the user asked for an experiment in "street smart" selection, so we will test it honestly.

Hard constraints:

  • Predict more than 5 of the remaining knockout matches; prefer more coverage when the expected-RPS improvement is similar (Boss preference).
  • Always predict the final 4 matches (semi-finals, 3rd-place, final).
  • Keep the existing v0.3 ensemble as the base model; only add a selection gate.
  • Any methodology change must pass a no-regression backtest and receive independent WC reviewer sign-off before going live.

2. Scientific framing: selective prediction / reject option

This is the classic selective prediction or classification with a reject option problem:

  • Chow (1970): optimum rejection is based on the posterior probability of the most likely class.
  • Franc & Průša (2019, JMLR): a proper uncertainty score is enough to build the Bayes-optimal reject function.
  • Geifman & El-Yaniv (2019, ICML): SelectiveNet trains a network to optimise the risk-coverage trade-off.

For probabilistic sports forecasting the same principle applies: abstain when the forecast distribution is close to uniform and/or the models disagree. Under a proper scoring rule like RPS, a confident wrong call is penalised, but an uncertain wrong call is penalised even more on expectation.

3. How we define "hard"

For each knockout match we first convert the v0.3 1X2 probabilities into a binary "who advances" probability:

p_home_adv = p_home + 0.5 * p_draw
p_away_adv = p_away + 0.5 * p_draw
p_home_adv, p_away_adv /= (p_home_adv + p_away_adv)   # normalise

Then we compute three difficulty signals.

3.1 Binary confidence (c)

c = max(p_home_adv, p_away_adv)   # in [0.5, 1.0]

c near 0.5 is a coin-flip; c near 1.0 is a heavy favourite. Lower confidence → harder match.

3.2 Expected binary RPS under the model’s own distribution (eRPS)

eRPS = p_home_adv * (1 - p_home_adv)^2 + p_away_adv * (1 - p_away_adv)^2
     = p_home_adv * p_away_adv

For binary RPS this is the model’s expected own loss if its probabilities are true. Higher eRPS → harder match.

3.3 Ensemble disagreement (δ)

v0.3 is an ensemble of Elo, Dixon–Coles and GBDT. Each component has its own advance probability:

adv_elo  = elo_probs[0]  + 0.5 * elo_probs[1]
adv_dc   = dc_probs[0]   + 0.5 * dc_probs[1]
adv_gbdt = gbdt_probs[0] + 0.5 * gbdt_probs[1]
δ = std([adv_elo, adv_dc, adv_gbdt])

If the three models disagree strongly, the match is intrinsically harder or the available information is contradictory.

3.4 Composite hardness score

H = eRPS + δ

eRPS = p_home_adv * p_away_adv is the model’s own expected binary RPS; δ penalises internal model disagreement. A match is hard if H is large. Using eRPS instead of 1 - c makes the score directly interpretable in the same units as the leaderboard metric.

4. Backtest on the group stage (proxy)

We re-ran every group-stage match through the same binary conversion and applied the candidate selection rules. The sample is 49 group-stage matches whose prediction logs contain full component probabilities. Group-stage draws are treated as “home advances” so the task is comparable to a knockout advancement label; this is a proxy, not real historical KO validation.

Rule Predicted Mean binary RPS Improvement vs. predicting all
Predict all 49 0.1673 —
eRPS <= 0.2275 (≈ c ≥ 0.65) 23 0.0893 -47 %
eRPS <= 0.24 (≈ c ≥ 0.60) 32 0.1188 -29 %
eRPS <= 0.2475 (≈ c ≥ 0.55) 40 0.1466 -12 %
eRPS <= 0.24 and δ <= 0.18 29 0.1079 -35 %
eRPS <= 0.2275 and δ <= 0.18 22 0.0887 -47 %

The pattern is stable: skipping low-confidence and high-disagreement matches cuts the mean binary RPS by roughly one-third to one-half in this proxy.

5. Selection function and batch plan

Single-match logic:

def select_knockout_match(p_home, p_draw, p_away, component_advs,
                          max_eRPS=0.24, max_disagreement=0.18,
                          fallback_eRPS=0.2475, min_predictions=10,
                          already_selected=0, is_final_four=False):
    home_adv = p_home + 0.5 * p_draw
    away_adv = p_away + 0.5 * p_draw
    s = home_adv + away_adv
    home_adv /= s
    away_adv /= s

    confidence = max(home_adv, away_adv)
    eRPS = home_adv * away_adv
    disagreement = float(np.std(component_advs))
    hardness = eRPS + disagreement

    if is_final_four:
        return True, {"confidence": confidence, "eRPS": eRPS,
                      "disagreement": disagreement, "reason": "final_four_mandatory"}

    if eRPS <= max_eRPS and disagreement <= max_disagreement:
        return True, {"confidence": confidence, "eRPS": eRPS,
                      "disagreement": disagreement, "reason": "primary_gate"}

    need = max(0, min_predictions - already_selected)
    if need > 0 and eRPS <= fallback_eRPS and disagreement <= max_disagreement:
        return True, {"confidence": confidence, "eRPS": eRPS,
                      "disagreement": disagreement, "reason": "fallback_gate",
                      "fallback": True}

    return False, {"confidence": confidence, "eRPS": eRPS,
                   "disagreement": disagreement, "reason": "too_hard"}

Batch plan and coverage enforcement

Live submissions use plan_ko_predictions() in scripts/ko_selector.py:

python scripts/ko_selector.py --plan logs/ko/m*.json --output data/approved_ko_plan.json

The planner:

  1. Validates every remaining KO log (1X2 probs and all three component vectors).
  2. Selects final four first.
  3. Selects primary-gate matches sorted by hardness.
  4. Adds fallback-gate matches if needed to reach min_predictions.
  5. Adds coverage-fill matches (lowest remaining eRPS, then δ) if the minimum is still not met.
  6. Raises CoverageError if total predicted (selected + already submitted) is below hard_floor=6 or below min_predictions when enough matches exist.

This produces an auditable JSON plan consumed by the submission script:

scripts/submit_ko_prediction.sh --select-log logs/ko/m099.json --plan data/approved_ko_plan.json

6. What this means for the first four knockouts

Our already-submitted R32 predictions:

Match Home adv. Confidence c eRPS δ Primary gate (eRPS<=0.24, δ<=0.18)
m073 South Africa – Canada 0.503 0.503 0.249 0.149 Skip
m074 Brazil – Japan 0.502 0.502 0.250 0.273 Skip
m075 Germany – Paraguay 0.655 0.655 0.226 0.128 Predict
m076 Netherlands – Morocco 0.439 (away) 0.561 0.246 0.090 Fallback predict (eRPS<=0.2475)

So the gate would have played m075 on the primary rule and m076 on the fallback rule; m073 and m074 would be skipped. Going forward, we apply the gate to each new knockout match at submission time.

7. Honest caveats

  1. Leaderboard artefact, not true skill. Selection improves the average only because the skipped matches are excluded from the denominator. It does not make the underlying forecasts more accurate.
  2. In-sample threshold risk. The thresholds 0.24 / 0.18 were chosen by looking at group-stage results; they may be slightly overfit. The fallback mechanism (0.2475) and coverage fill are safety valves against being too picky.
  3. Small-sample proxy backtest. 49 group-stage matches with a draw→home proxy is not a huge or perfectly analogous validation set. Standard errors on the mean RPS numbers are meaningful.
  4. Final-four exposure. The four heaviest-weighted matches are played regardless of hardness. If they go badly, the average will still suffer.
  5. Live guardrails required. The gate must be run through the batch planner with validated component vectors; ad-hoc per-match selection can silently miss coverage or use missing disagreement evidence.

8. References

  • Chow, C. K. (1970). On optimum recognition error and reject tradeoff.
  • Franc, V., & Průša, D. (2019). On discriminative learning of prediction uncertainty. ICML; Franc, V., Průša, D., & Voracek, V. (2023). Optimal strategies for reject option classifiers. JMLR 24(11).
  • Geifman, Y., & El-Yaniv, R. (2019). SelectiveNet: A deep neural network with an integrated reject option. ICML.
  • Epstein, E. S. (1969). A scoring system for probability forecasts of ranked categories. J. Appl. Meteorol. (RPS).
  • WC Oracle layered explainer (allowed reference for this experiment): https://gist.github.com/hungson175/30be0144d833b7cfe80d0db0f0a0fd2b

This is a non-monetary, for-fun forecasting experiment. No betting, no stakes, no bookmaker names.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment