Skip to content

feat(competition): submission preflight go/no-go with per-benchmark gap-to-king - #408

Merged
James-CUDA merged 2 commits into
James-CUDA:mainfrom
galuis116:feat/competition-preflight
Jul 23, 2026
Merged

feat(competition): submission preflight go/no-go with per-benchmark gap-to-king#408
James-CUDA merged 2 commits into
James-CUDA:mainfrom
galuis116:feat/competition-preflight

Conversation

@galuis116

Copy link
Copy Markdown
Contributor

What does this PR do?

summarize_competition (#397) tells a miner the composite score-to-beat, but not whether their own run clears it, nor where it falls short. The competition is composite: pr_eval approves only when the mean of the per-benchmark scores clears the reigning best_composite_score by win_margin. So a miner needs to turn their own per-benchmark scores into that same composite and compare — before spending a PR.

How

Add trinity.competition_preflight: given the loaded leaderboard + a miner's {benchmark: score}, it

  • computes the composite the same way pr_eval does (mean over the competition's declared benchmarks, an unscored board counting as 0.0);
  • decides go/no-go against king + win_margin and reports the exact gap to close;
  • reports the per-benchmark delta versus the reigning king and the weakest board, so the miner sees which benchmark is dragging their composite down.

scripts/preflight_composite.py renders it and exits non-zero when the submission would not win:

$ preflight_composite.py --score math500=0.90 --score mmlu=0.90 --score livecodebench=0.75
- composite 0.8500 vs score-to-beat 0.9200 (king alice): would NOT win ❌
- shortfall: +0.0700 to close
| benchmark | my score | vs king |
| livecodebench | 0.7500 | +0.0500 |
- weakest board (improve here first): livecodebench (0.7500)

Scope

Pure stdlib — no torch, no network. New module + script + 10 offline tests; the substantive change touches no existing file. Companion to the merged competition-object toolkit (#384 integrity, #387 standings, #397 score-to-beat).

Note on CI / the analysis/init.py change in this diff

main currently has unresolved git merge-conflict markers in src/trinity/analysis/__init__.py (a botched merge → the package is invalid Python, so ruff/import fail on every branch). This PR carries the identical repair only to keep its own CI green — it's the same fix as the in-flight #402 / #405 and drops out on rebase once main is green. The preflight feature itself is the real content.

Duplication note

Verified against every open PR's file list: competition_preflight.py + its CLI/tests are unique to this PR; the only overlap is the shared analysis/__init__.py CI-unblock (transient).

…nblocks CI)

A botched merge left nested git conflict markers in trinity/analysis/__init__.py
(union_diagnostics vs oracle_column vs definition_of_done import blocks), so the
package is invalid Python and ruff/import fail on every branch. __all__ already
lists the symbols of all three modules, so all three import blocks are intended;
keep them. Stacked here only to green this PR's CI while the same repair lands via
James-CUDA#402/James-CUDA#405 -- it drops out on rebase once main is fixed.
…ap-to-king

summarize_competition (James-CUDA#397) tells a miner the composite score-to-beat, but not
whether their own run clears it or WHERE it falls short. The competition is
composite: pr_eval approves only when the mean of the per-benchmark scores clears
the reigning best_composite_score by win_margin.

Add trinity.competition_preflight: given the leaderboard + a miner's {benchmark:
score}, compute the composite the same way pr_eval does (mean over the declared
benchmarks, an unscored board counting as 0.0), decide go/no-go against king +
win_margin, and report the per-benchmark delta versus the reigning king plus the
weakest board -- so the miner sees which benchmark is dragging the composite down
before spending a PR. scripts/preflight_composite.py exits non-zero when the
submission would not win. Pure stdlib -- no torch, no network. Offline tests only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants