feat(leaderboard): surface the composite score-to-beat (the number that decides approval) - #397
Merged
James-CUDA merged 1 commit intoJul 22, 2026
Conversation
…benchmark targets trinity.leaderboard answers 'what score must a submission beat?' but only builds per-benchmark BenchmarkTargets from benchmarks.*. The crown is COMPOSITE, though: pr_eval evaluates one head on every benchmark and approves a submission only when its MEAN score clears the reigning competition.best_composite_score by win_margin. So the single number a miner must actually beat to be crowned -- best_composite_score + win_margin -- was surfaced nowhere; the per-benchmark best_scores a contributor saw do NOT decide approval. Add CompetitionTarget + summarize_competition(leaderboard): reads the competition object and reports score_to_beat = best_composite_score + win_margin (pr_eval's APPROVE threshold), the reigning king + per-benchmark breakdown, the baselines, and a 'reachable' flag (score_to_beat <= 1.0). Returns None for an older benchmarks-only leaderboard. scripts/leaderboard_targets.py now prints the composite headline above the per-benchmark reference rows (and includes it in --json). Pure stdlib -- no torch, no network. The committed leaderboard.json parses (seed: beat >= 0.02, no king yet). Offline tests only.
James-CUDA
force-pushed
the
feat/competition-score-to-beat
branch
from
July 22, 2026 13:02
9110b6f to
14a717b
Compare
This was referenced Jul 22, 2026
James-CUDA
pushed a commit
that referenced
this pull request
Jul 23, 2026
…ap-to-king summarize_competition (#397) tells a miner the composite score-to-beat, but not whether their own run clears it or WHERE it falls short. The competition is composite: pr_eval approves only when the mean of the per-benchmark scores clears the reigning best_composite_score by win_margin. Add trinity.competition_preflight: given the leaderboard + a miner's {benchmark: score}, compute the composite the same way pr_eval does (mean over the declared benchmarks, an unscored board counting as 0.0), decide go/no-go against king + win_margin, and report the per-benchmark delta versus the reigning king plus the weakest board -- so the miner sees which benchmark is dragging the composite down before spending a PR. scripts/preflight_composite.py exits non-zero when the submission would not win. Pure stdlib -- no torch, no network. Offline tests only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
trinity.leaderboardanswers "what score must a submission beat?" — but it only builds per-benchmarkBenchmarkTargets frombenchmarks.*. The crown is composite:pr_evalevaluates one head on every benchmark and approves a submission only when its mean score clears the reigningcompetition.best_composite_scorebywin_margin.So the single number a miner must actually beat to be crowned —
best_composite_score + win_margin— is surfaced nowhere. The per-benchmarkbest_scores a contributor sees do not decide approval. This is the same "the competition is composite" gap I just fixed instandings(#387) and hardened in the integrity verifier (#384).How
Add
CompetitionTarget+summarize_competition(leaderboard): it reads thecompetitionobject and reportsscore_to_beat = best_composite_score + win_margin—pr_eval's exact APPROVE threshold;baseline_best_single/baseline_randomreference points;reachableflag (score_to_beat <= 1.0) — a king at 0.99 with a 0.02 margin is effectively unbeatable.It returns
Nonefor an older benchmarks-only leaderboard (nocompetition), and tolerates missing / string / null fields.scripts/leaderboard_targets.pynow prints the composite headline above the per-benchmark reference rows:Scope
Pure stdlib — no torch, no network. Read-only; the committed
leaderboard.jsonparses (seed:beat >= 0.02, no king yet). New tests cover the threshold, the seed, the unbeatable case, the no-competitionNone, field coercion, and JSON round-trip; they exercise the newsummarize_competitionsymbol (absent onmain).ruff+mypyclean;pytest tests/test_leaderboard.pygreen (22 tests).Duplication note
Touches only
trinity/leaderboard.py+ its CLI/tests. The two open PRs (#377 definition-of-done, #373 R11 coordinator-vs-LLM) are both inanalysis/and touch neither — verified against their file lists.