feat(analysis): verify SPEC R11 (trained coordinator > LLM-as-coordinator) offline - #373
Closed
philluiz2323 wants to merge 1 commit into
Closed
Conversation
…ator) offline SPEC §1.3 lists R11 -- "Trained coordinator > LLM-as-coordinator" (Table 8) -- as a replication invariant, but nothing verified it (R5/R7/R12 have checks, R6 is in flight). R11 justifies training a tiny derivative-free head at all: a ~10K-param linear head should route better than prompting a full LLM to pick model/role. trinity.analysis.coordinator_vs_llm reads per-benchmark trained-coordinator (TRINITY) accuracy and the LLM-as-coordinator baseline accuracy and reports, per benchmark, the margin TRINITY - LLM and whether TRINITY wins, plus the R11 verdict on the equal-weight union (default: win on every benchmark; a union mode holds on the average). SPEC §6 notes the paper's LLM-as-coordinator average is 53.76 (Table 8). scripts/coordinator_vs_llm_report.py is the CLI (exit non-zero on violation), mirroring the R5/R7 report scripts. Pure numpy/stdlib. Adds tests.
Owner
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
General improvement
What does this PR do?
Closes #372.
Adds the offline verifier for SPEC §1.3 invariant R11 — "Trained coordinator > LLM-as-coordinator" (Table 8), which had no check (R5/R7/R12 do; R6 is in flight in #351).
src/trinity/analysis/coordinator_vs_llm.py— per benchmark, computes the marginTRINITY - LLM-as-coordinatorand whether TRINITY wins;analyze_benchmarksrolls up the R11 verdict on the equal-weight union (default: TRINITY must win on every benchmark, the SPEC reading;require_all=Falseholds on the average).scripts/coordinator_vs_llm_report.py— CLI reading{benchmark: {trinity, llm_coordinator}}, printing the report and exiting non-zero on a violation. Mirrorsscripts/turns_monotonicity_report.py(R7) / the merged R5 report.Pure numpy/stdlib — no torch, no network, no GPU.
Why is it needed?
R11 is the justification for training a tiny derivative-free head at all: a ~10K-param linear head optimized with sep-CMA-ES should route better than prompting a full LLM to pick model/role each turn. Without an offline check the claim can't be reported alongside the other invariants, and a regression that lets the LLM baseline catch up would go unnoticed. SPEC §6 notes the paper's LLM-as-coordinator average is 53.76 (Table 8), not the text's 64.14.
Example
Checklist
pytest tests/— addedtests/test_coordinator_vs_llm.py(9 cases: per-benchmark win/loss/tie, the require-all verdict, the union-average mode, key aliases + skipping of missing/non-numeric entries, empty input, and render). Purely additive (3 new files).ruff check src/— clean.mypy src/—Success: no issues found.trinity.analysis.turns_monotonicity/ the mergedper_question_best).