feat(analysis): routing-composition diagnostic (empirical model/role usage, collapse detection) - #412
Merged
James-CUDA merged 1 commit intoJul 23, 2026
Conversation
This was referenced Jul 22, 2026
fix(analysis): resolve unresolved merge-conflict markers in __init__ (package failed to import)
#405
Merged
…actually picks) The submission gate audit_head_routing_diversity (gate 10) inspects a head's WEIGHTS and warns when the agent logit rows collapse. That is a static check on W alone; it says nothing about what the head does on real inputs. A head with diverse-looking weights can still, on a given benchmark, route almost everything to one model or never call the Verifier. trinity.analysis.routing_composition answers that empirically from a coordinator's decision log — the (model, role) picks it actually made. Per benchmark and for the pooled union it reports each model's and role's usage share, which pool models / roles were never used, the normalized entropy of the model distribution (1.0 uniform, 0.0 one model), and a collapsed flag when one model's share is >= a threshold. A collapsed router is a valid but usually poor submission (it ignores the query); a head that never routes to the Verifier can't use the accept/revise loop — both are worth seeing offline before a training run. scripts/routing_composition_report.py is the CLI (exit non-zero if any router collapsed). Pure stdlib/math over (model, role) pairs. Adds synthetic unit tests.
philluiz2323
force-pushed
the
feat/routing-composition-analysis
branch
from
July 23, 2026 11:39
257e948 to
839594c
Compare
James-CUDA
force-pushed
the
feat/routing-composition-analysis
branch
from
July 23, 2026 11:40
839594c to
e45b613
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
General improvement
What does this PR do?
Closes #411.
Adds
trinity.analysis.routing_composition— an offline diagnostic for what the coordinator actually routes to, complementing the submission gateaudit_head_routing_diversity(which inspects head weights statically; this reads the empirical decisions).From a coordinator's decision log — the
(model, role)picks it made across each benchmark's turns — it reports, per benchmark and for the pooled union:scripts/routing_composition_report.pyis the CLI (exit non-zero if any benchmark's router collapsed). Pure stdlib/math over(model, role)pairs — no torch, no network, no GPU.Why is it needed?
A head with diverse-looking weights can still route almost everything to one model on a given benchmark, or never call the Verifier — the static gate can't see that. A collapsed router is a valid but usually strategically poor submission (it ignores the query), and a head that never routes to the Verifier can't use the accept/revise loop. This surfaces both offline, for free, before a training run.
Example
Note on CI
maincurrently does not import —src/trinity/analysis/__init__.pyhas unresolved merge-conflict markers (SyntaxError), which reds the whole suite regardless of this change. This PR is only the 3 new files and deliberately does not touch__init__.py; it goes green as soon as the init fix lands (#402 / #405). Locally, with that fix applied,tests/test_routing_composition.pyis 10 passed.Checklist
pytest tests/test_routing_composition.py— 10 passed locally (shares/entropy/collapse/unused, input-shape coercion incl. mapping/bare-model/Role-enum, empty-entry skipping, per-benchmark + pooled union, render). Depends on the__init__init-fix for CI collection (see above).ruff check src/— clean on the new files.mypy src/—Success: no issues found.trinity.analysis.turns_monotonicityand its report/tests).