Skip to content

feat(analysis): routing-composition diagnostic (empirical model/role usage, collapse detection) - #412

Merged
James-CUDA merged 1 commit into
James-CUDA:mainfrom
philluiz2323:feat/routing-composition-analysis
Jul 23, 2026
Merged

feat(analysis): routing-composition diagnostic (empirical model/role usage, collapse detection)#412
James-CUDA merged 1 commit into
James-CUDA:mainfrom
philluiz2323:feat/routing-composition-analysis

Conversation

@philluiz2323

Copy link
Copy Markdown
Contributor

Type

  • General improvement — bug fix, docs, refactor, or infrastructure (welcome but earns no TAO)

General improvement

What does this PR do?

Closes #411.

Adds trinity.analysis.routing_composition — an offline diagnostic for what the coordinator actually routes to, complementing the submission gate audit_head_routing_diversity (which inspects head weights statically; this reads the empirical decisions).

From a coordinator's decision log — the (model, role) picks it made across each benchmark's turns — it reports, per benchmark and for the pooled union:

  • each model's and each role's (Thinker/Worker/Verifier) usage share,
  • which pool models / roles were never used,
  • the normalized entropy of the model distribution (1.0 = uniform, 0.0 = one model), and
  • a collapsed flag when one model's share is at/above a threshold (default 0.90).

scripts/routing_composition_report.py is the CLI (exit non-zero if any benchmark's router collapsed). Pure stdlib/math over (model, role) pairs — no torch, no network, no GPU.

Why is it needed?

A head with diverse-looking weights can still route almost everything to one model on a given benchmark, or never call the Verifier — the static gate can't see that. A collapsed router is a valid but usually strategically poor submission (it ignores the query), and a head that never routes to the Verifier can't use the accept/revise loop. This surfaces both offline, for free, before a training run.

Example

| benchmark | n | model shares                         | role shares                  | entropy | collapsed |
| math500   | 4 | glm-5p2 0.50, deepseek 0.25, minimax 0.25 | worker 0.50, thinker 0.25, verifier 0.25 | 0.95 | no  |
| mmlu      | 4 | glm-5p2 1.00                         | worker 1.00                  | 0.00    | YES |
| union     | 8 | glm-5p2 0.75, deepseek 0.12, minimax 0.12 | ...                      | 0.67    | no  |

collapsed (one model >= threshold) on: mmlu
roles never used (union): —

Note on CI

main currently does not import — src/trinity/analysis/__init__.py has unresolved merge-conflict markers (SyntaxError), which reds the whole suite regardless of this change. This PR is only the 3 new files and deliberately does not touch __init__.py; it goes green as soon as the init fix lands (#402 / #405). Locally, with that fix applied, tests/test_routing_composition.py is 10 passed.

Checklist

  • Tests pass: pytest tests/test_routing_composition.py — 10 passed locally (shares/entropy/collapse/unused, input-shape coercion incl. mapping/bare-model/Role-enum, empty-entry skipping, per-benchmark + pooled union, render). Depends on the __init__ init-fix for CI collection (see above).
  • Lint passes: ruff check src/ — clean on the new files.
  • Type check passes: mypy src/Success: no issues found.
  • New code follows existing style (mirrors trinity.analysis.turns_monotonicity and its report/tests).
  • Public functions have docstrings and type annotations.

…actually picks)

The submission gate audit_head_routing_diversity (gate 10) inspects a head's
WEIGHTS and warns when the agent logit rows collapse. That is a static check on W
alone; it says nothing about what the head does on real inputs. A head with
diverse-looking weights can still, on a given benchmark, route almost everything
to one model or never call the Verifier.

trinity.analysis.routing_composition answers that empirically from a coordinator's
decision log — the (model, role) picks it actually made. Per benchmark and for the
pooled union it reports each model's and role's usage share, which pool models /
roles were never used, the normalized entropy of the model distribution (1.0
uniform, 0.0 one model), and a collapsed flag when one model's share is >= a
threshold. A collapsed router is a valid but usually poor submission (it ignores
the query); a head that never routes to the Verifier can't use the accept/revise
loop — both are worth seeing offline before a training run.

scripts/routing_composition_report.py is the CLI (exit non-zero if any router
collapsed). Pure stdlib/math over (model, role) pairs. Adds synthetic unit tests.
@philluiz2323
philluiz2323 force-pushed the feat/routing-composition-analysis branch from 257e948 to 839594c Compare July 23, 2026 11:39
@James-CUDA
James-CUDA force-pushed the feat/routing-composition-analysis branch from 839594c to e45b613 Compare July 23, 2026 11:40
@James-CUDA
James-CUDA merged commit 08059d3 into James-CUDA:main Jul 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Offline routing-composition diagnostic: what model/role the coordinator actually picks

2 participants