Skip to content

feat(analysis): verify SPEC R4 (TRINITY > random routing) offline - #407

Merged
James-CUDA merged 2 commits into
James-CUDA:mainfrom
joaovictor91123:feat/verify-spec-r4-random-routing
Jul 23, 2026
Merged

feat(analysis): verify SPEC R4 (TRINITY > random routing) offline#407
James-CUDA merged 2 commits into
James-CUDA:mainfrom
joaovictor91123:feat/verify-spec-r4-random-routing

Conversation

@joaovictor91123

Copy link
Copy Markdown
Contributor

Fixes #406

Stacked on #405 (the analysis/__init__.py merge-conflict fix). The first commit in this diff is #405's fix, included only so this PR's CI is green while main's analysis package is unimportable; it disappears once #405 merges. The change here is the R4 module + tests.

What

docs/SPEC.md §1.3 lists R4 — "TRINITY > random routing" (RLPR: 0.41 vs 0.32). eval.py checks it live, but there was no offline verifier module — R4 is the last §1.3 invariant without one (R1/R2/R3/R5/R7/R8/R9/R10/R11/R12/R13 are covered).

This adds src/trinity/analysis/random_routing.py (+ tests) — the basic sanity floor: does the trained head beat random model+role selection, per benchmark and on the equally-weighted union.

API (mirrors the merged R11 coordinator_vs_llm module)

  • analyze_benchmark(trinity, random_routing, *, benchmark, tol)RandomRoutingComparison (margin, comparable, holds)
  • analyze_benchmarks({benchmark: (trinity, random_routing)}) → per-benchmark rows + r4_holds, equal-weight union accuracies/margin, violations
  • render(...) → markdown report

Strict win (a tie is not a pass); tolerant to missing/non-numeric scores; accepts pair tuples or {trinity, random_routing} mappings (alias random).

Verification

Pure numpy/stdlib — no torch, no network, no GPU. 10 offline tests; ruff + mypy clean on src/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify SPEC R4 (TRINITY > random routing) offline

2 participants