Skip to content

feat(analysis): verify SPEC R3 (TRINITY > best multi-agent baseline) offline - #386

Merged
James-CUDA merged 1 commit into
James-CUDA:mainfrom
joaovictor91123:feat/verify-spec-r3-multi-agent-baseline
Jul 22, 2026
Merged

feat(analysis): verify SPEC R3 (TRINITY > best multi-agent baseline) offline#386
James-CUDA merged 1 commit into
James-CUDA:mainfrom
joaovictor91123:feat/verify-spec-r3-multi-agent-baseline

Conversation

@joaovictor91123

Copy link
Copy Markdown
Contributor

Fixes #385

What

docs/SPEC.md §1.3 lists R3 — "TRINITY > best multi-agent baseline (MoA / MasRouter / RouterDC / Smoothie)" (§4.2, Fig. 3) as a replication invariant, but nothing verified it (R5/R7/R8/R11/R12 have offline checks; R3 had none). R3 is the claim that separates a learned coordinator from the prior art it is measured against — routing better than any multi-agent ensembling/routing baseline is strictly stronger than beating a single model (R1) or random routing (R4).

This adds src/trinity/analysis/multi_agent_baseline.py (+ tests) — an offline check that reads per-benchmark TRINITY accuracy and a {baseline: score} map and reports whether TRINITY strictly beats the best baseline present, per benchmark and on the equally-weighted union.

API (mirrors the R11 coordinator_vs_llm module)

  • analyze_benchmark(trinity, {baseline: score}, *, benchmark, tol)MultiAgentComparison (best baseline named, margin, comparable, holds)
  • analyze_benchmarks({benchmark: (trinity, baselines)}) → per-benchmark rows + r3_holds, equal-weight union accuracies/margin, violations
  • render(...) → compact markdown report

Strict win (a tie is not a pass); the bar is the strongest baseline present; tolerant to missing TRINITY / non-numeric / absent baselines (excluded from the union, never crash); accepts pair tuples or {trinity, baselines} mappings. The four SPEC-named baselines are recognized for reporting but the set is open.

Verification

Pure numpy/stdlib — no torch, no network, no GPU. 13 offline tests; ruff + mypy clean on src/.

@James-CUDA
James-CUDA merged commit 0233a50 into James-CUDA:main Jul 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify SPEC R3 (TRINITY > best multi-agent baseline) offline

2 participants