Skip to content

feat(analysis): verify SPEC R8 (sep-CMA-ES > SFT > RS > REINFORCE) offline#382

Merged
James-CUDA merged 1 commit into
James-CUDA:mainfrom
joaovictor91123:feat/verify-spec-r8-optimizer-ranking
Jul 22, 2026
Merged

feat(analysis): verify SPEC R8 (sep-CMA-ES > SFT > RS > REINFORCE) offline#382
James-CUDA merged 1 commit into
James-CUDA:mainfrom
joaovictor91123:feat/verify-spec-r8-optimizer-ranking

Conversation

@joaovictor91123

Copy link
Copy Markdown
Contributor

Fixes #381

What

docs/SPEC.md §1.3 lists R8 — "sep-CMA-ES > SFT > RS > REINFORCE on all 4 tasks" (Table 4) as a replication invariant, but nothing verified it (R5/R7/R11/R12 have offline checks; R8 had none). The SPEC itself flags R8 as "a hypothesis to test, not a given" — the block-ε-separability that justifies sep-CMA-ES was measured on the paper's 7-agent representation, not our 3-model pool. R8 is the justification for a derivative-free ES outer loop at all: if RS or REINFORCE matched sep-CMA-ES, the evolutionary search wouldn't be earning its keep.

This adds src/trinity/analysis/optimizer_ranking.py (+ tests) — an offline check that reads per-task {optimizer: final_fitness} and reports whether the observed best→worst order matches the expected chain, which adjacent pairs invert it, and whether R8 holds on every scored task.

API (mirrors the R7 turns_monotonicity / R11 coordinator_vs_llm modules)

  • canonical_optimizer(name) → canonical sep_cmaes/sft/rs/reinforce (case/separator-insensitive; Sep-CMA-ES, random search, policy_gradient, …), else None
  • analyze_task({optimizer: fitness})OptimizerRanking (observed order, expected order among present, inversions, holds)
  • analyze_tasks({task: {...}}) → per-task rows + r8_holds (True iff ≥1 scored task and every scored task holds — "on all 4 tasks"), violations
  • render(...) → compact markdown report

Strict ordering (a tie is not a pass), tolerant to a subset of optimizers (chain checked among those present, in expected order), and to non-numeric/unknown entries (dropped, never crash).

Verification

Pure numpy/stdlib — no torch, no network, no GPU. 14 offline tests; ruff + mypy clean on src/.

@James-CUDA
James-CUDA merged commit fbeec3c into James-CUDA:main Jul 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify SPEC R8 (sep-CMA-ES > SFT > RS > REINFORCE) offline

2 participants