feat: eval-as-a-gate — score sessions against named criteria and fail CI on regression

[Siddhant-K-code]

Problem

agent-trace captures what an agent did. The missing layer is: was it good enough? Teams running agents in CI have no way to assert that a session met quality criteria — cost ceiling, no errors, task completed, tool call count within bounds — and block a merge if it didn't.

agent-strace compare and diff exist for manual inspection. What's missing is a declarative scoring layer that runs unattended and produces a pass/fail signal.

---

Proposed solution

agent-strace eval — score a session against named criteria defined in .agent-evals.yaml.

# Score the latest session
agent-strace eval

# Score a specific session
agent-strace eval SESSION_ID

# Score and fail if below baseline
agent-strace eval --baseline .agent-evals-baseline.json --tolerance 0.05

Config format: .agent-evals.yaml

evals:
  - name: cost-ceiling
    scorer: cost_usd
    threshold: 0.50
    fail_on: above

  - name: no-errors
    scorer: error_count
    threshold: 0
    fail_on: above

  - name: task-completed
    scorer: session_status
    expected: completed
    fail_on: not_equal

  - name: tool-efficiency
    scorer: redundant_read_ratio
    threshold: 0.10
    fail_on: above

  - name: context-saturation
    scorer: max_context_fill_ratio
    threshold: 0.85
    fail_on: above

Output

Eval results — session abc123
──────────────────────────────────────────────────────
Eval                  Score      Threshold   Result
──────────────────────────────────────────────────────
cost-ceiling          $0.31      $0.50       ✅ pass
no-errors             0          0           ✅ pass
task-completed        completed  completed   ✅ pass
tool-efficiency       0.04       0.10        ✅ pass
context-saturation    0.91       0.85        ❌ fail
──────────────────────────────────────────────────────
Overall: FAIL (1/5 evals failed)
Exit code: 1

Baseline regression mode

# Save current scores as baseline
agent-strace eval --save-baseline .agent-evals-baseline.json

# On next run: fail if any score regresses by more than 5%
agent-strace eval --baseline .agent-evals-baseline.json --tolerance 0.05

GitHub Actions integration

The reusable GitHub Action (already shipped) wires directly to this:

- uses: Siddhant-K-code/agent-trace@gha-v1
  with:
    config: .agent-evals.yaml
    baseline: .agent-evals-baseline.json
    tolerance: "0.05"

Built-in scorers

Scorer	What it measures
cost_usd	Total session cost
error_count	Number of error events
session_status	completed / killed / timeout
redundant_read_ratio	Fraction of file reads that are redundant
tool_call_count	Total tool calls
context_fill_ratio	Max context window utilization
duration_seconds	Wall-clock session length
lint_violations	Count of lint rule violations

Custom scorers via Python callable in config (advanced).

---

Acceptance criteria

• agent-strace eval reads .agent-evals.yaml and scores the latest session
• All built-in scorers implemented and tested
• --baseline and --tolerance flags work for regression gating
• --save-baseline writes current scores to a JSON file
• Exit code 1 on any eval failure (for CI integration)
• GitHub Actions step summary output (markdown table)
• --format json for machine-readable output
• Docs: .agent-evals.yaml reference and CI integration guide