feat(framework): score committed solutions without an agent run

[czenko]

Closes AI-1080

Stacked on #205. Review that one first, or read this diff against its branch.

Problem

Checking whether a scorer's checks behave takes an agent run. That run is nondeterministic and costs a model call, so you can't separate a scorer defect from agent variance, and a check that passes broken work is hard to catch before the eval ships. #205 tells contributors to score example solutions. Nothing scored them.

Solution

• Adds pnpm eval:solution -- --eval <id>, which boots the eval's environment, copies a directory from its solutions/ over the workspace, and runs the scorer with no agent. --solution <name> scores one.
• Copies the overlay between localDir and supabase start, so a solution's migrations apply on boot with no per-eval apply step. That is the whole mechanism.
• Moves eval discovery and run setup to eval-discovery.ts, so the agent and solution entry points can't drift on what an eval is.
• Writes nothing to results/. Exported workspaces land in .solution-runs/, gitignored.
• Ships two solutions for build-cli-002-declarative-schema and the expected-failure list that goes with them.
• Documents the command in README.md and in the guidance from docs: cover example solutions and check design for scorers #205.

Local-stack evals only. A tools eval's starting state is a seeded hosted project, not a workspace to copy files into, so it needs its own apply path.

Manual testing

1. Stop any local Supabase stack, since the run needs ports 54321-54329.

2. Score both solutions:

   pnpm eval:solution -- --eval build-cli-002-declarative-schema

   Output matches solutions/README.md. green fails only the tool-call check, migration-only also fails the schema-file check:

   green
     ❌ supabase db diff used to generate the migration
     ✅ schema file updated to include description column
     ✅ a new migration was generated for the change
     ✅ description column exists in the live database
     3/4 checks passed
   
   migration-only
     ❌ supabase db diff used to generate the migration
     ❌ schema file updated to include description column
     ✅ a new migration was generated for the change
     ✅ description column exists in the live database
     2/4 checks passed
   

3. Run git status results/. It is clean, and .solution-runs/build-cli-002-declarative-schema/ holds both exported workspaces.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored	Preview	Aug 14, 2026 7:58pm