feat(eval): add strict local-run gates and provenance receipts by barryroodt · Pull Request #194 · supabase/evals

barryroodt · 2026-08-13T15:28:43Z

Stack 2/4 for #128. Base: #193.

This keeps pnpm eval as the single entrypoint for local and CI runs:

pnpm eval -- <eval...> --experiment <id> --strict [--mcp <path>]

--strict turns error-class skips into exit 1, and CI enables it.
Existing runner paths own experiment validation and per-agent readiness. Judge keys and declared SKILL.md files fail before spend.
--mcp uses the override mechanism from feat(core): launch a local mcp build via SUPABASE_MCP_SERVER_PATH #193.
Canonical results in results/<experiment>/<eval>.json include provenance and use atomic replacement. Corrupt partial results rerun.

The separate pnpm local CLI, results-local/, and published-result helpers are gone. pnpm eval -- list remains the discovery command.

Verification: smoke 14/14, core 114/114, sandbox 39/39, typecheck + format passed.

The pre-existing test:framework suite reached 8 PASS markers, then stopped at investigate-security-001-public-table because OPENAI_API_KEY was absent. A credentialed run still needs:

pnpm --filter @supabase-evals/framework test:framework

vercel · 2026-08-13T15:28:49Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored	Preview	Aug 18, 2026 1:19pm

mattrossman · 2026-08-17T21:09:39Z

I think some of the complexity here comes from legacy behavior where there's an opportunity to step back and simplify.

At one point (when evals primarily ran in a shared process) it was desirable for evals to fail passively, so that it wouldn't terminate the whole workflow in CI. That's why a missing credential still shows SKIP + exit 0 instead of failing the run.

Since then, we've moved to a GitHub Actions matrix with parallel processes, and further still to Vercel Sandboxes. In both cases, we only run one eval pair per runner/sandbox. Now we actually prefer behavior where bad evals fail loudly so we know which pairs need to be re-run.

I think this translates into a couple simplification opportunities:

Keep pnpm eval the entrypoint, and put the gates there instead of a second entrypoint
Add a --strict flag to pnpm eval that turns failures into exit 1 instead of passive SKIP, and enable this in CI

That along w/ the other notes above could shave a couple hundred LOC off the PR.

@Rodriguespn

Stack 1/4 for #128. Base: `main`. (`compare` moved out of the chain into #195, parked on #194.) Re-lands the core of #118 without the submodule, so a workspace can test an unpublished [`mcp-server-supabase`](https://github.com/supabase/mcp) change against the evals without publishing to npm. Carries the review fixes from that thread (canonicalize the base once, receipt-visible mounts) and the tests. Thanks @Rodriguespn. Set `SUPABASE_MCP_SERVER_PATH` and `createConfig` launches that build with `node` instead of the pinned package with `npx`. It takes a repo dir, a package dir, or a direct `.js`/`.mjs`/`.cjs` entrypoint, and resolves relative paths against the evals checkout root rather than the process CWD. ## Three details worth your attention These are what the tests pin, and each exists because a CLI agent spawns its MCP server **inside** the sandbox container, which inherits nothing from the harness process. - **`node`, not `process.execPath`.** The host's absolute node path does not exist in-container, so both sides resolve `node` through `PATH`. - **The base is canonicalized once, and the entry derived from it.** Docker resolves bind-mount sources against the daemon's view, so command and mount must agree on one filesystem view. Realpathing the entry separately would let a symlinked `dist/` resolve outside the mounted directory. - **The mount covers the build's whole git toplevel**, not just `dist/`, because the build is unbundled and needs its `node_modules` at runtime. Hence `SandboxMount` and `supabaseMcpServerMounts()`, threaded `local-stack-runtime` -> `agent-environment` -> `docker-sandbox` as `--volume host:host:ro`. Mounting at the identical real path is what lets one command config work on the host and in the container. ## Nothing changes when the var is unset `supabaseMcpServerMounts()` returns `[]`, and I checked the config rather than assuming: ran `createConfig({apiUrl})` against `main` and against this branch with the var cleared, and diffed. Identical, matching md5 `875658e9…`, same `npx` command and same seven args in the same order. ## How to verify ```bash git submodule update --init && pnpm install # zero cost pnpm --filter @supabase-evals/core test # 114/114 pnpm --filter @supabase-evals/sandbox test # 39 passed, 4 skipped pnpm typecheck && pnpm format:check # end to end (costs one agent run): make a visible change to a tool response in # your mcp checkout, `pnpm install && pnpm build` there, then SUPABASE_MCP_SERVER_PATH=<mcp-checkout> pnpm eval --eval investigate-db-001-table-row-counts --experiment claude-haiku-4.5 ``` Unset the var and the config is byte-identical to `main`, which is what the new core tests pin. ## Verified - core 114/114 (10 new), sandbox 39 passed, 4 skipped (counts after the rebase onto current `main`) - `tsc --noEmit` and `biome check` clean

…nd gates Runs an eval against YOUR inputs and records which world it measured. Nothing here mutates a git tree, so it is safe to run beside in-flight work in another worktree. pnpm local run <eval...> [--experiment <id>] [--runs N] [--mcp <path>] pnpm local experiments Every run writes results-local/<eval>.treatment.json: host sha and dirty-file count, plus the --mcp override's own git state when one is passed. `--mcp` is sugar over the SUPABASE_MCP_SERVER_PATH resolution added in the previous commit, and it accepts either the mcp monorepo root or the server package dir. The gates are the point. The harness SKIPs an experiment with exit 0 when credentials are missing, so without them a misconfigured world surfaces only after a paid agent run: - eval metadata parses via parseEvalMarkdown, and the eval dir exists - the experiment exists, listing what is available when it does not - ANTHROPIC_API_KEY is set AND non-empty. Set-but-empty is its own case because `node --env-file` never overrides an existing var, so a stray `export ANTHROPIC_API_KEY=` silently shadows .env (observed live). - OPENAI_API_KEY is present for judge-scored evals. The judge is an OpenAI grader regardless of the agent under test, and a missing key would otherwise surface after the agent run finished. - the experiment's declared skills exist in this checkout, or the treatment would quietly run skill-less - the --mcp path exists and is actually built, warning on fixture drift when the build's version differs from the pin published-log.ts parses the `git log` record for a published export. It is split out so the smoke suite can exercise the merge-commit case directly: `%P` expands to every parent, so a space-split would put a second sha where the timestamp belongs, and origin/main has no merge touching an export for an end-to-end check to catch it. `pnpm test:local` is a zero-cost smoke suite: 11 checks, no model call and no docker. LOCAL_EVAL_CMD fakes the run, LOCAL_NO_FETCH skips the fetch, and LOCAL_RESULTS_ROOT redirects every write into a temp sandbox so a real in-flight run cannot be clobbered. Verified: smoke 11/11, tsc and biome clean.

barryroodt · 2026-08-18T13:55:28Z

I think some of the complexity here comes from legacy behavior where there's an opportunity to step back and simplify.

At one point (when evals primarily ran in a shared process) it was desirable for evals to fail passively, so that it wouldn't terminate the whole workflow in CI. That's why a missing credential still shows SKIP + exit 0 instead of failing the run.

Since then, we've moved to a GitHub Actions matrix with parallel processes, and further still to Vercel Sandboxes. In both cases, we only run one eval pair per runner/sandbox. Now we actually prefer behavior where bad evals fail loudly so we know which pairs need to be re-run.

I think this translates into a couple simplification opportunities:

Keep pnpm eval the entrypoint, and put the gates there instead of a second entrypoint

Add a --strict flag to pnpm eval that turns failures into exit 1 instead of passive SKIP, and enable this in CI

That along w/ the other notes above could shave a couple hundred LOC off the PR.

Thanks for the feedback.

I've now implemented this shape.

pnpm eval -- --strict turns error-class skips into exit 1, and the refresh workflow enables it in 4604995.
moved the judge-key check into run-eval, so plain runs now fail before spend when a required judge key is missing (c100322).
unknown CLI arguments now fail loudly instead of being silently ignored (a61e6f1).

One boundary: a published pair that does not exist in the current checkout stays a visible SKIP, including under strict mode. That lets an older branch run after main publishes a newer eval; malformed pairs present in the checkout still fail.

barryroodt requested a review from a team August 17, 2026 10:15

barryroodt force-pushed the local-runner-run branch from 368b67a to 0e4b4ca Compare August 17, 2026 10:15