Fold the eval-source workspace into evals (M2) - #118
Closed
barryroodt wants to merge 10 commits into
Closed
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
barryroodt
force-pushed
the
workspace-fold
branch
from
July 24, 2026 08:15
086371f to
a0f6e6a
Compare
barryroodt
force-pushed
the
workspace-fold
branch
from
July 24, 2026 12:05
a766305 to
e4a639c
Compare
SUPABASE_CONTENT_API_URL (or the contentApiUrl option) adds --content-api-url to the server args, pointing search_docs at an alternative docs index (e.g. a locally built one). Requires a server that understands the flag — the SUPABASE_MCP_SERVER_PATH build; inert otherwise. Retires the eval-workspace enabler patch of the same name.
Verbatim copy of the eval-workspace repo's scripts/, patches/, and manifest (minus the two evals enabler patches: the local-build override shipped with the submodule PR, and content-api-url just landed as a real commit), plus the mise task file at the root and gitignore entries for the glue's artifacts. Re-rooting the scripts for life inside evals is the next commits' job — this one is the reviewable copy boundary.
evals is the host now, not a managed clone: every evals/ path prefix drops, 'cd evals && pnpm eval' becomes a root pnpm eval, and the evals manifest entry + marker plumbing die. The mcp clone is replaced by the pinned submodules/mcp working tree (patches apply there as marker commits; submodule.submodules/mcp.ignore=all keeps that patch dirt out of host status/commits — pin bumps are a deliberate git add). supabase stays the opt-in sparse clone. Keychain service name is unchanged so existing stored keys keep working. Self-tests from the repo root: status.test 19/0, hooks.test 8/0 (real install/chain/reinstall lifecycle against the initialized mcp submodule), ab.test 22/0.
…ee self-test workflow workspace/README.md is the ops doc (bootstrap, the three loops, A/B, provenance, self-tests); the root README points at it. The self-test workflow runs the three shell-only suites on workspace/** PRs: no keys, no model spend, no docker.
- ab.test.sh cleanup could destroy a user's unstaged stdio.ts edit: the EXIT trap reverted the mcp fixture file unconditionally while the fixture block only runs on a CLEAN file - exactly the state where the revert nukes user work. Cleanup now only reverts what the test itself wrote (verified: planted user edit survives a skipped-fixture run). - fold demo/canary-eval (ab-demo died at cp: fixtures were never folded). - affected.ts emits source-selecting commands: mcp/server-wide paths get mise run mcp-eval; docs paths get SUPABASE_CONTENT_API_URL + mcp-eval (a bare 'mise run eval' measured production, not the local edit). - workspace/README.md documents the docs-loop measurement step. - workspace-selftest.yml: pin setup-node to the repo's SHA convention, node-version-file instead of an inline 22. - dedupe the paid docs-embed preamble (docs-embed-env.sh sourced by docs-index/docs-seed) and provenance's untracked-hashing (one hashUntracked helper for receipt + stamp). Self-tests: status 19/0, hooks 8/0, ab 22/0.
- canary EVAL.ts imports @supabase-evals/core (real package; was elided at runtime but broke type resolution) - SUPABASE_CONTENT_API_URL env fallback now applies only alongside the SUPABASE_MCP_SERVER_PATH override: a stray env var can no longer feed --content-api-url to the published npx server (which rejects unknown flags). Explicit contentApiUrl option stays unconditional. Tests adjusted, +1 covering the stray-env npx case. - affected.ts: skips an eval dir missing PROMPT.md instead of aborting; the docs command's '# needs:' hint moved to its own line - content-API URL literals cross-referenced at each site (4-way sync note) - PUBLISH_REPOS identity alias collapsed into PATCH_REPOS everywhere - mise.toml pins pnpm 10.24.0 (matches packageManager/lockfile) - status.sh drops the stale 'incl. in agent-skills' guard comment - ab.sh fullwidth parens -> ASCII; .gitignore double blank dropped - status.test.sh anchors on the exact 'Ready. Try:' banner - hooks.test.sh/ab.test.sh fail actionably on zero checks instead of passing vacuously when submodules are missing Verified: status 19/0, hooks 8/0, ab 22/0, core mcp-server 12/12, framework typecheck clean, affected smoke on a real docs path.
apply-patches verified 'state matches the canonical patch' two ways:
byte-exact tree-OID reconstruction for an existing marker commit, but a
grep-payload diff for a fresh build - the weaker check, blind to an
identical +/- line reattributed across files within a multi-file patch.
Extract patch_tree() (throwaway index: parent-tree + patch -> tree OID)
and compare tree identity on BOTH paths; the payload-diff mechanism is
deleted and its blind spot closes as a side effect.
Verified live: existing-commit path ('already committed'), fresh path
(marker dropped via rebase --onto, rebuilt through the new check);
hooks 8/0, ab 22/0, status 19/0.
1. mcp-build built the UNPATCHED server and orphaned the marker commit: it delegated to the root 'pnpm mcp:build', whose first step is another 'git submodule update --init' - on an initialized submodule that re-checks out the pin, discarding the marker apply-patches created a step earlier (and any user commits). The task now runs init -> install -> apply-patches -> build directly; the root script remains the evals-native unpatched flow. Verified: after mcp-build the marker is HEAD and the built stdio.js carries --content-api-url. 2. Arm receipts mislabeled the run: ab.sh passed RUN_ENV to the eval but not to the 'provenance --embed' step, so a local-build arm recorded mcp_override: null. The embed now runs under the same env; ab.test gains a receipt-records-override assertion (23/0). Found by a real cold-clone demo: GitHub clone -> setup -> unstaged mcp edit -> live A/B (baseline PASS 5/5 -> treatment FAIL 3/5, REGRESSED).
…e next release + pin bump
barryroodt
force-pushed
the
workspace-fold
branch
from
July 24, 2026 12:09
e4a639c to
25b5f73
Compare
barryroodt
marked this pull request as draft
July 27, 2026 07:21
Contributor
Author
|
Parking this alongside #119 while we redo the local-dev story TypeScript-first (cross-platform matters once this repo is open-sourced, and the bash + macOS-specific glue doesn't get us there). The core piece here (SUPABASE_MCP_SERVER_PATH + tests) is exactly what the new PR re-lands, minus the submodule, and the review feedback from this thread carries over. Branch stays for reference. |
barryroodt
added a commit
that referenced
this pull request
Aug 17, 2026
Re-lands the core of #118 without the submodule, so a workspace can test an unpublished mcp-server-supabase change against the evals without publishing to npm. Includes the review fixes from that thread (canonicalize the base once, receipt-visible mounts) and the accompanying tests. Thanks @Rodriguespn. When SUPABASE_MCP_SERVER_PATH is set, createConfig launches that build with `node` instead of the pinned package with `npx`. It accepts a repo dir, a package dir, or a direct .js/.mjs/.cjs entrypoint, and resolves relative paths against the evals checkout root rather than the process CWD. Three details that are not obvious, and are what the tests pin: - `node`, not process.execPath. A CLI agent spawns this command INSIDE the sandbox container, where the host's node path does not exist. Both sides resolve `node` through PATH. - The base is canonicalized ONCE and the entry derived from it. Docker resolves bind-mount sources against the daemon's view, so command and mount must agree on one filesystem view; realpathing the entry separately would let a symlinked dist/ resolve outside the mounted directory. - The mount covers the build's whole git toplevel, not just dist/, because the build is unbundled and needs its node_modules at runtime. Hence the new SandboxMount type and supabaseMcpServerMounts(), threaded through local-stack-runtime -> agent-environment -> docker-sandbox as `--volume host:host:ro`. Mounting at the identical real path is what lets one command config work on the host and in the container. Empty when the env var is unset, so nothing changes for existing runs. Verified: core 113/113 (10 new), sandbox 39 passed, tsc and biome clean. With the override unset, createConfig's output is byte-identical to before (same npx command, same args, same order).
barryroodt
added a commit
that referenced
this pull request
Aug 18, 2026
Re-lands the core of #118 without the submodule, so a workspace can test an unpublished mcp-server-supabase change against the evals without publishing to npm. Includes the review fixes from that thread (canonicalize the base once, receipt-visible mounts) and the accompanying tests. Thanks @Rodriguespn. When SUPABASE_MCP_SERVER_PATH is set, createConfig launches that build with `node` instead of the pinned package with `npx`. It accepts a repo dir, a package dir, or a direct .js/.mjs/.cjs entrypoint, and resolves relative paths against the evals checkout root rather than the process CWD. Three details that are not obvious, and are what the tests pin: - `node`, not process.execPath. A CLI agent spawns this command INSIDE the sandbox container, where the host's node path does not exist. Both sides resolve `node` through PATH. - The base is canonicalized ONCE and the entry derived from it. Docker resolves bind-mount sources against the daemon's view, so command and mount must agree on one filesystem view; realpathing the entry separately would let a symlinked dist/ resolve outside the mounted directory. - The mount covers the build's whole git toplevel, not just dist/, because the build is unbundled and needs its node_modules at runtime. Hence the new SandboxMount type and supabaseMcpServerMounts(), threaded through local-stack-runtime -> agent-environment -> docker-sandbox as `--volume host:host:ro`. Mounting at the identical real path is what lets one command config work on the host and in the container. Empty when the env var is unset, so nothing changes for existing runs. Verified: core 113/113 (10 new), sandbox 39 passed, tsc and biome clean. With the override unset, createConfig's output is byte-identical to before (same npx command, same args, same order).
barryroodt
added a commit
that referenced
this pull request
Aug 18, 2026
Stack 1/4 for #128. Base: `main`. (`compare` moved out of the chain into #195, parked on #194.) Re-lands the core of #118 without the submodule, so a workspace can test an unpublished [`mcp-server-supabase`](https://github.com/supabase/mcp) change against the evals without publishing to npm. Carries the review fixes from that thread (canonicalize the base once, receipt-visible mounts) and the tests. Thanks @Rodriguespn. Set `SUPABASE_MCP_SERVER_PATH` and `createConfig` launches that build with `node` instead of the pinned package with `npx`. It takes a repo dir, a package dir, or a direct `.js`/`.mjs`/`.cjs` entrypoint, and resolves relative paths against the evals checkout root rather than the process CWD. ## Three details worth your attention These are what the tests pin, and each exists because a CLI agent spawns its MCP server **inside** the sandbox container, which inherits nothing from the harness process. - **`node`, not `process.execPath`.** The host's absolute node path does not exist in-container, so both sides resolve `node` through `PATH`. - **The base is canonicalized once, and the entry derived from it.** Docker resolves bind-mount sources against the daemon's view, so command and mount must agree on one filesystem view. Realpathing the entry separately would let a symlinked `dist/` resolve outside the mounted directory. - **The mount covers the build's whole git toplevel**, not just `dist/`, because the build is unbundled and needs its `node_modules` at runtime. Hence `SandboxMount` and `supabaseMcpServerMounts()`, threaded `local-stack-runtime` -> `agent-environment` -> `docker-sandbox` as `--volume host:host:ro`. Mounting at the identical real path is what lets one command config work on the host and in the container. ## Nothing changes when the var is unset `supabaseMcpServerMounts()` returns `[]`, and I checked the config rather than assuming: ran `createConfig({apiUrl})` against `main` and against this branch with the var cleared, and diffed. Identical, matching md5 `875658e9…`, same `npx` command and same seven args in the same order. ## How to verify ```bash git submodule update --init && pnpm install # zero cost pnpm --filter @supabase-evals/core test # 114/114 pnpm --filter @supabase-evals/sandbox test # 39 passed, 4 skipped pnpm typecheck && pnpm format:check # end to end (costs one agent run): make a visible change to a tool response in # your mcp checkout, `pnpm install && pnpm build` there, then SUPABASE_MCP_SERVER_PATH=<mcp-checkout> pnpm eval --eval investigate-db-001-table-row-counts --experiment claude-haiku-4.5 ``` Unset the var and the config is byte-identical to `main`, which is what the new core tests pin. ## Verified - core 114/114 (10 new), sandbox 39 passed, 4 skipped (counts after the rebase onto current `main`) - `tsc --noEmit` and `biome check` clean
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #109 (retarget to main when it merges). Moves the eval-workspace glue into this repo, per the direction agreed in the workspace-layout Slack thread: the sources under test (docs, skills, MCP server) now live beside the harness that tests them.
What's here (commit per step)
packages/core):SUPABASE_CONTENT_API_URL/ acontentApiUrloption adds--content-api-urlto the server args, pointingsearch_docsat an alternative docs index. Requires a server that understands the flag (theSUPABASE_MCP_SERVER_PATHbuild; upstream PR: mcp#343); inert otherwise. Retires the last evals enabler patch.workspace/+ themisetask file at the root (Supabase convention) — the reviewable copy boundary, no adaptation.evals/path prefixes drop,cd evals && pnpm evalbecomes a rootpnpm eval, the evals manifest entry + marker plumbing die, and the mcp clone is replaced by the pinnedsubmodules/mcpworking tree (enabler patches apply there as marker commits;submodule."submodules/mcp".ignore=allkeeps patch dirt out of host status, so pin bumps are a deliberategit add). The supabase docs monorepo stays an opt-in sparse clone.workspace/README.mdis the ops doc (bootstrap, the three edit→eval loops, head-to-head A/B, provenance receipts); a free self-test workflow (shell + node stdlib, no keys/docker/model spend) runs onworkspace/**PRs.Verification
git clone→statusprints the exact bootstrap steps →setup.shreaches Ready in ~16s (install, selective submodule init, patch markers, keychain keys) → all three self-tests pass in the fresh clone.Not in scope
Archiving the eval-workspace repo (after this merges), the docs-monorepo submodule (M3, planned), and the pin-sync/canary companions.