Skip to content

fix: keep LLM credentials out of the eval scorer environment #188

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mattrossman wants to merge 3 commits into
base: main
Choose a base branch
from
+64 −15
Open

fix: keep LLM credentials out of the eval scorer environment #188

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
62bb9a8
fix(framework): scope scorer subprocesses to an explicit env
mattrossman Aug 11, 2026
ec2af2e
fix(framework): supply Supabase env vars to scored vite/vitest runs
mattrossman Aug 11, 2026
0a7f481
fix(framework): replace agent-installed node_modules when linking
mattrossman Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 19 additions & 7 deletions apps/framework/harness/project-runner.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import {
existsSync,
mkdirSync,
readFileSync,
rmSync,
symlinkSync,
writeFileSync,
} from 'node:fs';
Expand All @@ -12,21 +13,32 @@ import type { CommandResult, VitestResult } from '@supabase-evals/core';

const __dirname = dirname(fileURLToPath(import.meta.url));
const ROOT = join(__dirname, '..');
const PROJECT_DB_URL = 'http://supabase-evals.local';
const PROJECT_DB_ANON_KEY = 'supabase-evals-anon-key';
const PROJECT_DB_JWT_SECRET = 'supabase-evals-dev-secret';
const PROJECT_ENV = {
VITE_SUPABASE_URL: PROJECT_DB_URL,
VITE_SUPABASE_ANON_KEY: PROJECT_DB_ANON_KEY,
};

// Vite/vitest resolve their own package (and the workspace's deps, e.g. react)
// by walking up from the workspace looking for a node_modules dir. Workspaces
// live under results/, outside ROOT, so link ROOT's node_modules in directly.
// The link replaces anything already there, since an agent that ran npm install
// in the sandbox exports its own node_modules.
function linkNodeModules(workspace: string) {
const link = join(workspace, 'node_modules');
if (!existsSync(link)) symlinkSync(join(ROOT, 'node_modules'), link, 'dir');
rmSync(link, { recursive: true, force: true });
symlinkSync(join(ROOT, 'node_modules'), link, 'dir');
}

export async function viteBuild(workspace: string): Promise<CommandResult> {
linkNodeModules(workspace);
return runNodeBin(
join(ROOT, 'node_modules', 'vite', 'bin', 'vite.js'),
['build'],
workspace
workspace,
PROJECT_ENV
);
}

Expand Down Expand Up @@ -63,7 +75,7 @@ export async function vitestRun(workspace: string): Promise<VitestResult> {
`--outputFile=${reportPath}`,
],
workspace,
{ SUPABASE_EVALS_WORKSPACE: workspace }
{ ...PROJECT_ENV, SUPABASE_EVALS_WORKSPACE: workspace }
);
const parsed = existsSync(reportPath)
? parseVitestReport(reportPath)
Expand All @@ -79,9 +91,9 @@ import { afterAll } from "vitest";
import { App, getAuthSchemaSql, SUPABASE_AUTH_HELPERS_SQL } from "@supabase/lite";
import { createPgliteConnection } from "@supabase/lite/pglite";

const PROJECT_DB_URL = "http://supabase-evals.local";
const PROJECT_DB_ANON_KEY = "supabase-evals-anon-key";
const PROJECT_DB_JWT_SECRET = "supabase-evals-dev-secret";
const PROJECT_DB_URL = ${JSON.stringify(PROJECT_DB_URL)};
const PROJECT_DB_ANON_KEY = ${JSON.stringify(PROJECT_DB_ANON_KEY)};
const PROJECT_DB_JWT_SECRET = ${JSON.stringify(PROJECT_DB_JWT_SECRET)};
const AUTH_SQL = \`
CREATE ROLE anon NOLOGIN;
CREATE ROLE authenticated NOLOGIN;
Expand Down Expand Up @@ -157,7 +169,7 @@ async function runNodeBin(
return new Promise((resolve) => {
const child = spawn(process.execPath, [bin, ...args], {
cwd,
env: { ...process.env, ...env },
env,

@barryroodt barryroodt Aug 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flipped the fourth parameter's meaning from "extras merged over process.env" to "the child's whole environment", while keeping the name env and the = {} default. The invariant now lives at both call sites, so a future runNodeBin(bin, args, cwd) gets a bare env with no signal anything's wrong.

Non-blocking, but building it here deletes PROJECT_ENV and two of the four hunks in this file:

const PROJECT_CHILD_ENV = {
  VITE_SUPABASE_URL: PROJECT_DB_URL,
  VITE_SUPABASE_ANON_KEY: PROJECT_DB_ANON_KEY,
};
// ...
env: { ...PROJECT_CHILD_ENV, ...extraEnv },

viteBuild then goes back to its three-argument call, and vitestRun to { SUPABASE_EVALS_WORKSPACE: workspace }.

stdio: ['ignore', 'pipe', 'pipe'],
});
let stdout = '';
Expand Down
53 changes: 45 additions & 8 deletions apps/framework/scripts/smoke-framework.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -475,20 +475,57 @@ async function smokeFrontendBuildTooling() {
cpSync(join(ROOT, FRONTEND_EVAL, 'tests'), join(workspace, 'tests'), {
recursive: true,
});
writeFileSync(join(workspace, 'src', 'App.tsx'), GOOD_FRONTEND_APP);
writeFileSync(
join(workspace, '.env.local'),
join(workspace, 'vite.config.ts'),
[
'VITE_SUPABASE_URL=http://supabase-evals.local',
'VITE_SUPABASE_ANON_KEY=supabase-evals-anon-key',
"import { defineConfig } from 'vite';",
"import react from '@vitejs/plugin-react';",
'',
'if (',
' process.env.ANTHROPIC_API_KEY ||',
' process.env.OPENAI_API_KEY ||',
' process.env.AI_GATEWAY_API_KEY',
") throw new Error('inherited LLM credential');",
'',
'export default defineConfig({ plugins: [react()] });',
'',
].join('\n')
);
writeFileSync(join(workspace, 'src', 'App.tsx'), GOOD_FRONTEND_APP);
writeFileSync(
join(workspace, 'tests', 'environment.test.ts'),
[
"import { expect, test } from 'vitest';",
'',
"test('does not inherit LLM credentials', () => {",
' expect(process.env.ANTHROPIC_API_KEY).toBeUndefined();',
' expect(process.env.OPENAI_API_KEY).toBeUndefined();',
' expect(process.env.AI_GATEWAY_API_KEY).toBeUndefined();',
Comment on lines +501 to +503

@barryroodt barryroodt Aug 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These three names are the only thing the fixture samples, so a revert that spreads process.env and filters out just these keys keeps build.ok and vitest.ok true while GITHUB_TOKEN and the rest still reach the child. Same goes for the vite.config.ts guard above.

Non-blocking: expect(Object.keys(process.env).sort()).toEqual([...]) closes it, and it's easiest once the allowlist has one home (see my comment on runNodeBin).

'});',
'',
].join('\n')
);

const originalEnv = {
ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY,
OPENAI_API_KEY: process.env.OPENAI_API_KEY,
AI_GATEWAY_API_KEY: process.env.AI_GATEWAY_API_KEY,
};
process.env.ANTHROPIC_API_KEY = 'test-anthropic-key';
process.env.OPENAI_API_KEY = 'test-openai-key';
process.env.AI_GATEWAY_API_KEY = 'test-ai-gateway-key';

const build = await viteBuild(workspace);
assert.equal(build.ok, true, build.stderr || build.stdout);
const vitest = await vitestRun(workspace);
assert.equal(vitest.ok, true, vitest.stderr || vitest.stdout);
try {
const build = await viteBuild(workspace);
assert.equal(build.ok, true, build.stderr || build.stdout);
const vitest = await vitestRun(workspace);
assert.equal(vitest.ok, true, vitest.stderr || vitest.stdout);
} finally {
for (const [name, value] of Object.entries(originalEnv)) {
if (value === undefined) delete process.env[name];
else process.env[name] = value;
}
}

console.log('PASS frontend vite/react/supalite build + test tooling');
}
Expand Down