[nicknisi]

Why

After #265, the Opus 5.5 installer passed every scenario but was slow and used many tool calls. The transcripts showed structural problems in the installer, not just in the skills. The settings-isolation fix from this PR shipped separately in #267; this PR is rebased on it.

What

fix(installer): Kotlin, PHP, and Laravel installs aborted before the agent ran. runAgentInstaller read package.json for every integration, and a missing file exits the process ("Could not find package.json… Installer setup canceled."). Vanilla JS checked for a package named workos, then asked "Vanilla JavaScript does not seem to be installed. Do you still want to continue?" (default No), which fails outright in CI and agent mode.

fix(installer): give the agent Claude Code's system prompt and state its rules.

• The system prompt. When none is set, the SDK sends an empty system prompt. The agent got no tool guidance and discovered the Bash allowlist one denial at a time: about 80% of Bash calls were denied.
• What it now says:
  • the allowlist, and to use Read/Grep/Glob for inspection
  • no network, workos CLI, or dev servers
  • the installer handles dashboard settings
  • whether this run gets post-run build feedback, which only runners with validation do
• Allowlist additions:
  • read-only git
  • cd <working dir> && <allowed>
  • per-tool syntax checks: py_compile, ruby -c, php -l
  • dependency changes: remove, go get, composer require
• Review fix (security): shell operators are now checked on the raw command before a cd prefix is stripped, and cd targets must be plain paths. Before this, cd "/proj/$(id)/.." && git status passed.

fix(installer): treat a timed-out build or typecheck as inconclusive. The post-run check killed builds at 60s and sent the agent a bare "Build failed". On a busy machine, agents then guessed at code changes. A killed check now logs a warning and doesn't trigger a correction. Limits are now 60s (typecheck) and 180s (build).

test(evals): the eval runs the real installer integration. Before, it used its own prompt, which named a skill that no longer exists, and called runAgent directly. Now:

• It calls each integration's run() after configureInstallEnvironment, the same env step the installer runs.
• An installer failure fails the scenario.
• Each scenario writes its own installer log, via runWithLogFile in debug.ts.
• Four Next.js fixtures moved from 14.2 to 15.3, the installer's minimum.

chore(deps): @workos/skills 0.8.1.

Results

50 scenarios: all JS frameworks plus Python, Ruby, Go, and Elixir. They run through the real installer flow with production tool gating, settings isolation (#267), and the current skills main (0.8.2 content). "Before" is the #265 run.

	Opus 4.5 before → after	Opus 5.5 before → after
Passed	50/50 → 50/50	50/50 → 50/50
First-attempt pass	72% → 90%	84% → 94%
Scenario duration p50	179s → 126s	364s → 123s
Scenario duration p95	380s → 235s	534s → 260s
Tool calls	1,654 → 1,145	2,824 → 1,475
Wall time (concurrency 8)	25m → 16m	47m → 15m
TTFT p50	12.3s → 5.0s	13.0s → 23.9s

Opus 5.5 leads on first-attempt passes and matches 4.5 on typical install time. Its time to first token (about 24s against 5s) and its slowest installs are still higher.

PHP, Laravel, and Kotlin weren't eval'd because their toolchains aren't installed locally; their abort fix is covered by unit tests. This is one run per model.

Verification

bun run test: 3,490 passed. Typecheck, lint, format:check, and build are clean.

Follow-up

Bump @workos/skills to 0.8.2 (workos/skills#56 backend call shapes) once workos/skills#57 is released.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The latest changes appear safe to merge.

Summary

The latest change closes the reported Git helper bypass.

• Kotlin, PHP, and Laravel installs reach the agent with credentials in .env.
• The installer agent starts with Claude Code guidance and WorkOS rules.
• Installer agents can run more project checks while shell commands stay constrained.
• Timed-out builds and typechecks no longer trigger guessed code fixes.
• Eval scenarios now run through the installer's real integration flow.

No new actionable issues were found. The earlier threads are unnumbered, so they have no numbered finding entries.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Agent requests a Git command] --> B[Check command and arguments]
  B --> C{Can it use diff helpers?}
  C -->|No| D[Allow read-only command]
  C -->|Yes| E{Can protective flags be added?}
  E -->|Yes| F[Allow rewritten command]
  E -->|No| G[Deny and explain supported form]

Loading

_{Reviews (5) · Last reviewed commit: "fix(installer): deny read-only git diff/..." · Reviewed by Greptile}

[nicknisi]

@greptileai review

[nicknisi]

@greptileai review

[nicknisi]

@greptileai review — the quoted-flag bypass is fixed in 3f7cfa2: git commands containing quotes or backslashes are denied, and allowed diff/show/log run with --no-ext-diff --no-textconv.

[nicknisi]

@greptileai review