diff --git a/docs/design/agent-workflows/projects/runner-desloppify-redo/HANDOFF.md b/docs/design/agent-workflows/projects/runner-desloppify-redo/HANDOFF.md new file mode 100644 index 0000000000..27accef63b --- /dev/null +++ b/docs/design/agent-workflows/projects/runner-desloppify-redo/HANDOFF.md @@ -0,0 +1,316 @@ +# Handoff: runner sandbox-agent decomposition (redo on current main) + +This document is the running record for the overnight refactor that redoes the sandbox-agent +decomposition on current `origin/main`. It is written for a reader who is comfortable with the +product but does not write TypeScript. Every section uses plain sentences. The goal of the work +is to split one very large runner file into small, well-named files, and to draw three clean +seams (sandbox provider, harness, and tool delivery), without changing any behavior at all. + +## What "no behavior change" means here + +The runner is the service that runs an agent inside a sandbox. Its main file, +`services/runner/src/engines/sandbox_agent.ts`, has grown to about 2,477 lines. This work moves +code out of that one file into several smaller files, and groups those files into folders by +subsystem. It does not change what the code does. Every test that passed before must still pass, +the network messages the runner sends and receives must be identical, and the way it behaves at +runtime must be identical. If a real bug is found while moving code, it is written down in the +findings section below and the code is moved as-is, not fixed. + +## Starting state (verified 2026-07-17) + +- The main file `sandbox_agent.ts` is 2,477 lines. +- Main ALREADY has a `sandbox_agent/` folder with 30 smaller files from earlier work. What is + still too large is `sandbox_agent.ts` itself and `sandbox_agent/session-pool.ts`. +- Baseline before any change, with the pi-openai lanes applied: typecheck passes, and all 1,190 + runner unit tests pass across 76 files. +- The reference for the target shape is a read-only worktree at + `.worktrees/desloppify-sandbox-agent/`. It shows the same split done on month-old code. Its + file names and module boundaries are the approved template. Its CONTENT is old and is not + copied; today's code is extracted into files with those names. + +## Decision: unapply the pi-openai lane for a clean base + +The lane `feat/pi-openai-compatible-models` is applied in the workspace and changes four of the +files this refactor relocates: `sandbox_agent.ts` (73 added lines spread across the import block, +`shouldSuppressPausedToolCallUpdate`, the `SessionEnvironment` interface, and four regions inside +`acquireEnvironment`), plus `daytona.ts`, `pi-assets.ts`, and the new `pi-model-config.ts`. +Moving code inside a file that another applied lane also edits scrambles which change belongs to +which lane. Because that lane is already committed and pushed, unapplying it is safe and +reversible. The plan is to unapply it, do the decomposition on a clean base equal to +`origin/main`, and note that in the morning it is re-applied and its four-file diff is rebased +over the new file layout. Only the 73 lines inside `sandbox_agent.ts` need re-homing; the other +three files are barely touched by this refactor. + +## Phase 1 extraction plan (all moves of existing code, dependency order) + +`sandbox-ports.ts` from the template is NOT a Phase 1 move. Those port interfaces do not exist in +today's code; they are a Phase 3 design. Phase 1 only moves code that already exists. + +Two decompositions: + +Decomposition A, slim the session pool: +1. `session-identity.ts` — move the identity, fingerprint, credential-epoch, and pool-key + helpers OUT of `session-pool.ts` (readKeepaliveConfig, resolvesToLocalProvider, + configFingerprint, historyFingerprint, expectedNextHistoryFingerprint, priorConversation, + approvalDecisionForToolCall, tailIsFreshUserMessage, KeepaliveConfig, CredentialEpoch, + computeCredentialEpoch, mountCredentialsExpired, credentialEpochMismatch, credentialEpochValid, + PoolScope helpers, poolKeyFor). Leaves `session-pool.ts` as just the SessionPool class and its + directly related types. + +Decomposition B, split the monolith `sandbox_agent.ts`: +2. `runtime-policy.ts` — the small pure policy functions (runCredential, + serverPermissionsFromRequest, shouldSuppressPausedToolCallUpdate, applyClaudeConnectionEnv, + modelResolutionStrict, defaultResolveLocalRunnerOwner, isTransportEndpointDisconnected, + containsTransportEndpointDisconnected). +3. `runtime-contracts.ts` — the interfaces and types (SandboxAgentDeps, CurrentTurn, + ParkedApproval, ResumeApprovalInput, RunTurnOptions, sendLastMessageOnly, SessionEnvironment, + AcquireEnvironmentResult, the RUN_LIMIT_TRIPPED symbol). +4. `session-events.ts` — routeSessionEventToActiveTurn, routePermissionRequestToActiveTurn. +5. `environment-setup.ts` — the setup helper (prepareEnvironmentSetup), which the template + carved out of the middle of `acquireEnvironment`. This is the one genuinely new boundary; it + is a region extraction, guided by the template. +6. `environment.ts` — acquireEnvironment, destroyInFlightSandboxes, + destroyInFlightSandboxesForSession, resolveKeepaliveMount, invalidateContinuity. +7. `run-turn.ts` — runTurn. +8. `engine.ts` — runSandboxAgent and shouldPark (the facade's brain). +9. `sandbox_agent.ts` — becomes a thin file that only re-exports the public surface. + +After EACH extraction: run typecheck and the runner unit tests; both must pass; then commit that +one step to the lane with a clear message. + +## Phase 2 grouping plan + +After the flat extraction, group `sandbox_agent/` into subfolders: `environment/`, `session/`, +`turn/`, and `tools-delivery/`, with top-level facade, engine, runtime-contracts, runtime-policy, +and errors. Each subfolder gets an `index.ts` public entry, and imports from outside a subfolder +go through that index. Placement rationale is recorded here as it is decided. + +## Phase 3 ports + +Three seams become explicit interfaces with one dispatch point each: sandbox provider, harness, +and tool delivery. The full requirements from the coordinator, including the E2B and harness +credential details and the anti-goals, are in the "Phase 3 port requirements" section at the end. + +## Milestone log + +- 2026-07-17: Context gathered. Baseline green (typecheck, 1,190 tests). Template boundaries + mapped. Decision recorded to unapply `feat/pi-openai-compatible-models` for a clean base. + Phase 1 order fixed. Project docs created. + + +- 2026-07-17 (~21:24Z): PHASE 1 COMPLETE. Lane `refactor/runner-sandbox-agent-decomp`, 8 commits, each typecheck-clean with all 1158 runner unit tests passing: + 1. session-identity.ts out of session-pool.ts (f9d19abbea) + 2. runtime-policy.ts out of the monolith (c6eaf6e23e) + 3. runtime-contracts.ts out of the monolith (8bd1dc9135) + 4. session-events.ts out of the monolith (7c48302046) + 5. environment.ts (whole acquireEnvironment + destroy/keepalive helpers) out of the monolith (98b7369225) + 6. run-turn.ts (runTurn) out of the monolith (5e53a9ebd4) + 7. engine.ts (runSandboxAgent + shouldPark) and sandbox_agent.ts reduced to a 44-line re-export facade (4e676476f3) + 8. environment-setup.ts carved out of acquireEnvironment (prepareEnvironmentSetup returns a 21-key bundle; typecheck enforced completeness) (111d4c65ac) + The monolith went from 2,477 lines to a 44-line facade. Public export surface unchanged (facade re-exports the same names). New-since-template behaviors survived verbatim (the subagents added today's newer fields/imports that the month-old template lacked: signAgentMountCredentials, agentMountCreds, seedForRun, toolSpecsByName, the 7 extra setup bundle keys, etc.). + +- 2026-07-17 (~21:37Z): DRAFT PR #5369 opened (https://github.com/Agenta-AI/agenta/pull/5369), base main, lane refactor/runner-sandbox-agent-decomp, head d980dedf17 (remote==local verified). coderabbit review requested. Phase 1 shipped; Phases 2-3 deferred with designs (see below and port-design.md). Do-not-merge until the morning review. agent-release-gate not run tonight (would require a disruptive runner redeploy on the shared local stack); recommended pre-merge. + +## Decisions + +- Base for the new PR is `origin/main`. +- `sandbox-ports.ts` is Phase 3, not Phase 1, because its interfaces are new. +- pi-openai lane unapplied during the work, re-applied and rebased in the morning. + +## Open questions + +- None blocking yet. + +## Findings: bugs seen but not fixed + +None yet. Add file, line, what looks wrong, and why it was left alone. Do not fix inline. The +client-tools silent-drop in `run-plan.ts` is owned by the other orchestrator; do not touch it. + +## Rebase map: where the moved code now lives (for PR #5363 and the pi-openai re-apply) + +The public export surface of `engines/sandbox_agent.ts` is unchanged (the facade re-exports the +same names), so anything that imports FROM the facade needs no change. Only code that edited the +INTERNALS of the old monolith needs to point at the new file. + +For JP's PR #5363 (Extend sessions): +- The reconnect-failure branch and the pointer-write-after-hydrate branch inside + `acquireEnvironment` now live in `engines/sandbox_agent/environment.ts` (the acquire back-half of + `acquireEnvironment`). If a change touches the setup prefix instead (ownership check, mount + signing, buildRunPlan, daemon env), that is now `engines/sandbox_agent/environment-setup.ts`. +- The turn-completion sync call inside `runTurn` now lives in `engines/sandbox_agent/run-turn.ts`. +- The `SandboxAgentDeps` interface now lives in `engines/sandbox_agent/runtime-contracts.ts`. +- The rewrite of `sandbox-reconnect.ts` and the rewrite of `session-continuity-durable.ts` land on + those same files — they were NOT moved (Phase 2 grouping is deferred), so those hunks apply as-is. +- Anything that imported identity/fingerprint/pool-key helpers from `session-pool.ts` (for example + `configFingerprint`, `poolKeyFor`, `computeCredentialEpoch`) now imports them from + `session-identity.ts`. The `SessionPool` class and `LiveSession`/`ParkInput`/`SessionState` + stayed in `session-pool.ts`. + +For the pi-openai lane (feat/pi-openai-compatible-models, PRs #5345/#5346), when it is re-applied and +rebased over this refactor, its four-file diff re-homes like this: +- Its `daytona.ts`, `pi-assets.ts`, and new `pi-model-config.ts` are unchanged locations (those files + were not moved), so those apply as-is. +- Its 73 added lines in the old `sandbox_agent.ts` re-home by region: the import-block additions map + to the module that now owns that symbol; its `shouldSuppressPausedToolCallUpdate` edit is now in + `runtime-policy.ts`; its `SessionEnvironment` interface edit is now in `runtime-contracts.ts`; and + its four edits inside `acquireEnvironment` are now in `environment.ts` (or `environment-setup.ts` + if the edited lines fell in the setup prefix). This is the "small conflict to rebase over in the + morning" the task anticipated. + +## Phase 3 port requirements (captured from the coordinator; keep in spirit) + +See the companion notes captured in the milestone log. The sandbox provider port must use two +axes (is-remote and provider-identity) expressed as declared capability flags, never scattered +booleans, and must cover: is-remote, working-directory-is-a-FUSE-mount, can-enforce-network- +policy, can-inject-runtime-credentials, and who-installs-the-harness-binary. Keepalive is a +declared trait (Daytona native autostop is a no-op; E2B-style needs a runner refresh loop keyed +on a stable sandbox id, plus a create-time self-reap backstop that fires even if the runner +dies). The port is a thin adapter over the sandbox-agent library, owning only extended lifecycle, +keepalive, typed create options (no `any`), and capability declaration. The filesystem part has +built-in path containment and path-flavor awareness; the process part runs an argument vector +with no shell. The harness port models credentials as environment keys plus an optional +credential file (path, render, required-in-managed-mode), uses strict per-file upload allowlists +(never a directory copy), renders per-run config that wins over uploaded files, dispatches asset +preparation in one place keyed on harness id, and keeps the daemon's credential-env blanking list +a superset of every harness's env keys. Anti-goals: boolean accretion on the run plan, cloned +per-cell files, informational-only env flags, and `any` casts. + +## Verification status (Phase 1) + +On lane `refactor/runner-sandbox-agent-decomp`, after all 8 extraction commits: +- `pnpm run typecheck` in services/runner: passes. +- `pnpm run test:unit`: 75 files, 1,158 tests, all passing. +- `pnpm run build:extension`: builds the Pi extension and the stdio tool shim, exit 0. +Each of the 8 commits was individually verified green before the next, so the history bisects cleanly. + +## Decision: defer Phase 2 (subfolder grouping) and Phase 3 (ports) to a follow-up, ship Phase 1 now + +This is a judgment call driven by three concrete findings. The task explicitly invited judgment on +edge cases, and behavior preservation is the absolute constraint. + +1. A real, hard-to-verify-tonight behavior risk in grouping. `daemon.ts` computes its package root + as `dirname(dirname(dirname(fileURLToPath(import.meta.url))))` — three directory levels up from + the file's own location. That package root drives where the Pi and adapter binaries are found at + runtime. Moving `daemon.ts` one level deeper into a subfolder silently changes that root and + breaks binary resolution in a real deployment, and the unit tests do not exercise real binary + resolution, so typecheck and the unit suite would BOTH stay green while the runtime broke. It is + the only file with self-location path logic (verified by grep for import.meta.url / __dirname / + fileURLToPath across the whole folder), but it means grouping is not the pure, test-gated + mechanical change it appears to be. +2. Cross-lane friction, present tense. The other orchestrator is actively editing `run-plan.ts` in + this same workspace. Moving that file now would hunk-lock against their in-flight change. So + `run-plan.ts` cannot be grouped tonight at all. +3. Cross-lane friction, future tense. The pi-openai PRs (#5345 and #5346) are open and edit + `daytona.ts` and `pi-assets.ts`. Renaming those into subfolders turns their eventual rebase into a + rename-plus-edit conflict for no behavior benefit. + +Grouping under those constraints becomes an exception-riddled half-move (daemon, run-plan, daytona, +pi-assets all stuck at top level) with large import churn across ~30 external importers, a real +runtime risk on daemon, and zero behavior improvement. The same risk profile applies to a large +Phase 3 port refactor: consolidating the currently-scattered provider, harness, and tool-delivery +logic into one dispatch point each is exactly where subtle runtime-behavior differences hide, and it +touches the same pi-openai-conflicting `daytona.ts`. The unit suite is strong but, as the daemon +case shows, not a full runtime guarantee. + +Therefore Phase 1 (the decomposition redo — the primary ask, "redo #5264's shape on current main") +ships now as a clean, fully verified, behavior-preserving draft PR. Phases 2 and 3 are captured as +concrete designs below and in `port-design.md`, ready for a focused follow-up that can run the +agent release gate and a real end-to-end run to verify the runtime behavior the unit tests miss. +The recommended follow-up sequence: land pi-openai (#5345/#5346) and the run-plan fix first, then do +grouping and ports on a clean base with runtime verification. + +## Phase 2 grouping plan (deferred; do after pi-openai and the run-plan fix land) + +Target subfolders under `services/runner/src/engines/sandbox_agent/`, each with an `index.ts` public +entry; imports from outside a subfolder go through its index only. +- `environment/`: environment.ts, environment-setup.ts, provider.ts, daytona-provider.ts, mount.ts, + agent-mount.ts, agent-mount-guidance.ts, workspace.ts, model.ts, capabilities.ts, acp-fetch.ts. +- `session/`: session-pool.ts, session-identity.ts, session-continuity.ts, + session-continuity-durable.ts, session-events.ts, sandbox-reconnect.ts. +- `turn/`: run-turn.ts, transcript.ts, usage.ts, run-limits.ts, pause.ts, acp-interactions.ts, + pi-error.ts. +- `tools-delivery/`: mcp.ts, tool-mcp-assets.ts, relay-guard.ts, pi-gate-envelope.ts, client-tools.ts. +- Stay at top level: sandbox_agent.ts (the facade), engine.ts, runtime-contracts.ts, + runtime-policy.ts, errors.ts. +- Stay at top level as documented EXCEPTIONS, not because they belong there: + - `daemon.ts` — location-dependent package-root path (see finding 1). If ever moved, add one + `dirname(...)` per level of new depth and verify binary resolution against a real run. + - `run-plan.ts` — actively edited by the other orchestrator; move it only after that lane lands. + - `daytona.ts`, `pi-assets.ts` — edited by pi-openai (#5345/#5346); move them only after those land + (they belong in `environment/`). `pi-assets.ts` could alternatively live in `tools-delivery/` + since it prepares the Pi bundled extension; environment/ is the simpler home because + environment-setup.ts drives it. +Grouping is behavior-safe for every file EXCEPT daemon.ts (path depth) once the cross-lane files can +move, and it is fully gated by typecheck plus the unit suite for the non-daemon files. + +## Milestone: rebased onto v0.105.4 main (2026-07-18) + +The lane was rebased from old main onto the released v0.105.4 main and is now conflict-free, +green, and pushed. PR #5369 flipped from CONFLICTING to MERGEABLE. + +- **New base:** `80daf23257` (Merge #5368 release/v0.105.4). Reached via `but pull`, which also + archived the three now-merged lanes it saw applied (feat/pi-openai-compatible-ui, + feat-runsh-overrides-and-recreate, fix/runner-daytona-client-only-tools-gate) and cleanly + rebased the two unrelated applied lanes (fix/claude-fable-model-id, qa-agent-release-gate). +- **Lane tip:** `db13a17add`. Remote == local verified after `but push -f`. + +### Conflicts hit and how resolved + +`but pull` left five of the nine extraction commits conflicted, each in `sandbox_agent.ts`, all +because a release fix touched a region the refactor was moving out of the monolith. Resolved +bottom-up with `but resolve ` / `but resolve finish`: + +1. **runtime-policy extraction** — the `shouldSuppressPausedToolCallUpdate` region. The only + base-vs-ancestor difference was cosmetic type-union reformatting (no behavior). Accepted the + refactor's deletion (functions live in `runtime-policy.ts`). +2. **runtime-contracts extraction** — the `AcquireEnvironmentResult` region; again only cosmetic + union reformatting. Accepted the deletion (types live in `runtime-contracts.ts`). +3. **environment extraction** — two regions: the import block (took the refactor's minimal facade + imports) and the whole `acquireEnvironment` body (accepted deletion — it moved to + `environment.ts`). Main's #5345 delta to `acquireEnvironment` was extracted and set aside here, + then re-homed (see below), so nothing was dropped. +4. **run-turn extraction, engine extraction** — cleared automatically once the lower commits were + resolved (no residual markers). + +### Re-homing the #5345 delta (the one real behavioral merge) + +Main's #5345 added the OpenAI-compatible model-config plan inside `acquireEnvironment` (7 hunks). +Because the refactor split that function across `environment-setup.ts` (the setup prefix) and +`environment.ts` (the acquire body), the delta was threaded through the split in a dedicated +commit `db13a17add`: +- `prepareEnvironmentSetup` (environment-setup.ts) now builds `piModelConfig` / `piModelConfigError`, + passes `piModelConfig` into `prepareLocalPiAssets`, computes `localModelConfigUnwritable`, and + returns all three in its bundle. Imports `buildPiModelConfigPlan` / `PiModelConfigPlan`. +- `acquireEnvironment` (environment.ts) destructures those three, throws `piModelConfigError` / + `PI_MODEL_CONFIG_WRITE_FAILED_MESSAGE` at the fail-loud/fail-closed gates, passes `piModelConfig` + into `prepareDaytonaPiAssets`, and selects the fully-qualified `wantedModel`. Imports + `PI_MODEL_CONFIG_WRITE_FAILED_MESSAGE`. +This is a faithful transcription of main's logic, verified below. + +### No release code lost + +Every unmoved release file in the lane tip is byte-identical to `origin/main` (0-line diff): +transcript.ts (#5364), run-plan.ts (#5366), daytona.ts, pi-assets.ts, pi-model-config.ts (#5345), +tracing/otel.ts (#5362), package.json, pnpm-lock.yaml. + +### Verification + +- `pnpm run typecheck`: clean (after `pnpm install` picked up the release's OTel 2.x / Daytona + 0.198 bump — the stale node_modules was the only typecheck failure and is not a lane change). +- `pnpm test`: 76 files, **1192 tests, all pass** (was 1158/75; the +34/+1 are the release's own + new suites now running green on the rebased tree). +- `pnpm run build:extension`: pass. +- Runner runtime smoke test: booted the rebased runner with the new deps; `/health` -> 200 with the + full harness list, `/run {}` -> correct structured validation error, no crash. Evidence in + `debug/qa-refactor-rebase/` (runner-boot.log, health.json). +- Full product `agent-release-gate`: NOT run. It needs a rebuild + services-repoint on the shared + custom-named EE-dev stack (whose product agent path is currently pointed at a dead sidecar). The + v0.105.4 release is already product-gate-clean (`debug/qa-105.4/`) and this change is + behavior-identical to it, so it is deferred as the recommended pre-merge step on a dedicated stack. + +### State + +Draft PR #5369, base main, head `db13a17add`, MERGEABLE. Do not merge (Mahmoud merges). The +morning re-apply of pi-openai (#5345/#5346) noted earlier in this doc is now moot — pi-openai is +merged into main and its runner delta is already in the rebased base; only the acquireEnvironment +re-home (commit db13a17add) was needed. diff --git a/docs/design/agent-workflows/projects/runner-desloppify-redo/port-design.md b/docs/design/agent-workflows/projects/runner-desloppify-redo/port-design.md new file mode 100644 index 0000000000..59c61f592c --- /dev/null +++ b/docs/design/agent-workflows/projects/runner-desloppify-redo/port-design.md @@ -0,0 +1,137 @@ +# Port design: sandbox provider, harness, and tool delivery + +This document is the design for Phase 3 of the runner decomposition: turning three implicit seams +in the runner into three explicit interfaces, each with a single dispatch point. It is written for +a reader who understands the product but does not write TypeScript. The design is ready to +implement; the implementation is deferred to a follow-up so that each seam can be checked with a +real end-to-end run, because these three areas control runtime behavior that the unit tests do not +fully exercise (the same lesson as the daemon.ts path finding in the handoff). + +The goal of all three ports is the same. Today, "which provider is this" and "which harness is +this" are answered by scattered boolean checks (is it Daytona, is it remote, is it Pi) spread +across many files. Adding a new provider (for example E2B) or a new harness (for example Codex) +means finding and editing every one of those checks. The ports replace the scattered checks with +one object per provider and one object per harness that DECLARES its traits, plus one place that +reads those declarations. After the change, "add a provider" and "add a harness" each become "write +one object that declares its capabilities and register it in one place." + +## Seam 1: the sandbox provider + +A sandbox provider is the thing that creates and runs the sandbox where the agent executes. Today +there are two: a local Docker-style provider and Daytona. The runner leans on an embedded library +called `sandbox-agent` for the basics (create, destroy, filesystem, run a process, get a URL). The +port is a thin adapter over that library; it owns only what the library cannot do. + +Two independent axes, never one boolean. The first axis is "is this sandbox remote", meaning it is +reachable only through a filesystem API, with no host directory mounted and no loopback network. +The second axis is "which provider is this" (local, Daytona, E2B). These are different questions and +must not be collapsed into a single flag. Both are expressed as declared capability flags on the +provider, not as booleans re-derived at each call site. + +Capabilities each provider declares: +- is remote (the first axis above). +- whether the working directory is a FUSE mount. This drives where the tool relay is placed and the + logic that remounts the working directory after a transport disconnect (the ENOTCONN handling in + acquireEnvironment today). +- whether the provider can enforce a network egress policy. Daytona can. Local and E2B cannot, and a + run that requests such a policy on those providers must be refused, not silently allowed. +- whether the provider can inject runtime-provided credentials. Today this is gated behind an + explicit "is it Daytona" check; it becomes a declared capability instead. +- who installs the harness binary. For Pi the runner installs it; for other harnesses the sandbox + image or daemon bakes it in. + +Keepalive is a declared trait, because providers keep sandboxes alive very differently. Daytona +keeps itself alive with native idle autostop, so the runner does nothing. An E2B-style provider +needs the runner to run a refresh loop keyed on a stable sandbox id, because the embedded library +never hands back the raw provider handle, so any out-of-band keepalive call needs the id plus the +ambient environment. Every provider must also set a create-time self-reap backstop (an autostop, an +autodelete, or a timeout) that still fires if the runner process dies, so a crashed runner never +leaks a running sandbox. + +Lifecycle surface the port owns beyond the library: connect and reconnect (Daytona has a full state +machine and reconverges its network policy on reconnect; other providers may do nothing), pause, and +refresh-activity. Create, destroy, filesystem, process, and get-url all delegate straight to the +library. The port also owns a real type for the create options (both create paths cast to `any` +today; give them a typed shape) and the capability declaration itself. + +Filesystem part of the port: make-directory, write, and read, each with built-in containment so a +path can never escape the working directory, and each aware of path flavor (host operating-system +separators for a local sandbox, always POSIX separators for a remote one). Process part: run an +argument vector directly, never through a shell. + +Single dispatch point: the provider registry stays the one admission point. The known provider ids, +which are enabled, and which are planned live in `runner-config.ts`, and there is exactly one +dispatch in `provider.ts` (its `buildSandboxProvider` is the seed to grow into this port). E2B and +Docker then slot in by writing one provider object with its capability declaration and registering +it; no turn logic changes. + +## Seam 2: the harness + +A harness is the coding agent that runs inside the sandbox (Pi, Claude, Codex, Opencode). The +harness port models everything that differs per harness so that adding one is writing a single +object. + +Credential material is per harness and splits into environment variables versus a file: +- Pi uses an environment variable or the file `~/.pi/agent/auth.json`. +- Claude uses the `ANTHROPIC_API_KEY` environment variable in managed mode, or the file + `~/.claude/.credentials.json` when the user logs in with their own account. +- Codex uses the `OPENAI_API_KEY` environment variable and ALWAYS also writes a `~/.codex/auth.json` + file, even in managed mode. +- Opencode uses environment variables only. +So the shape is: a set of environment keys, plus an optional credential file described by its path, +how it is rendered, and whether it is required even in managed mode. + +Own-login uploads use strict per-file allowlists, never a whole-directory copy. Uploading the entire +`~/.claude` directory once leaked the `.mcp.json` tokens and other settings secrets. Config that is +rendered fresh for the run must win over any uploaded host file. + +Asset preparation is owned by the harness and dispatched in exactly one place, keyed on the harness +id, and only when the provider is remote. An earlier attempt grew three competing places for this; +the port collapses them to one. + +Model naming differs per harness (provider-then-id, or an alias, or a bare id), plus capability +flags such as whether the harness has built-in tools (only Pi does), its connection modes, and its +deployments. + +One coupling to protect: the daemon blanks a fixed set of credential environment variables before +applying fresh ones (the known-env-vars list in `daemon.ts`). That list must stay a superset of +every harness's environment keys. The harness port's documentation must say, in one sentence, that +adding a harness means extending that list, or a stale key from a previous run could leak into the +next. + +## Seam 3: tool delivery + +There are three ways tools reach the agent today, and one policy question that decides which tools +are even deliverable. The port puts the three mechanisms behind one interface and computes the +policy in one place. +- The Pi bundled extension: for a Pi harness, tools ride along inside Pi's own Agenta extension, and + every tool call relays back to the runner. +- Loopback HTTP: for a local sandbox, tools are served over a loopback HTTP relay. +- In-sandbox stdio shim: for a remote sandbox, a small stdio MCP shim inside the sandbox forwards + tool calls. +The one policy: "which tools are deliverable for this harness and this sandbox". For example, a +remote non-Pi sandbox's stdio shim delivers only executable (gateway or callback) tools and omits +client-kind tools, so a run whose tools are all client-kind on that path has nothing to advertise +and must be refused. That deliverability rule is computed in exactly one place instead of being +re-derived at each mechanism. + +## How E2B and Docker slot in without touching turn logic + +With the three ports in place, E2B is a new sandbox provider object that declares: is remote true, +working directory is a FUSE mount true or false depending on its filesystem, can enforce network +policy false (so a policy request is refused), keepalive is a runner refresh loop keyed on a stable +sandbox id with a create-time self-reap backstop, harness binary owner is the sandbox image. Docker +is a provider object that declares is remote false and a host mount. Neither requires any change to +`run-turn.ts` or the acquire logic, because the turn and acquire code reads capabilities and +dispatches through the single points rather than asking "is it Daytona" inline. + +## Why implementation is deferred + +Each of these three consolidations moves currently-scattered runtime logic into one place. That is +exactly where a subtle behavior difference can hide (the order of capability checks, an edge case in +the deliverability rule, a credential file that must be written even in managed mode). The runner +unit suite is strong but, as the daemon.ts package-root finding showed, does not fully guarantee +runtime behavior. So the port implementation should land in a focused follow-up that runs the agent +release gate and a real end-to-end run against a live deployment, ideally after the pi-openai PRs +(#5345 and #5346) land so the port work does not fight a rename conflict on `daytona.ts` and +`pi-assets.ts`. Phase 1 (the decomposition) is independent of all of this and ships now. diff --git a/services/runner/AGENTS.md b/services/runner/AGENTS.md index de7e148347..a85db64fd8 100644 --- a/services/runner/AGENTS.md +++ b/services/runner/AGENTS.md @@ -32,6 +32,28 @@ pnpm run typecheck # tsc --noEmit (src + tests + vitest.config) - Runtime code: `src/` — `engines/` (one engine: `sandbox_agent`), `tools/`, `tracing/`, `extensions/`. Entrypoints: `cli.ts`, `server.ts`. The `/run` wire contract is `protocol.ts`. +- The `sandbox_agent` engine is split into small files under `engines/sandbox_agent/`, and + `engines/sandbox_agent.ts` is a thin facade that only re-exports the public surface (import the + engine from there, not from the internal files). Where each concern lives: + - Composition and lifecycle: `engine.ts` (`runSandboxAgent` = acquire, run one turn, tear down; + plus `shouldPark`). Shared interfaces and types: `runtime-contracts.ts`. Small pure policy + helpers (credential extraction, permission map, strict-model flag, transport-disconnect + checks): `runtime-policy.ts`. + - Acquire a session-scoped environment: `environment.ts` (`acquireEnvironment`, the destroy and + keepalive helpers) and `environment-setup.ts` (`prepareEnvironmentSetup`, the setup prefix it + runs first). Providers and placement: `provider.ts`, `daytona.ts`, `daytona-provider.ts`, + `mount.ts`, `agent-mount*.ts`, `workspace.ts`, `model.ts`, `capabilities.ts`. Run plan: + `run-plan.ts`. + - Run one turn: `run-turn.ts` (`runTurn`), with `transcript.ts`, `usage.ts`, `run-limits.ts`, + `pause.ts`, `acp-interactions.ts`, `pi-error.ts`. + - Session keepalive: `session-pool.ts` (the mutable pool) and `session-identity.ts` (fingerprints, + credential epochs, pool keys), plus `session-continuity*.ts`, `session-events.ts`, + `sandbox-reconnect.ts`. + - Tool delivery: `mcp.ts`, `tool-mcp-assets.ts`, `relay-guard.ts`, `client-tools.ts`, + `pi-gate-envelope.ts`, `pi-assets.ts`. + - `daemon.ts` resolves the harness binary using a package root derived from its own file location + (three directory levels up). Do NOT move it into a subfolder without adjusting that derivation, + or binary resolution breaks at runtime (the unit tests will not catch it). - Tests: `tests/unit/**/*.test.ts` (vitest, `node:assert` is fine inside `it`). Shared test helpers and fixtures live in `tests/utils/`. This mirrors `web/packages/*` and the repo testing.structure spec. Do not add tests back under a flat `test/` directory. diff --git a/services/runner/src/engines/sandbox_agent.ts b/services/runner/src/engines/sandbox_agent.ts index ef959193a5..a337b9b715 100644 --- a/services/runner/src/engines/sandbox_agent.ts +++ b/services/runner/src/engines/sandbox_agent.ts @@ -6,2472 +6,39 @@ * engine. It serves the same /run contract (AgentRunRequest -> AgentRunResult), so the * Python side stays thin and the choice of harness/sandbox is config, not new code. * - * Per invoke (cold), mirroring the shipped code-evaluator DaytonaRunner pattern: - * - * SandboxAgent.start({ sandbox: local({ env }) | daytona({ create }) }) - * -> createSession({ agent: , cwd, model }) - * -> write AGENTS.md into cwd - * -> session.prompt([{ type: "text", text }]) - * -> accumulate ACP `agent_message_chunk` text + build the trace - * -> destroySandbox() - * - * Two orthogonal axes swap independently: the sandbox (where the daemon runs) and the - * harness (which engine). The ACP boundary is daemon-to-harness; the service-to-sandbox-agent - * hop stays harness-agnostic behind the Harness port. - * - * Session keep-alive (flag-gated, off by default) splits the per-invoke work into - * `acquireEnvironment` (session-scoped: sandbox, mount, session, MCP wiring) and `runTurn` - * (per-turn: otel run, prompt, usage, trace). `runSandboxAgent` composes them exactly as - * before (acquire -> runTurn -> destroy), so with the flag off behavior is byte-identical. - * The dispatch in `server.ts` reuses the two halves to continue a live session across a turn - * boundary. See docs/design/agent-workflows/projects/session-keepalive/plan.md. - * - * Tracing is built here from the ACP event stream (see tracing/otel.ts createSandboxAgentOtel), - * so it is uniform across every harness and always nests under the caller's /invoke - * span. stdout is reserved for the JSON result (see cli.ts); logs go to stderr. + * This file is the public facade. The implementation lives in the sibling `sandbox_agent/` + * modules and is grouped by subsystem: `environment.ts` / `environment-setup.ts` acquire the + * session-scoped sandbox, mount, session, and MCP wiring; `run-turn.ts` runs one per-turn + * prompt (otel run, prompt, usage, trace); `engine.ts` composes them (`runSandboxAgent` = + * acquire -> runTurn -> destroy) and decides whether a finished turn may be parked + * (`shouldPark`); `runtime-contracts.ts` holds the shared interfaces and `runtime-policy.ts` + * the small pure policy helpers. Keeping this entrypoint to re-exports makes the engine's + * supported surface obvious to the CLI, the server, and the tests. Behavior is unchanged from + * when all of this lived in one file. */ -import { mkdirSync, rmSync } from "node:fs"; - -import { apiBase } from "../apiBase.ts"; -import { seedForRun } from "../redaction.ts"; - -import { SandboxAgent, InMemorySessionPersistDriver } from "sandbox-agent"; - -import { - createSandboxAgentOtel, - TOOL_NOT_EXECUTED_PAUSED, -} from "../tracing/otel.ts"; -import { - localRelayHost, - sandboxRelayHost, - startToolRelay, - type RelayExecutionGuard, -} from "../tools/relay.ts"; -import { - ApprovalResponder, - ApprovedExecutionGrants, - ConversationDecisions, - extractApprovalDecisions, - extractClientToolOutputs, - type ClientToolOutcome, - type Responder, -} from "../responder.ts"; -import type { ClientToolRelay } from "../tools/client-tool-relay.ts"; -import { - buildClientToolRelay, - createToolCallCorrelationIndex, -} from "./sandbox_agent/client-tools.ts"; -import { - type AgentRunRequest, - type AgentRunResult, - type EmitEvent, - type HarnessCapabilities, - type ToolCallbackContext, - type ToolPermission, - resolvePromptText, - resolveRunSessionId, -} from "../protocol.ts"; -import { - assert, - assertRequiredCapabilities, - probeCapabilities, -} from "./sandbox_agent/capabilities.ts"; -import { createAcpFetch } from "./sandbox_agent/acp-fetch.ts"; -import { buildDaemonEnv, resolveDaemonBinary } from "./sandbox_agent/daemon.ts"; -import { - createCookieFetch, - prepareDaytonaPiAssets, - DAYTONA_PI_DIR, -} from "./sandbox_agent/daytona.ts"; -import { conciseError } from "./sandbox_agent/errors.ts"; -import { buildSessionMcpServers } from "./sandbox_agent/mcp.ts"; -import { applyModel } from "./sandbox_agent/model.ts"; -import { findSwallowedPiError } from "./sandbox_agent/pi-error.ts"; -import { - buildPiExtensionEnv, - configurePiSessionWorkspace, - configurePiSkillSnapshot, - PI_PERMISSION_EXTENSION_UNAVAILABLE_MESSAGE, - PI_MODEL_CONFIG_WRITE_FAILED_MESSAGE, - prepareLocalPiAssets, - resolvePiSkillSnapshot, - uploadSystemPromptToSandbox, - writeSystemPromptLocal, - writeOtlpAuthFile, -} from "./sandbox_agent/pi-assets.ts"; -import { - buildPiModelConfigPlan, - type PiModelConfigPlan, -} from "./sandbox_agent/pi-model-config.ts"; -import { - uploadToolMcpAssets, - type ToolMcpAssets, -} from "./sandbox_agent/tool-mcp-assets.ts"; -import { advertisedToolSpecs, toolSpecsByName } from "../tools/public-spec.ts"; -import { buildRelayExecutionGuard } from "./sandbox_agent/relay-guard.ts"; -import { - PendingApprovalLatch, - permissionsFromRequest, -} from "../permission-plan.ts"; -import { - attachPermissionResponder, - type ParkedApprovalGateType, -} from "./sandbox_agent/acp-interactions.ts"; -import { - PAUSED, - PendingApprovalPauseController, -} from "./sandbox_agent/pause.ts"; -import { - createRunLimits, - resolveRunLimits, -} from "./sandbox_agent/run-limits.ts"; -import { - createInteraction, - resolveInteraction, - buildWorkflowReferences, -} from "../sessions/interactions.ts"; -import { claimSessionOwnership, REPLICA_ID } from "../sessions/alive.ts"; -import { - teardownDisposition, - type TeardownReason, -} from "./sandbox_agent/teardown.ts"; -import { buildSandboxProvider } from "./sandbox_agent/provider.ts"; -import { loadRunnerConfig } from "../config/runner-config.ts"; -import { DaytonaReconnectTerminalError } from "./sandbox_agent/daytona-provider.ts"; -import { - buildRunPlan, - type BuildRunPlanDeps, - type RunPlan, -} from "./sandbox_agent/run-plan.ts"; -import { priorMessages } from "./sandbox_agent/transcript.ts"; -import { resolveRunUsage } from "./sandbox_agent/usage.ts"; -import { prepareWorkspace } from "./sandbox_agent/workspace.ts"; -import { - signSessionMountCredentials, - mountStorage, - mountStorageRemote, - unmountStorage, - discoverTunnelEndpoint, - mountHarnessSessionDirs, - harnessSessionMounts, - storeReachableFromSandbox, - type MountCredentials, -} from "./sandbox_agent/mount.ts"; -import { - AGENT_MOUNT_ENV_VAR, - agentMountPath, - linkAgentFiles, - linkAgentFilesRemote, - seedAgentReadme, - seedAgentReadmeRemote, - signAgentMountCredentials, -} from "./sandbox_agent/agent-mount.ts"; -import { - AGENT_MOUNT_SYSTEM_PROMPT_SEGMENT, - claudeMountSystemPromptMeta, - combineAppendSystemPrompt, - type ClaudeSystemPromptMeta, -} from "./sandbox_agent/agent-mount-guidance.ts"; -import { - hydrateHarnessSessionFromDurable, - syncHarnessSessionDurable, -} from "./sandbox_agent/session-continuity-durable.ts"; -import { - readStoredSandboxPointer, - clearSandboxPointer, - writeSandboxPointer, -} from "./sandbox_agent/sandbox-reconnect.ts"; -import { - assertLocalRunnerOwnership, - eligibleAgentSessionId, - nextTurnIndex, - sessionContinuityStore, - type SessionContinuityStore, -} from "./sandbox_agent/session-continuity.ts"; -import { - projectScopeFor, - resolvesToLocalProvider, -} from "./sandbox_agent/session-pool.ts"; +export { + acquireEnvironment, + destroyInFlightSandboxes, + destroyInFlightSandboxesForSession, + resolveKeepaliveMount, +} from "./sandbox_agent/environment.ts"; +export { + sendLastMessageOnly, + type AcquireEnvironmentResult, + type ParkedApproval, + type ResumeApprovalInput, + type RunTurnOptions, + type SandboxAgentDeps, + type SessionEnvironment, +} from "./sandbox_agent/runtime-contracts.ts"; +export { + runSandboxAgent, + shouldPark, +} from "./sandbox_agent/engine.ts"; +export { runTurn } from "./sandbox_agent/run-turn.ts"; export { buildTurnText, messageTranscript, } from "./sandbox_agent/transcript.ts"; export { toAcpMcpServers } from "./sandbox_agent/mcp.ts"; - -function log(message: string): void { - process.stderr.write(`[sandbox-agent] ${message}\n`); -} - -/** Extract the run credential from the OTLP export headers (initial value, constant for the run). */ -function runCredential(request: AgentRunRequest): string { - const headers = (request.telemetry?.exporters?.otlp?.headers ?? {}) as Record< - string, - string - >; - return (headers["authorization"] ?? headers["Authorization"] ?? "").trim(); -} - -function serverPermissionsFromRequest( - request: AgentRunRequest, -): ReadonlyMap { - const permissions = new Map(); - for (const server of request.mcpServers ?? []) { - if (server.policy?.permission !== undefined) { - permissions.set(server.name, server.policy.permission); - } - } - return permissions; -} - -type Log = (message: string) => void; -const LOCAL_DURABLE_CWD_ENOTCONN_REMOUNT_LIMIT = 1; - -// In-flight sandbox handles, by run. A process KILL (docker stop / SIGTERM / OOM mid-run) skips -// the per-run teardown — so a shutdown signal handler (see `server.ts`) drains this set to -// best-effort delete any still-running sandbox before exit. Remote (Daytona) sandboxes that even a -// signal can never reach (SIGKILL/OOM) self-reap via the lifecycle reapers in `provider.ts`. -const inFlightSandboxes = new Set<{ - destroy: (opts?: { reason?: TeardownReason }) => Promise; - sessionId: string; - mountProjectId?: string; - projectScopeId?: string; -}>(); - -/** - * Best-effort delete every sandbox currently mid-run, bounded so it can never hang shutdown. - * Called from the process signal handler so `docker stop` reaps remote sandboxes instead of - * leaking them. Each delete is independent and its own failure is swallowed; the whole sweep is - * raced against `timeoutMs` so a slow Daytona API call cannot block the exit. - */ -export async function destroyInFlightSandboxes( - timeoutMs = 5000, - reason: TeardownReason = "shutdown-in-flight", -): Promise { - const pending = [...inFlightSandboxes]; - if (pending.length === 0) return; - const sweep = Promise.allSettled( - pending.map((environment) => - Promise.resolve(environment.destroy({ reason })).catch(() => {}), - ), - ); - await Promise.race([ - sweep, - new Promise((resolve) => setTimeout(resolve, timeoutMs)), - ]); -} - -/** - * Same drain as `destroyInFlightSandboxes`, scoped to one session (and, when supplied, its - * owning project). Backs the HTTP `/kill` route so a caller can only tear down its own - * session's in-flight sandbox(es) — the unscoped sweep above stays an in-process-only call - * (the shutdown handler). - * - * Filters on `projectScopeId` (same run-context-preferred, mount-fallback precedence as - * `poolKeyFor`/`projectScopeFor` — never `mountProjectId` alone, which is undefined for a - * mountless run and would make a scoped kill silently match nothing). A sandbox whose run had - * no project scope at all (`projectScopeId` undefined) never matches a scoped `projectId` - * filter: `/kill` requires a non-blank `projectId`, so there is no caller this in-flight entry - * could ever be proven to belong to — the same no-scope-no-park invariant the pool enforces, - * mirrored here as no-scope-no-scoped-kill. It still falls to the unscoped shutdown sweep. - */ -export async function destroyInFlightSandboxesForSession( - sessionId: string, - projectId: string | undefined, - timeoutMs = 5000, - reason: TeardownReason = "kill", -): Promise { - const pending = [...inFlightSandboxes].filter( - (environment) => - environment.sessionId === sessionId && - (!projectId || environment.projectScopeId === projectId), - ); - if (pending.length === 0) return; - const sweep = Promise.allSettled( - pending.map((environment) => - Promise.resolve(environment.destroy({ reason })).catch(() => {}), - ), - ); - await Promise.race([ - sweep, - new Promise((resolve) => setTimeout(resolve, timeoutMs)), - ]); -} - -function shouldSuppressPausedToolCallUpdate( - update: unknown, - pause: PendingApprovalPauseController, -): boolean { - const frame = update as - { sessionUpdate?: unknown; toolCallId?: unknown } | undefined; - const kind = frame?.sessionUpdate; - if (kind !== "tool_call" && kind !== "tool_call_update") return false; - const toolCallId = - typeof frame?.toolCallId === "string" ? frame.toolCallId : undefined; - return pause.isPausedToolCall(toolCallId); -} - -const CLAUDE_STRICT_DEPLOYMENTS = new Set([ - "custom", - "bedrock", - "vertex", - "vertex_ai", -]); - -function applyClaudeConnectionEnv( - env: Record, - request: AgentRunRequest, - acpAgent: string, - logger: Log, -): void { - if (acpAgent !== "claude") return; - - // Disable the Claude Agent SDK's Tool-Search feature for every Claude run. The bundled - // SDK defaults Tool-Search ON, which makes Claude DEFER the `agenta-tools` MCP tools and - // call them before their `inputSchema` is loaded — so it emits an empty `input: {}` and - // tools-with-args (reference workflows, commit_revision) never receive their arguments. - // Our tool count is small, so deferral buys nothing and only strips the schema. The SDK - // treats only `false`/`0`/`no`/`off` as off, so the string must be "false" (not "0"/"100"). - // This is applied after `buildDaemonEnv`'s clear and is not in `KNOWN_PROVIDER_ENV_VARS`, - // so it is never stripped, and it reaches the Daytona sandbox like `ANTHROPIC_BASE_URL`. - env.ENABLE_TOOL_SEARCH = "false"; - - const deployment = request.deployment; - const selectedModel = request.model; - const baseUrl = request.endpoint?.baseUrl; - if (baseUrl) { - env.ANTHROPIC_BASE_URL = baseUrl; - logger(`claude base_url: ${baseUrl}`); - } - - if (deployment === "bedrock") { - env.CLAUDE_CODE_USE_BEDROCK = "1"; - const region = request.endpoint?.region; - if (region) { - env.AWS_REGION = region; - env.AWS_DEFAULT_REGION ??= region; - } - } else if (deployment === "vertex" || deployment === "vertex_ai") { - env.CLAUDE_CODE_USE_VERTEX = "1"; - } - - if ( - selectedModel && - (baseUrl || (deployment && CLAUDE_STRICT_DEPLOYMENTS.has(deployment))) - ) { - env.ANTHROPIC_MODEL = selectedModel; - env.ANTHROPIC_CUSTOM_MODEL_OPTION = selectedModel; - logger( - `claude model=${selectedModel} deployment=${deployment ?? ""}`, - ); - } -} - -/** - * Whether a requested-but-unsettable model fails the run (F-007). Strict by default on every - * harness path: a user who picks a model either runs that model or sees a loud error, never a - * silent (often pricier) fallback to the harness default. `AGENTA_AGENT_MODEL_STRICT=false` is - * the explicit opt-out that restores the legacy warn-and-fallback behavior. A run that requests - * no model is unaffected either way — it keeps the harness default. - */ -function modelResolutionStrict(): boolean { - return process.env.AGENTA_AGENT_MODEL_STRICT !== "false"; -} - -export interface SandboxAgentDeps extends BuildRunPlanDeps { - startSandboxAgent?: typeof SandboxAgent.start; - createPersist?: () => InMemorySessionPersistDriver; - createOtel?: typeof createSandboxAgentOtel; - buildDaemonEnv?: typeof buildDaemonEnv; - resolveDaemonBinary?: typeof resolveDaemonBinary; - buildSandboxProvider?: typeof buildSandboxProvider; - createCookieFetch?: typeof createCookieFetch; - createAcpFetch?: typeof createAcpFetch; - prepareWorkspace?: typeof prepareWorkspace; - prepareDaytonaPiAssets?: typeof prepareDaytonaPiAssets; - uploadToolMcpAssets?: typeof uploadToolMcpAssets; - probeCapabilities?: typeof probeCapabilities; - applyModel?: typeof applyModel; - startToolRelay?: typeof startToolRelay; - localRelayHost?: typeof localRelayHost; - sandboxRelayHost?: typeof sandboxRelayHost; - signSessionMountCredentials?: typeof signSessionMountCredentials; - signAgentMountCredentials?: typeof signAgentMountCredentials; - mountStorage?: typeof mountStorage; - mountStorageRemote?: typeof mountStorageRemote; - unmountStorage?: typeof unmountStorage; - discoverTunnelEndpoint?: typeof discoverTunnelEndpoint; - /** Per-harness transcript mounts (remote only; see mount.ts). */ - mountHarnessSessionDirs?: typeof mountHarnessSessionDirs; - responderFactory?: (request: AgentRunRequest) => Responder; - resolveRunLimits?: typeof resolveRunLimits; - createRunLimits?: typeof createRunLimits; - /** Session-continuity store override (tests inject their own; default is the process singleton). */ - sessionContinuityStore?: SessionContinuityStore; - /** Durable read-back/write-forward of the continuity store (tests inject fakes). */ - hydrateHarnessSessionFromDurable?: typeof hydrateHarnessSessionFromDurable; - syncHarnessSessionDurable?: typeof syncHarnessSessionDurable; - /** Durable read/write of the sandbox pointer, for the remote reconnect ladder. */ - readStoredSandboxPointer?: typeof readStoredSandboxPointer; - clearSandboxPointer?: typeof clearSandboxPointer; - writeSandboxPointer?: typeof writeSandboxPointer; - /** - * Resolve `{replicaId, ownerReplicaId}` for a session-owned local-sandbox run, so - * `acquireEnvironment` can fail loudly instead of silently cold-starting on a non-owner - * replica. The default claims the `owner` affinity key via the coordination plane and reads - * back the actual owner (`claimSessionOwnership`); tests inject their own. `authorization` is - * the run credential (the claim authenticates as the invoke caller). - */ - resolveLocalRunnerOwner?: ( - sessionId: string, - authorization: string, - ) => Promise<{ replicaId: string; ownerReplicaId: string | undefined }>; - log?: Log; -} - -async function defaultResolveLocalRunnerOwner( - sessionId: string, - authorization: string, -): Promise<{ replicaId: string; ownerReplicaId: string | undefined }> { - // No credential ⇒ the claim would 401; treat as "no known owner" (pass), never worse than today. - if (!authorization) { - return { replicaId: REPLICA_ID, ownerReplicaId: undefined }; - } - return claimSessionOwnership(sessionId, authorization); -} - -function isTransportEndpointDisconnected(err: unknown): boolean { - const message = String(err instanceof Error ? err.message : err); - const code = - typeof err === "object" && err !== null && "code" in err - ? String((err as { code?: unknown }).code) - : ""; - return ( - code === "ENOTCONN" || - message.includes("ENOTCONN") || - message.includes("Transport endpoint is not connected") - ); -} - -function containsTransportEndpointDisconnected(value: unknown): boolean { - const seen = new Set(); - - const visit = (current: unknown): boolean => { - if (typeof current === "string") { - return isTransportEndpointDisconnected(current); - } - if (current instanceof Error) { - return isTransportEndpointDisconnected(current); - } - if (!current || typeof current !== "object") { - return false; - } - if (seen.has(current)) { - return false; - } - seen.add(current); - - const code = - "code" in current ? String((current as { code?: unknown }).code) : ""; - if (code === "ENOTCONN") { - return true; - } - - if (Array.isArray(current)) { - return current.some(visit); - } - return Object.values(current as Record).some(visit); - }; - - return visit(value); -} - -/** - * Race sentinel: a run-limits deadline (total/idle/TTFB/per-tool-call) tripped mid-turn. Distinct - * from `PAUSED` so the prompt race can tell a human pause (keep the session) from a wedge deadline - * (end the turn as an error, letting the caller's teardown reclaim the sandbox). - */ -const RUN_LIMIT_TRIPPED = Symbol("run-limit-tripped"); - -/** - * The per-turn sink the session-lifetime listeners demux into. `runTurn` swaps a fresh one in - * at turn start (`env.currentTurn`) and the dispatch clears it at turn end. The `sandbox-agent` - * listener registries are plain Sets — an event with no listener is dropped and a permission - * request with no listener is CANCELLED — so the listeners stay attached for the session's whole - * life and route into whichever turn is active, with no detach/attach window between turns. - */ -interface CurrentTurn { - run: ReturnType; - pause: PendingApprovalPauseController; - toolRelay?: { ready?: Promise; stop: () => Promise }; - /** Route a session/update for the active turn (suppress + handleUpdate + pause re-sweep). */ - handleUpdate: (update: unknown) => void; - /** Route a permission reverse-RPC for the active turn (built by attachPermissionResponder). */ - onPermissionRequest?: (req: unknown) => void; -} - -/** - * A permission gate that paused the turn and can be answered later on the SAME live session. - * Recorded for a Claude ACP permission gate (keep-alive slice 2) or a Pi ACP permission gate - * (Pi approval parking: the gate rides the extension's `ctx.ui.confirm` onto the same ACP - * permission plane). NOT recorded for a client-tool MCP pause — that cannot be answered across - * a turn boundary and stays on the cold path. Existence of this record is what makes the - * dispatch park a paused session in `awaiting_approval` instead of tearing it down. - */ -export interface ParkedApproval { - /** Which gate paused; the dispatch resumes only a recognized type and treats others as cold. */ - gateType: ParkedApprovalGateType; - /** The ACP permission-request id, answered later via `session.respondPermission`. */ - permissionId: string; - /** The gated tool call's id — matched against the incoming approval envelope's toolCallId. */ - toolCallId: string; - /** The gated tool name (logging + the durable interaction row); never its args, in logs. */ - toolName: string | undefined; - /** The gated call's original args, used to seed the resume turn's trace/egress tool span. */ - args: unknown; - /** The durable interaction row token, resolved on the answer via the onResolveInteraction hook. */ - interactionToken: string; - /** The held original `prompt()` promise; the resume awaits it after `respondPermission`. */ - promptPromise?: Promise; -} - -/** Answer a parked Claude ACP permission gate on the live session (the keep-alive resume input). */ -export interface ResumeApprovalInput { - permissionId: string; - reply: "once" | "reject"; - toolCallId: string; - toolName: string | undefined; - args: unknown; - interactionToken: string; - promptPromise?: Promise; -} - -/** Per-turn options for `runTurn`. Absent (flag off / cold) means today's byte-identical path. */ -export interface RunTurnOptions { - /** A live continuation: send only the new user text instead of the full cold transcript. */ - continuation?: boolean; - /** - * The session was rehydrated via `session/load` (the patched `resumeSession`), so the harness - * already holds the prior turns natively. Like `continuation`, the prompt is only the new user - * text; `buildTurnText` must not run. Distinct field from `continuation` because the two arrive - * through different acquire paths (live pool checkout vs a fresh cold acquire that loaded an - * old session) — `runTurn` treats them identically for the text-selection decision. - */ - loaded?: boolean; - /** - * Keep-alive approval park mode: on a Claude ACP permission gate the pause keeps the session - * alive (no settle/abort/destroy) so a later resume can answer it. A non-parkable pause (Pi - * relay, client tool) still tears down exactly as today, so this is safe to set on any eligible - * keep-alive turn. - */ - approvalParkMode?: boolean; - /** A live approval resume: answer the parked gate and stream the continued prompt's events. */ - resume?: ResumeApprovalInput; -} - -/** - * Send only the new user text (not the full cold transcript) when the harness already holds the - * prior turns: a live continuation, or a session rehydrated via `session/load`. `runTurn` calls - * this, so a test that pins it pins the shipped decision. - */ -export function sendLastMessageOnly(opts: RunTurnOptions): boolean { - return Boolean(opts.continuation || opts.loaded); -} - -/** - * A session-scoped environment that can serve many turns. Everything expensive to build lives - * here (sandbox, session, internal tool-MCP server, mounted cwd, relay/temp dirs); `destroy()` - * is the one complete idempotent teardown the pool, the shutdown handler, and the cold path all - * call. Per-turn state rides `currentTurn`, swapped in by `runTurn`. - */ -export interface SessionEnvironment { - plan: RunPlan; - logger: Log; - deps: SandboxAgentDeps; - sandbox: any; - session: any; - sessionId: string; - model: string | undefined; - capabilities: HarnessCapabilities; - strictModel: boolean; - toolCallIndex: ReturnType; - /** The current turn's client-tool relay, read by the deferred ref baked into the MCP server. */ - clientToolRelayRef: { current?: ClientToolRelay }; - mcpAbort: AbortController; - runAgentDir: string | undefined; - otlpAuthFilePath: string | undefined; - mountCreds: MountCredentials | null; - agentMountCreds?: MountCredentials | null; - /** The mount's owning project id (keep-alive pool key FALLBACK scope, preferred is - * `runContext.project.id`); undefined when there is no mount. */ - mountProjectId?: string; - /** This run's resolved project scope (`projectScopeFor`: run-context preferred, mount - * fallback) — the same scope `poolKeyFor` keys on. Undefined when neither source yields - * one; a scoped `/kill` can then never claim this sandbox (see `destroyInFlightSandboxesForSession`). */ - projectScopeId?: string; - /** This acquire resumed the harness's native session via `session/load` (not cold). */ - loadedFromContinuity: boolean; - /** A remote, session-owned run whose sandbox can be parked (warm) rather than deleted at end. */ - resumable: boolean; - /** The conversation turn index this acquire's continuity record was read/written at. */ - continuityTurnIndex: number | undefined; - // Mutable teardown/turn state shared across acquire, runTurn, and destroy. - sessionDestroyRequested: boolean; - mountedCwd: string | undefined; - agentMountedPath?: string; - durableCwdSafeToDelete: boolean; - workspace: { cleanup: () => Promise } | undefined; - runtimeRemount: Promise | undefined; - closeToolMcp: (() => Promise) | undefined; - currentTurn?: CurrentTurn; - /** - * The unique ACP tool-call ids the LAST completed turn emitted (reset at each turn start). - * The keep-alive dispatch folds them into the expected next-history fingerprint at park time, - * so a tool-using turn still matches its own continuation (the FE keeps assistant tool parts). - */ - lastTurnToolCallIds: string[]; - /** - * The Claude ACP permission gate the LAST turn paused on, or undefined. Set only for a harness - * ACP permission gate, reset at each turn start; the dispatch reads it after a paused turn to - * decide whether to park in `awaiting_approval` and, on the next request, how to resume. - */ - parkedApproval?: ParkedApproval; - /** - * How many Claude ACP permission gates resolved to pendingApproval THIS turn (reset at turn - * start). More than one means parallel gates the single-gate resume cannot answer, so the - * dispatch does not park (tears down cold as today). - */ - approvalGateCount: number; - destroyed: boolean; - /** Complete, idempotent teardown selected from the typed teardown reason. */ - destroy: (opts?: { reason?: TeardownReason }) => Promise; - /** End the active turn: clear the current-turn sink (called before a park). */ - clearTurn: () => void; -} - -export type AcquireEnvironmentResult = - { ok: true; env: SessionEnvironment } | { ok: false; error: string }; - -/** - * Sign the session's durable mount up front so keep-alive can build a pool key (the mount's - * owning `projectId`, the FALLBACK project scope when the run carries no service-stamped - * `runContext.project.id`) and credential epoch without acquiring the whole environment. Returns - * exactly what the sign yielded: `null` when there is no session/credential to sign with, or - * the sign returned no usable mount (store unconfigured, 503, ephemeral fallback). The caller - * threads the result — null included — into `acquireEnvironment` as `presignedMount`, so the - * mount is signed exactly once per run on every path. A null result no longer forces a cold run - * on its own: the request still parks when the run context supplied a project scope, and only - * skips parking when NEITHER source yields one (`poolKeyFor` returns null). - */ -export async function resolveKeepaliveMount( - request: AgentRunRequest, - deps: SandboxAgentDeps = {}, -): Promise { - const logger = deps.log ?? log; - const sessionForMount = request.sessionId?.trim(); - const runCred = runCredential(request); - if (!sessionForMount || !runCred) return null; - const signMount = - deps.signSessionMountCredentials ?? signSessionMountCredentials; - return signMount(sessionForMount, { - apiBase: apiBase(), - authorization: runCred, - log: logger, - }); -} - -/** - * Build the session-scoped environment: sign the mount, build the run plan, start the sandbox, - * mount the durable cwd, prepare the workspace, probe capabilities, wire the internal tool-MCP - * server, and open the ACP session. Session-lifetime `onEvent`/`onPermissionRequest` listeners - * are attached once here and demux into `env.currentTurn`. - * - * Finalizers register incrementally on `env` as each resource is acquired; a mid-acquire failure - * runs `env.destroy()` (which null-checks every resource, so a half-built environment cannot - * leak) and returns `{ ok: false }`, mirroring today's shared teardown. When `presignedMount` is - * supplied (the keep-alive cold path already signed to build the pool key) the initial sign is - * skipped so the mount is signed once per run. - */ -export async function acquireEnvironment( - request: AgentRunRequest, - deps: SandboxAgentDeps = {}, - signal?: AbortSignal, - presignedMount?: MountCredentials | null, -): Promise { - const logger = deps.log ?? log; - const acquireStartedAt = Date.now(); - const timingLog = (stage: string, startedAt: number, fields = ""): void => { - const sandboxId = environment?.sandbox?.sandboxId ?? "-"; - const sessionId = - environment?.sessionId ?? request.sessionId?.trim() ?? "-"; - logger( - `[timing] stage=${stage} ms=${Math.round(Date.now() - startedAt)} sandbox=${sandboxId} session=${sessionId}${fields}`, - ); - }; - - // Local multi-runner fails loudly. Session-owned + local-sandbox only (a non-session run - // has no cross-replica identity to protect, and a remote sandbox has no runner-local pooled - // state to protect it FROM). The resolver claims the `owner` affinity key and reads the actual - // owner back; a KNOWN different owner throws (never a silent wrong-host cold start). - const continuitySessionForOwnership = request.sessionId?.trim(); - if ( - continuitySessionForOwnership && - resolvesToLocalProvider(request.sandbox) - ) { - const { replicaId, ownerReplicaId } = await ( - deps.resolveLocalRunnerOwner ?? defaultResolveLocalRunnerOwner - )(continuitySessionForOwnership, runCredential(request)); - try { - assertLocalRunnerOwnership( - continuitySessionForOwnership, - replicaId, - ownerReplicaId, - ); - } catch (err) { - return { ok: false, error: conciseError(err, request.harness ?? "") }; - } - } - - // Sign BEFORE buildRunPlan so the prefix is available for the durable cwd derivation. - // Inputs (sessionId, apiBase, credential) are independent of the plan. Best-effort: null on - // failure leaves durableCwd undefined and buildRunPlan falls back to the ephemeral path. - const sessionForMount = request.sessionId?.trim(); - const runCred = runCredential(request); - const signMount = - deps.signSessionMountCredentials ?? signSessionMountCredentials; - let mountCreds: MountCredentials | null = - presignedMount !== undefined - ? presignedMount - : sessionForMount && runCred - ? await signMount(sessionForMount, { - apiBase: apiBase(), - authorization: runCred, - log: logger, - }) - : null; - // A session-owned run expects a durable session cwd mount. When signing returns nothing the run - // still proceeds on an ephemeral cwd (behavior unchanged, RSH-11); emit one structured warning - // keyed by mount kind so durable-to-ephemeral degradation is measurable, not silent. - if (sessionForMount && !mountCreds) { - logger( - `mount degraded kind=session_cwd cause=sign_returned_no_mount session=${sessionForMount}`, - ); - } - - const artifactId = request.runContext?.workflow?.artifact?.id?.trim(); - const signAgentMount = - deps.signAgentMountCredentials ?? signAgentMountCredentials; - const agentMountCreds: MountCredentials | null = - artifactId && runCred - ? await signAgentMount(artifactId, { - apiBase: apiBase(), - authorization: runCred, - log: logger, - }) - : null; - // A workflow-artifact run expects an agent mount; same structured degrade signal when unsigned. - if (artifactId && !agentMountCreds) { - logger( - `mount degraded kind=agent_mount cause=sign_returned_no_mount artifact=${artifactId}`, - ); - } - // Derive the durable cwd from the sign prefix (one source of truth, both providers). - // local: /tmp/agenta/ — daytona: /home/sandbox/agenta/ - // is already "mounts//", so no extra slug is needed. - let durableCwd: string | undefined; - if (mountCreds?.prefix) { - const isDaytonaReq = - (request.sandbox ?? loadRunnerConfig().providers.default) === "daytona"; - durableCwd = isDaytonaReq - ? `/home/sandbox/agenta/${mountCreds.prefix}` - : `/tmp/agenta/${mountCreds.prefix}`; - } - - const planResult = buildRunPlan(request, { - sandboxProvider: deps.sandboxProvider, - createLocalCwd: deps.createLocalCwd, - createDaytonaCwd: deps.createDaytonaCwd, - durableCwd, - resolveSkillDirs: deps.resolveSkillDirs, - log: logger, - }); - if (!planResult.ok) return { ok: false, error: planResult.error }; - const plan = planResult.plan; - const piSkillSnapshot = resolvePiSkillSnapshot(plan); - const agentMountDir = agentMountCreds ? agentMountPath(plan.cwd) : undefined; - - // Clear-then-apply (Security rule 5): on a managed run (credentialMode "env") the daemon - // inherits NONE of the sidecar's own provider keys, so only the resolved `plan.secrets` are - // present and an inherited key for another provider cannot leak. For runtime_provided/none/ - // un-migrated runs the harness uses its own login, so the inherited keys stay. - const clearProviderEnv = plan.credentialMode === "env"; - const env = (deps.buildDaemonEnv ?? buildDaemonEnv)(plan.acpAgent, { - clearProviderEnv, - provider: request.provider, - deployment: request.deployment, - }); - Object.assign(env, plan.secrets); // apply only the resolved provider keys - applyClaudeConnectionEnv(env, request, plan.acpAgent, logger); - const piSessionDir = configurePiSessionWorkspace(plan, env); - configurePiSkillSnapshot(piSkillSnapshot, env); - const strictModel = modelResolutionStrict(); - // Pi self-instruments locally: propagate the trace context + public tool metadata into Pi - // via the Agenta extension. Tool execution always relays back to this runner, which keeps - // private specs, scoped env, callback endpoints, and callback auth in memory. - // local Pi's OTLP bearer rides a runner-written 0600 file, never a plain env var — - // Daytona never receives telemetry env here at all (`!plan.isDaytona` gates it off above). - const otlpAuthFilePath = - plan.isPi && !plan.isDaytona ? `${plan.relayDir}.otlp-auth` : undefined; - const otlpAuthorization = - request.telemetry?.exporters?.otlp?.headers?.authorization; - if (otlpAuthFilePath && otlpAuthorization) { - writeOtlpAuthFile(otlpAuthFilePath, otlpAuthorization, logger); - } - const piExtEnv = plan.isPi - ? buildPiExtensionEnv(request, !plan.isDaytona, { - relayDir: plan.relayDir, - usageOutPath: plan.usageOutPath, - otlpAuthFilePath, - builtinGatingActive: plan.builtinGatingActive, - builtinGrants: plan.builtinGrants, - // The materialized skill names (author + forced `_agenta.*`) so Pi's own agent span - // records which skills loaded; local Pi self-instruments, so the runner's sandbox-agent - // otel has no span to stamp here. - skills: plan.skillDirs.map((s) => s.name), - }) - : {}; - // Daytona's provider is built from `piExtEnv` rather than the local daemon env. Keep the - // transcript location in both environment slices so Pi and pi-acp see the same durable path - // regardless of provider. - if (piSessionDir) piExtEnv.PI_CODING_AGENT_SESSION_DIR = piSessionDir; - configurePiSkillSnapshot(piSkillSnapshot, piExtEnv); - Object.assign(env, piExtEnv); // local daemon inherits it; daytona gets it via envVars - logger( - `tools=${plan.toolSpecs.length} executableTools=${plan.executableToolSpecs.length} ` + - `piPublicTools=${piExtEnv.AGENTA_AGENT_TOOLS_PUBLIC_SPECS ? "yes" : "no"}`, - ); - if (!plan.isPi && plan.isDaytona) { - const omittedClientTools = plan.toolSpecs - .filter((spec) => spec.kind === "client") - .map((spec) => spec.name); - if (omittedClientTools.length > 0) { - logger( - `omitting client tools from Daytona stdio MCP shim: ${omittedClientTools.join(", ")}`, - ); - } - } - // Translate a managed OpenAI-compatible custom connection into Pi's native models.json plan - // (design Decision 5). Non-applicable requests yield no plan (current behavior); an applicable - // but incomplete request throws — captured here and re-thrown inside the try below so the - // engine's own catch turns it into `{ ok: false, error }` and a visible error frame (fail loud, - // never a silent fall-back to a default provider). Only the env var NAME enters the plan. - let piModelConfig: PiModelConfigPlan | undefined; - let piModelConfigError: Error | undefined; - if (plan.isPi) { - try { - piModelConfig = buildPiModelConfigPlan(request, plan.secrets); - } catch (err) { - piModelConfigError = err as Error; - } - } - if (piModelConfig) { - logger( - `pi model-config plan provider=${piModelConfig.providerId} api=${piModelConfig.api} ` + - `model=${piModelConfig.models.map((m) => m.id).join(",")}`, - ); - } - - // undefined is fine: the local provider runs its own resolution and errors clearly. - const binaryPath = (deps.resolveDaemonBinary ?? resolveDaemonBinary)(); - const localPiAssets = prepareLocalPiAssets({ - plan, - env, - piModelConfig, - log: logger, - }); - let runAgentDir = localPiAssets.dir; - // Fail closed (Decision 6): a local managed custom run whose models.json could not be written - // must stop rather than run on a default provider. Recorded here (the write ran above) and - // thrown inside the try below, like the permission-extension gate. - const localModelConfigUnwritable = - plan.isPi && - !plan.isDaytona && - !!piModelConfig && - !localPiAssets.modelConfigWritten; - // Fail closed (Decision 2): when the policy could gate a Pi built-in tool but the permission - // extension did not install, the run must stop rather than run those tools unprotected. Recorded - // here (the install ran above) and thrown inside the try below so the engine's own catch turns it - // into `{ ok: false, error }` and a visible error frame. `builtinGatingActive` false means - // allow-everything, where the extension is not needed and a failed install is harmless. - const localBuiltinGatingUnenforceable = - plan.isPi && - !plan.isDaytona && - plan.builtinGatingActive && - !localPiAssets.extensionInstalled; - - // A local Claude subscription run reads and writes the operator's read-write mounted login - // DIRECTLY: `buildDaemonEnv` already carried `CLAUDE_CONFIG_DIR` (the mount) into the daemon env, - // and there is deliberately no per-run copy. Claude refreshes its OAuth token mid-run and writes - // it back to its config dir; copying that dir per run would discard the refresh, so the next run - // would fail as soon as the provider rotated the refresh token. The harness owns its own token - // lifecycle, exactly like a normal local install (interface.md section 6). buildRunPlan already - // rejected a runtime_provided Claude run with no configured CLAUDE_CONFIG_DIR. - - logger(`harness=${plan.harness} sandbox=${plan.sandboxId} cwd=${plan.cwd}`); - - // The resolved model ref as it reaches the runner (key NAMES only, never values) — the one - // line that answers "what model/provider/deployment/credential did this run actually use". - logger( - `resolved model=${request.model ?? ""} provider=${request.provider ?? ""} ` + - `deployment=${request.deployment ?? ""} ` + - `connection=${request.connection ? `${request.connection.mode}:${request.connection.slug ?? "-"}` : ""} ` + - `secretKeys=[${Object.keys(request.secrets ?? {}).join(",")}]`, - ); - - // The shared client-tool relay reference (the deferred ref baked into the MCP server reads it; - // each turn's `runTurn` sets `.current`). A `tools/call` can only arrive during a prompt — - // long after the relay is wired — so the server captures this reference and it resolves to the - // real relay before any call lands. - const clientToolRelayRef: { current?: ClientToolRelay } = {}; - const deferredClientToolRelay: ClientToolRelay = { - onClientTool: (req) => - clientToolRelayRef.current - ? clientToolRelayRef.current.onClientTool(req) - : Promise.resolve("deny" as ClientToolOutcome), - onPause: (req) => clientToolRelayRef.current?.onPause?.(req), - }; - - // Aborts any in-flight loopback `tools/call` (a paused Claude client tool) on pause/teardown, - // so its handler is torn down deterministically and cannot write a result after the turn ends. - const mcpAbort = new AbortController(); - - const environment: SessionEnvironment = { - plan, - logger, - deps, - sandbox: undefined, - session: undefined, - sessionId: resolveRunSessionId(request, ""), - model: undefined, - capabilities: {}, - strictModel, - toolCallIndex: createToolCallCorrelationIndex(), - clientToolRelayRef, - mcpAbort, - runAgentDir, - otlpAuthFilePath, - mountCreds, - agentMountCreds, - mountProjectId: mountCreds?.projectId, - projectScopeId: projectScopeFor(request, mountCreds?.projectId)?.id, - loadedFromContinuity: false, - resumable: false, - continuityTurnIndex: undefined, - sessionDestroyRequested: false, - mountedCwd: undefined, - agentMountedPath: undefined, - durableCwdSafeToDelete: true, - // Local runs get a plain rmSync cleanup for the throwaway cwd; Daytona has none on this host. - workspace: plan.isDaytona - ? undefined - : { - cleanup: async () => - rmSync(plan.cwd, { recursive: true, force: true }), - }, - runtimeRemount: undefined, - closeToolMcp: undefined, - currentTurn: undefined, - lastTurnToolCallIds: [], - parkedApproval: undefined, - approvalGateCount: 0, - destroyed: false, - destroy: async () => {}, - clearTurn: () => {}, - }; - - environment.clearTurn = () => { - environment.currentTurn = undefined; - }; - - // The one complete, idempotent teardown — the same steps the old per-run `finally` ran, in the - // same order. Every resource is null-checked, so it is safe after a partial acquire and safe to - // call twice (the guard returns on a second call). It must never throw. - environment.destroy = async (opts?: { reason?: TeardownReason }) => { - if (environment.destroyed) return; - environment.destroyed = true; - await environment.runtimeRemount?.catch(() => {}); - inFlightSandboxes.delete(environment); - await environment.currentTurn?.toolRelay?.stop().catch(() => {}); - // Teardown backstop: destroy any in-flight loopback `tools/call` before closing the server. - environment.mcpAbort.abort(); - await environment.closeToolMcp?.().catch(() => {}); - // Graceful `session/cancel` BEFORE tearing down the daemon, or the ACP adapter subprocess - // reparents to PID 1 and never exits. Skip if the pause path already sent it. - if (environment.session && !environment.sessionDestroyRequested) - await environment.sandbox - ?.destroySession?.(environment.session.id) - .catch(() => {}); - const disposition = teardownDisposition(opts?.reason ?? "failed-turn"); - let parked = false; - if ( - disposition === "stop" && - plan.isDaytona && - environment.sandbox?.pauseSandbox - ) { - const sandboxLogId = environment.sandbox.sandboxId ?? plan.sandboxId; - try { - await environment.sandbox.pauseSandbox(); - parked = true; - logger(`parked sandbox=${sandboxLogId}`); - } catch (err) { - logger( - `pause failed sandbox=${sandboxLogId}: ${conciseError(err, plan.harness)}`, - ); - } - } - if (!parked) await environment.sandbox?.destroySandbox().catch(() => {}); - await environment.sandbox?.dispose().catch(() => {}); - // Unmount the durable cwd BEFORE removing the dir: data lives in the store, only the host - // mountpoint is torn down. If unmount is not CONFIRMED gone, skip the delete: rmSync must - // never run against a possibly-live FUSE mount into the durable store. - if (environment.mountedCwd) { - environment.durableCwdSafeToDelete = await ( - environment.deps.unmountStorage ?? unmountStorage - )(environment.mountedCwd, { log }).catch(() => false); - } - if (!parked && !plan.isDaytona && environment.agentMountedPath) { - const agentMountSafeToDelete = await ( - environment.deps.unmountStorage ?? unmountStorage - )(environment.agentMountedPath, { log }).catch(() => false); - if (agentMountSafeToDelete) { - try { - rmSync(environment.agentMountedPath, { - recursive: true, - force: true, - }); - } catch (err) { - logger( - `agent mountpoint cleanup failed path=${environment.agentMountedPath}: ${conciseError(err, plan.harness)}`, - ); - } - } - } - if (!environment.durableCwdSafeToDelete) { - logger( - `durable cwd unmount not confirmed, skipping workspace cleanup cwd=${plan.cwd}`, - ); - } else { - await environment.workspace?.cleanup().catch(() => {}); - } - // The per-run Agenta agent dir (skills isolation) is throwaway; remove it too. This is only - // ever a temp dir: a subscription run leaves `runAgentDir` undefined precisely so that the - // operator's mounted login (which the harness runs out of directly) is never deleted here. - if (environment.runAgentDir) - rmSync(environment.runAgentDir, { recursive: true, force: true }); - // Backstop: the extension deletes this on read; remove it here too in case the harness never - // started (or crashed before reading it), so the bearer never lingers. - if (environment.otlpAuthFilePath) - rmSync(environment.otlpAuthFilePath, { force: true }); - // Remove the per-run skills temp root the materializer created (success or error). - plan.skillsCleanup(); - }; - - let agentMountGuidanceActive = false; - const activateAgentMountGuidance = async (): Promise => { - const mountedPath = environment.agentMountedPath; - if (!mountedPath || agentMountGuidanceActive) return; - agentMountGuidanceActive = true; - - // Only advertise durable storage after the mount is confirmed active. Local daemon env is - // still mutable here because local mounts run before SandboxAgent.start below. Daytona cannot - // change daemon env after sandbox creation, so its harness discovers the mount through the - // post-mount system-prompt channel and the cwd-local agent-files symlink instead. - if (!plan.isDaytona) { - env[AGENT_MOUNT_ENV_VAR] = mountedPath; - piExtEnv[AGENT_MOUNT_ENV_VAR] = mountedPath; - } - if (!plan.isPi) return; - - plan.appendSystemPrompt = combineAppendSystemPrompt( - plan.appendSystemPrompt, - AGENT_MOUNT_SYSTEM_PROMPT_SEGMENT, - ); - plan.hasSystemPrompt = true; - if (plan.isDaytona) { - await uploadSystemPromptToSandbox( - environment.sandbox, - DAYTONA_PI_DIR, - plan.systemPrompt, - plan.appendSystemPrompt, - logger, - ); - return; - } - if (environment.runAgentDir) { - writeSystemPromptLocal( - environment.runAgentDir, - plan.systemPrompt, - plan.appendSystemPrompt, - logger, - ); - return; - } - // Discarding `.extensionInstalled` here is safe, and a fail-closed throw here would be - // unsound anyway (both callers wrap this in a mount try/catch that logs and continues, so a - // throw could not stop the run). Reachability: managed/none local Pi runs always created a - // throwaway dir in the first prepareLocalPiAssets call, so `environment.runAgentDir` is set - // for them and they returned above — only the subscription (runtime_provided) path reaches - // this re-prep. That path installs into the SAME operator mount the first call already - // installed into, and the fail-closed gating check right after that first call stopped the - // run when the install was required but failed. So by the time this runs, either enforcement - // is not needed (policy allows everything) or the extension file is already on disk from the - // verified first install; a transient failure here cannot remove it. - runAgentDir = prepareLocalPiAssets({ plan, env, log: logger }).dir; - environment.runAgentDir = runAgentDir; - }; - - // --- local durable cwd mount helpers (session-scoped, close over environment) ------ // - const mountLocalDurableCwd = async (reason: string): Promise => { - if (!environment.mountCreds || plan.isDaytona) return false; - logger( - `local durable cwd mount (${reason}) session=${sessionForMount} cwd=${plan.cwd}`, - ); - environment.durableCwdSafeToDelete = false; - const mounted = await (deps.mountStorage ?? mountStorage)( - plan.cwd, - environment.mountCreds, - { - log: logger, - }, - ); - if (mounted) { - environment.mountedCwd = plan.cwd; - return true; - } - // A false result means mountStorage stopped the attempt and confirmed the path detached. - environment.durableCwdSafeToDelete = true; - return false; - }; - const mountLocalAgentCwd = async (): Promise => { - if (!environment.agentMountCreds || plan.isDaytona) return false; - const mountPath = agentMountPath(plan.cwd); - if (environment.agentMountedPath === mountPath) return true; - try { - mkdirSync(mountPath, { recursive: true }); - if ( - !(await (deps.mountStorage ?? mountStorage)( - mountPath, - environment.agentMountCreds, - { log: logger }, - )) - ) { - // false means mountStorage confirmed detach is safe. This path is a sibling of the - // session cwd, so workspace cleanup cannot remove the failed mountpoint stub. - rmSync(mountPath, { recursive: true, force: true }); - return false; - } - environment.agentMountedPath = mountPath; - await seedAgentReadme(mountPath, { log: logger }); - await linkAgentFiles(plan.cwd, mountPath, { log: logger }); - await activateAgentMountGuidance(); - return true; - } catch (err) { - logger( - `local agent mount failed artifact=${artifactId}: ${conciseError(err, plan.harness)}`, - ); - return false; - } - }; - let localAgentMountEnotconnRemounts = 0; - const reSignAndRemountLocalAgentMount = async (): Promise => { - if (!artifactId || !runCred || plan.isDaytona) return false; - if ( - localAgentMountEnotconnRemounts >= - LOCAL_DURABLE_CWD_ENOTCONN_REMOUNT_LIMIT - ) { - logger( - `local agent mount ENOTCONN remount limit reached artifact=${artifactId} path=${agentMountPath(plan.cwd)}`, - ); - return false; - } - localAgentMountEnotconnRemounts += 1; - logger( - `local agent mount ENOTCONN artifact=${artifactId}; re-signing and remounting`, - ); - const fresh = await signAgentMount(artifactId, { - apiBase: apiBase(), - authorization: runCred, - log: logger, - }); - if (!fresh) { - logger( - `local agent mount re-sign returned no credentials artifact=${artifactId}`, - ); - return false; - } - environment.agentMountCreds = fresh; - // Clear the marker so mountLocalAgentCwd remounts instead of short-circuiting. - environment.agentMountedPath = undefined; - return mountLocalAgentCwd(); - }; - let localDurableCwdEnotconnRemounts = 0; - const reSignAndRemountLocalCwd = async (): Promise => { - if (!sessionForMount || !runCred || plan.isDaytona) return false; - if ( - localDurableCwdEnotconnRemounts >= - LOCAL_DURABLE_CWD_ENOTCONN_REMOUNT_LIMIT - ) { - logger( - `local durable cwd ENOTCONN remount limit reached session=${sessionForMount} cwd=${plan.cwd}`, - ); - return false; - } - localDurableCwdEnotconnRemounts += 1; - logger( - `local durable cwd ENOTCONN session=${sessionForMount} cwd=${plan.cwd}; re-signing and remounting`, - ); - const fresh = await signMount(sessionForMount, { - apiBase: apiBase(), - authorization: runCred, - log: logger, - }); - if (!fresh) { - logger( - `local durable cwd re-sign returned no credentials session=${sessionForMount}`, - ); - return false; - } - environment.mountCreds = fresh; - return mountLocalDurableCwd("enotconn-retry"); - }; - const remountLocalCwdAfterRuntimeEnotconn = (event: unknown): void => { - if (plan.isDaytona) return; - // The event cannot say which mount broke; remount every eligible one (alive mounts no-op). - const cwdEligible = !!environment.mountCreds && !!environment.mountedCwd; - const agentEligible = - !!environment.agentMountCreds && !!environment.agentMountedPath; - if (!cwdEligible && !agentEligible) return; - if ( - environment.runtimeRemount || - !containsTransportEndpointDisconnected(event) - ) - return; - logger( - `local durable mount ENOTCONN observed in ACP event session=${sessionForMount} cwd=${plan.cwd}; re-signing and remounting`, - ); - environment.runtimeRemount = (async () => { - const cwdOk = cwdEligible ? await reSignAndRemountLocalCwd() : true; - const agentOk = agentEligible - ? await reSignAndRemountLocalAgentMount() - : true; - return cwdOk && agentOk; - })().catch((err) => { - logger( - `local durable mount runtime remount failed session=${sessionForMount}: ${conciseError(err, plan.harness)}`, - ); - return false; - }); - }; - - try { - // Fail loud before any sandbox/mount infra spins up: an applicable-but-incomplete - // OpenAI-compatible custom request is a hard error, never a silent fall-back (Decision 5). - if (piModelConfigError) { - throw piModelConfigError; - } - // Fail closed before any sandbox/mount infra spins up: a local Pi run whose policy could gate a - // built-in tool cannot proceed without the permission extension installed (Decision 2). - if (localBuiltinGatingUnenforceable) { - throw new Error(PI_PERMISSION_EXTENSION_UNAVAILABLE_MESSAGE); - } - // Fail closed: a local managed custom run whose models.json could not be materialized cannot - // fall through to a default provider (Decision 6). - if (localModelConfigUnwritable) { - throw new Error(PI_MODEL_CONFIG_WRITE_FAILED_MESSAGE); - } - // Persist events in-process so a follow-up turn can resume by session id. - const persist = - deps.createPersist?.() ?? new InMemorySessionPersistDriver(); - const startSandboxAgent = - deps.startSandboxAgent ?? - ((options: Parameters[0]) => - SandboxAgent.start(options)); - // Local geesefs runs on the host, so mount before spawning the daemon. This lets the - // mount-success path add guidance/env atomically, while a failed mount starts a normal - // scratch-only harness with no false durable-storage signal. - if (environment.mountCreds && !plan.isDaytona) { - await mountLocalDurableCwd("initial"); - } - if (environment.agentMountCreds && !plan.isDaytona) { - await mountLocalAgentCwd(); - } - const sandboxProvider = (deps.buildSandboxProvider ?? buildSandboxProvider)( - plan.sandboxId, - env, - binaryPath, - piExtEnv, - plan.secrets, - plan.sandboxPermission, - ); - const startOptions = { - sandbox: sandboxProvider, - persist, - // Propagate caller cancellation (a client disconnect on the streaming HTTP edge) so an - // in-flight run aborts instead of finishing unobserved. `destroy` still disposes. - ...(signal ? { signal } : {}), - // Long-timeout undici dispatcher so a paused HITL turn is not reaped by undici's default - // headersTimeout; Daytona additionally carries the per-sandbox auth cookie. - fetch: plan.isDaytona - ? (deps.createCookieFetch ?? createCookieFetch)() - : (deps.createAcpFetch ?? createAcpFetch)(), - }; - // A stored sandbox id is trusted: reconnect it by id and let reconnect converge its network - // policy to this run's plan. Any reconnect failure falls through to a fresh create. Snapshot - // and image drift are accepted as per-conversation version pinning, not grounds for a rebuild. - const storedSandboxPointer = - plan.isDaytona && sessionForMount && runCred - ? await (deps.readStoredSandboxPointer ?? readStoredSandboxPointer)( - sessionForMount, - { authorization: runCred, log: logger }, - ) - : undefined; - if (storedSandboxPointer) { - const sandboxStartStartedAt = Date.now(); - try { - environment.sandbox = await startSandboxAgent({ - ...startOptions, - sandboxId: storedSandboxPointer.sandboxId, - }); - logger( - `reconnected sandbox=${storedSandboxPointer.sandboxId} session=${sessionForMount}`, - ); - } catch (err) { - logger( - `reconnect failed sandbox=${storedSandboxPointer.sandboxId}, creating fresh: ${conciseError(err, plan.harness)}`, - ); - if ( - err instanceof DaytonaReconnectTerminalError && - sessionForMount && - runCred - ) { - // The post-hydrate write later in acquire is authoritative. This clear only prevents - // repeated doomed reconnects if acquire fails before reaching that write. Hydrate - // first: after a runner restart the in-memory store is behind the durable - // latest_turn_index, and an unhydrated guard token would be rejected as stale. - await ( - deps.hydrateHarnessSessionFromDurable ?? - hydrateHarnessSessionFromDurable - )( - sessionForMount, - plan.harness, - deps.sessionContinuityStore ?? sessionContinuityStore, - { authorization: runCred, log: logger }, - ); - await (deps.clearSandboxPointer ?? clearSandboxPointer)( - sessionForMount, - nextTurnIndex( - sessionForMount, - deps.sessionContinuityStore ?? sessionContinuityStore, - ), - { authorization: runCred, log: logger }, - ); - } - } finally { - timingLog("sandbox_start", sandboxStartStartedAt, " mode=reconnect"); - } - } - if (!environment.sandbox) { - const sandboxStartStartedAt = Date.now(); - try { - environment.sandbox = await startSandboxAgent(startOptions); - } finally { - timingLog("sandbox_start", sandboxStartStartedAt, " mode=create"); - } - } - environment.resumable = Boolean(plan.isDaytona && sessionForMount); - // Track the live handle so a shutdown signal handler can delete it if `destroy` is skipped by - // a process KILL; removed in `destroy` on every normal exit so it is never double-deleted. - if (environment.sandbox) inFlightSandboxes.add(environment); - - // On Daytona, push the harness login, the extension, and AGENTS.md into the remote sandbox. - // For a non-Pi harness with executable tools, also push the in-sandbox stdio MCP shim - // assets (bundle + public-specs file): a non-Pi harness in the sandbox cannot reach the - // runner-loopback HTTP MCP channel, so the harness's ACP adapter spawns the uploaded shim - // as the internal stdio MCP server instead. Uploaded unconditionally for non-Pi (the - // capability probe runs later; a harness that turns out to lack MCP fails loud in - // `assertRequiredCapabilities` below). Pi delivers via its extension; local non-Pi uses - // the loopback HTTP channel — neither needs this. The upload helper THROWS when the shim - // cannot be delivered (fail loud — this path requires it). - let internalToolMcp: ToolMcpAssets | undefined; - if (plan.isDaytona) { - const daytonaExtensionInstalled = await ( - deps.prepareDaytonaPiAssets ?? prepareDaytonaPiAssets - )({ - sandbox: environment.sandbox, - plan: { ...plan, skillDirs: [] }, - piModelConfig, - log: logger, - }); - // Fail closed (Decision 2): same guarantee as the local path. A genuine upload failure on the - // Daytona sandbox stops the run rather than running Pi's built-in tools unprotected. - if (plan.isPi && plan.builtinGatingActive && !daytonaExtensionInstalled) { - throw new Error(PI_PERMISSION_EXTENSION_UNAVAILABLE_MESSAGE); - } - if (!plan.isPi && plan.executableToolSpecs.length > 0) { - internalToolMcp = await ( - deps.uploadToolMcpAssets ?? uploadToolMcpAssets - )( - environment.sandbox, - plan.toolMcpDir, - advertisedToolSpecs(plan.executableToolSpecs), - logger, - ); - } - } - - // Durable cwd: mount BEFORE createSession (so the session opens inside it) and BEFORE - // workspace materialization (so AGENTS.md, harness files, and skills land in the durable - // prefix instead of being hidden under the FUSE mount). - if (environment.mountCreds && plan.isDaytona) { - const mountsStartedAt = Date.now(); - try { - // Mount against the store's own endpoint when the sandbox can reach it (public S3); fall - // back to the tunnel only for an in-network store. No tunnel + in-network store => skip. - const storeEndpoint = environment.mountCreds.endpoint; - const endpoint = storeReachableFromSandbox(storeEndpoint) - ? undefined - : ((await (deps.discoverTunnelEndpoint ?? discoverTunnelEndpoint)({ - log: logger, - })) ?? undefined); - const canMount = storeReachableFromSandbox(storeEndpoint) || !!endpoint; - if ( - canMount && - (await (deps.mountStorageRemote ?? mountStorageRemote)( - environment.sandbox, - plan.cwd, - environment.mountCreds, - { - endpoint, - log: logger, - }, - )) - ) { - logger(`remote durable cwd active for session=${sessionForMount}`); - } - // Per-harness session/transcript-dir mounts, remote-only by construction (this whole - // branch is `plan.isDaytona`) — local runs never reach here, so they stay mount-free/ - // byte-identical. Transcript mounts derive from the session contract (a durable cwd mount - // is active), with no separate public switch or credential/session-id path. - if (canMount && sessionForMount && runCred) { - const dirs = harnessSessionMounts( - plan.acpAgent, - "/home/sandbox", - DAYTONA_PI_DIR, - ); - await (deps.mountHarnessSessionDirs ?? mountHarnessSessionDirs)( - environment.sandbox, - sessionForMount, - dirs, - endpoint, - { - apiBase: apiBase(), - authorization: runCred, - log: logger, - }, - ); - } - } finally { - timingLog("mounts", mountsStartedAt); - } - } - if ( - environment.agentMountCreds && - agentMountDir && - plan.isDaytona && - !environment.agentMountedPath - ) { - const agentMountStartedAt = Date.now(); - try { - const storeEndpoint = environment.agentMountCreds.endpoint; - const endpoint = storeReachableFromSandbox(storeEndpoint) - ? undefined - : ((await (deps.discoverTunnelEndpoint ?? discoverTunnelEndpoint)({ - log: logger, - })) ?? undefined); - const canMount = storeReachableFromSandbox(storeEndpoint) || !!endpoint; - const mountPath = agentMountDir; - if ( - canMount && - (await (deps.mountStorageRemote ?? mountStorageRemote)( - environment.sandbox, - mountPath, - environment.agentMountCreds, - { endpoint, log: logger }, - )) - ) { - environment.agentMountedPath = mountPath; - await seedAgentReadmeRemote(environment.sandbox, mountPath, { - log: logger, - }); - await linkAgentFilesRemote(environment.sandbox, plan.cwd, mountPath, { - log: logger, - }); - await activateAgentMountGuidance(); - logger(`remote agent mount active for artifact=${artifactId}`); - } - } catch (err) { - logger( - `remote agent mount failed artifact=${artifactId}: ${conciseError(err, plan.harness)}`, - ); - } finally { - timingLog("agent_mount", agentMountStartedAt); - } - } - - const prepareWorkspaceStartedAt = Date.now(); - try { - environment.workspace = await (deps.prepareWorkspace ?? prepareWorkspace)( - { - sandbox: environment.sandbox, - plan, - piSkillSnapshot, - log: logger, - }, - ); - } catch (err) { - if ( - !plan.isDaytona && - environment.mountCreds && - isTransportEndpointDisconnected(err) && - (await reSignAndRemountLocalCwd()) - ) { - logger( - `retrying workspace preparation after local durable cwd remount`, - ); - environment.workspace = await ( - deps.prepareWorkspace ?? prepareWorkspace - )({ - sandbox: environment.sandbox, - plan, - piSkillSnapshot, - log: logger, - }); - } else { - throw err; - } - } finally { - timingLog("prepare_workspace", prepareWorkspaceStartedAt); - } - - // Pi native transcripts belong to the conversation workspace, not the temporary agent - // directory that holds credentials, settings, extensions, skills, and system prompts. - // The cwd mount is already active here on local and Daytona before Pi starts. - if (piSessionDir) { - if (plan.isDaytona) { - await environment.sandbox.mkdirFs({ path: piSessionDir }); - } else { - mkdirSync(piSessionDir, { recursive: true }); - } - } - - // Sandbox-start invariant: `startSandboxAgent` must hand back a usable handle. - assert( - environment.sandbox && - typeof environment.sandbox.createSession === "function", - `sandbox provider '${plan.sandboxId}' returned no usable sandbox handle`, - ); - - // Probe what this harness supports and branch on capabilities, not on the harness name. - const probeCapabilitiesStartedAt = Date.now(); - let probed; - try { - probed = await (deps.probeCapabilities ?? probeCapabilities)( - environment.sandbox, - plan.acpAgent, - ); - } finally { - timingLog("probe_capabilities", probeCapabilitiesStartedAt); - } - const capabilities = probed.capabilities; - environment.capabilities = capabilities; - - // Fail loud (A7): a run that REQUIRES a capability the harness lacks errors specifically - // rather than silently dropping the behavior. - assertRequiredCapabilities({ - harness: plan.harness, - isPi: plan.isPi, - probed, - toolSpecs: plan.toolSpecs, - log: logger, - }); - - const sessionMcp = await buildSessionMcpServers({ - isPi: plan.isPi, - capabilities, - harness: plan.harness, - isDaytona: plan.isDaytona, - toolSpecs: plan.toolSpecs, - userMcpServers: request.mcpServers, - relayDir: plan.relayDir, - clientToolRelay: deferredClientToolRelay, - signal: mcpAbort.signal, - // The uploaded in-sandbox stdio MCP shim assets, set only on Daytona + non-Pi + - // executable-tools; advertises the gateway tools the loopback channel cannot reach - // from inside the sandbox. No server to close for this entry (the harness owns the - // shim process), so `sessionMcp.close` semantics are unchanged. - internalToolMcp, - log: logger, - }); - // Close the internal gateway-tool MCP server (if one started) when the session is destroyed. - environment.closeToolMcp = sessionMcp.close; - - // Shared session-init payload for both the createSession and continuity-resume paths below. - // Built as a plain variable (not an inline object literal at the call site) so the extra - // `_meta` key survives the daemon SDK's narrow `Omit` types — - // the daemon's own runtime forwards `_meta` unconditionally (`normalizeSessionInit` / - // `buildLoadSessionParams` in the vendored `sandbox-agent` patch), only the published types - // are stricter than the wire protocol they describe. - const claudeSystemPromptMeta: ClaudeSystemPromptMeta | undefined = - environment.agentMountedPath && plan.acpAgent === "claude" - ? claudeMountSystemPromptMeta(AGENT_MOUNT_SYSTEM_PROMPT_SEGMENT) - : undefined; - const sessionInit = { - cwd: plan.cwd, - mcpServers: sessionMcp.servers, - ...(claudeSystemPromptMeta ? { _meta: claudeSystemPromptMeta } : {}), - }; - - // If this harness authored the conversation's most recent turn (staleness-guarded) and we - // still remember its native `agentSessionId`, seed the fresh persist driver with a synthetic - // record and resume-by-id so the patched `resumeSession` reaches `session/load` instead of - // `session/new`. Any failure inside `resumeSession` already degrades to a plain new session - // internally (the patch's own `catch {}` around `loadRemoteSession`), so this call is safe to - // attempt unconditionally whenever we have an eligible id — worst case it is exactly today's - // cold `createSession`. - const continuitySessionKey = request.sessionId?.trim(); - const continuityStore = - deps.sessionContinuityStore ?? sessionContinuityStore; - // Seed the in-memory store from the durable row before consulting it, so a resume after a - // runner restart (in-memory map lost) still sees the prior turn's eligibility. No-op (and - // cheap) when the store already has a live in-process record. - if (continuitySessionKey && runCred) { - await ( - deps.hydrateHarnessSessionFromDurable ?? - hydrateHarnessSessionFromDurable - )(continuitySessionKey, plan.harness, continuityStore, { - authorization: runCred, - log: logger, - }); - } - const priorAgentSessionId = continuitySessionKey - ? eligibleAgentSessionId( - continuitySessionKey, - plan.harness, - continuityStore, - ) - : undefined; - const localSessionId = continuitySessionKey - ? `${continuitySessionKey}:${plan.harness}` - : undefined; - // The index THIS turn will occupy once it completes: recorded post-turn against the SAME - // index read here, so a turn that authors turn N leaves the store agreeing with itself. - environment.continuityTurnIndex = continuitySessionKey - ? nextTurnIndex(continuitySessionKey, continuityStore) - : undefined; - // Daytona only: a local run must not overwrite a conversation's remote pointer (switching - // sandboxes mid-conversation would strand the parked Daytona instance). - if (plan.isDaytona && sessionForMount && runCred) { - const liveSandboxId = environment.sandbox?.sandboxId ?? plan.sandboxId; - const pointerWriteOutcome = await ( - deps.writeSandboxPointer ?? writeSandboxPointer - )( - sessionForMount, - { - sandboxId: liveSandboxId, - turnIndex: environment.continuityTurnIndex ?? 0, - }, - { authorization: runCred, log: logger }, - ); - logger( - `sandbox pointer write ${pointerWriteOutcome} session=${sessionForMount} sandbox=${liveSandboxId}`, - ); - } - let loadedFromContinuity = false; - if (priorAgentSessionId && localSessionId) { - await persist.updateSession({ - id: localSessionId, - agent: plan.acpAgent, - agentSessionId: priorAgentSessionId, - lastConnectionId: "", - createdAt: Date.now(), - sessionInit, - }); - const createSessionStartedAt = Date.now(); - try { - environment.session = - await environment.sandbox.resumeSession(localSessionId); - loadedFromContinuity = - environment.session.agentSessionId === priorAgentSessionId; - logger( - `[continuity] session/load attempted session=${continuitySessionKey} ` + - `harness=${plan.harness} loaded=${loadedFromContinuity}`, - ); - } catch (err) { - logger( - `[continuity] resumeSession failed, falling back to cold createSession: ` + - `${conciseError(err, plan.harness)}`, - ); - } finally { - timingLog("create_session", createSessionStartedAt, " mode=load"); - } - } - environment.loadedFromContinuity = loadedFromContinuity; - if (!environment.session) { - const createSessionStartedAt = Date.now(); - try { - environment.session = await environment.sandbox.createSession({ - ...(localSessionId ? { id: localSessionId } : {}), - agent: plan.acpAgent, - cwd: plan.cwd, - sessionInit, - }); - } finally { - timingLog("create_session", createSessionStartedAt, " mode=create"); - } - } - environment.sessionId = resolveRunSessionId( - request, - environment.session.id, - ); - - // Resolve the model first: when the harness rejects the requested id and keeps its own - // default, `model` is undefined and the chat span is labelled "chat". - // - // For a managed OpenAI-compatible custom run, request the FULLY QUALIFIED - // `/` that pi-acp advertises for this provider, not the bare wire - // model id (design Decision 7). `applyModel`/`pickModel` fall back to suffix matching, which - // returns the FIRST advertised id whose suffix matches — so a built-in `openai/` that - // Pi still advertises (the vault key rides in as `OPENAI_API_KEY`, keeping Pi's built-in - // openai provider live) would be selected ahead of the custom `/` when both share - // the model id. That would silently route to api.openai.com instead of the user's endpoint. - // The qualified id is an EXACT match, so it always wins over any bare-suffix collision. - const wantedModel = - piModelConfig && piModelConfig.models.length > 0 - ? `${piModelConfig.providerId}/${piModelConfig.models[0].id}` - : request.model; - environment.model = await (deps.applyModel ?? applyModel)( - environment.session, - wantedModel, - logger, - { strict: strictModel }, - ); - - // Session-lifetime listeners: attach ONCE, each demuxing into the active turn's sink. They - // outlive any single turn, so the routing lives in dedicated non-throwing helpers below. - environment.session.onEvent((event: any) => - routeSessionEventToActiveTurn( - environment, - remountLocalCwdAfterRuntimeEnotconn, - event, - ), - ); - environment.session.onPermissionRequest((req: any) => - routePermissionRequestToActiveTurn(environment, req), - ); - - timingLog("acquire_total", acquireStartedAt); - return { ok: true, env: environment }; - } catch (err) { - const error = conciseError(err, plan.harness, request.provider); - // Mirror today's shared teardown: no otel exists yet during acquire, so there is no partial - // trace to flush — just run the incrementally-registered finalizers and surface the error. - await environment.destroy({ reason: "failed-turn" }); - return { ok: false, error }; - } -} - -/** - * Route one harness event into the active turn's sink. - * - * Data flow: the ACP session emits an event -> we demux it -> the active turn - * (`environment.currentTurn`) consumes the update. The session listener is attached ONCE and - * outlives every turn, so this must never throw: the sandbox-agent registries are plain Sets and a - * thrown handler would corrupt the event stream, so any error is swallowed and logged. - * - * Steps: let the ENOTCONN watcher observe the raw event, extract the update payload (dropping events - * that carry none), record live tool_call ids for client-tool correlation, then hand the update to - * the active turn — or, between turns when no turn owns it, log and drop it. - */ -function routeSessionEventToActiveTurn( - environment: SessionEnvironment, - remountLocalCwdAfterRuntimeEnotconn: (event: unknown) => void, - event: any, -): void { - const { logger, plan } = environment; - try { - remountLocalCwdAfterRuntimeEnotconn(event); - const payload = event?.payload; - const update = payload?.params?.update ?? payload?.update; - if (!update) return; - // Record live ACP tool_call ids so a paused client_tool can correlate to Claude's bubble - // (session-scoped; a lookup CONSUMES its matched id). - environment.toolCallIndex.record(update); - const turn = environment.currentTurn; - if (turn) { - turn.handleUpdate(update); - } else { - // Between turns (parked/idle): no turn owns this event. Log and drop by decision. - logger(`[keepalive] between-turns event dropped`); - } - } catch (err) { - logger(`session onEvent handler error: ${conciseError(err, plan.harness)}`); - } -} - -/** - * Route one permission gate into the active turn's approval handler. - * - * Data flow: the harness raises a permission request -> the active turn (`environment.currentTurn`) - * decides it. Like the event listener this is attached ONCE and must never throw (a thrown handler - * would corrupt the sandbox-agent registries), so errors are swallowed and logged. - * - * Between turns no turn owns the gate. An approval park is always recorded DURING the active turn - * (the gate fires while a prompt runs, routing through currentTurn), and a parked-on-approval - * session leaves its harness suspended on that gate, so nothing new fires while parked. A gate that - * reaches here is therefore a genuine stray (e.g. a late teardown artifact): reject it by policy so - * it cannot hang. - */ -function routePermissionRequestToActiveTurn( - environment: SessionEnvironment, - req: any, -): void { - const { logger, plan } = environment; - try { - const turn = environment.currentTurn; - if (turn?.onPermissionRequest) { - turn.onPermissionRequest(req); - return; - } - logger( - `[keepalive] between-turns permission request, cancelling by policy id=${req?.id}`, - ); - void Promise.resolve( - environment.session?.respondPermission?.(req?.id, "reject"), - ).catch(() => {}); - } catch (err) { - logger( - `session onPermissionRequest handler error: ${conciseError(err, plan.harness)}`, - ); - } -} - -/** - * Run one turn against an acquired environment: start a fresh otel run, wire this turn's pause - * controller / latch / decisions / responder into `env.currentTurn`, restart the tool relay, - * send the prompt, resolve usage, and finish + flush the trace. It does NOT tear down the - * environment (the caller owns `env.destroy`). On a continuation the prompt is only the new user - * text (`buildTurnText` does not run); on a cold turn it is `plan.turnText`, exactly as before. - */ -export async function runTurn( - env: SessionEnvironment, - request: AgentRunRequest, - emit?: EmitEvent, - signal?: AbortSignal, - opts: RunTurnOptions = {}, -): Promise { - const { plan, logger, deps } = env; - const sessionId = env.sessionId; - // Reset the per-turn tool-call id record (the park folds the completed turn's ids into the - // expected next-history fingerprint). - env.lastTurnToolCallIds = []; - // Reset the per-turn approval-park bookkeeping. A fresh turn starts with no parked gate; this - // turn re-records it only if it pauses on a Claude ACP permission gate. (The dispatch has - // already captured any prior park into `opts.resume` before calling us.) - env.parkedApproval = undefined; - env.approvalGateCount = 0; - // Hoisted so the catch can flush a partial trace (mirroring the pre-split `otel?` handling — - // a createOtel throw must still return `{ ok: false }`, not propagate raw) and the finally can - // stop this turn's relay on EVERY exit path (a cleared sink must never orphan it). - let otel: ReturnType | undefined; - let activeTurn: CurrentTurn | undefined; - - // Time-based run deadlines (total/idle/TTFB/per-tool-call) for THIS turn: an idle/wedged harness - // has no deadline anywhere, so a silent or hung turn would hold its sandbox forever. Tripping a - // limit resolves the prompt race with `RUN_LIMIT_TRIPPED`, which ends the turn as an error so the - // caller's teardown (`runSandboxAgent`'s `finally`, or the keep-alive dispatch's evict-on-failure) - // reclaims the sandbox exactly as any other error does. Disposed in the `finally` on every path. - // A human pause retires the deadlines (`notePaused`): a HITL wait is legitimate, not a wedge. - const runLimits = (deps.createRunLimits ?? createRunLimits)( - (deps.resolveRunLimits ?? resolveRunLimits)(logger), - { log: logger }, - ); - let runLimitTrip: (() => void) | undefined; - let runLimitReason: string | undefined; - const runLimitTripped = new Promise((resolve) => { - runLimitTrip = resolve; - }); - runLimits.onTrip((reason) => { - runLimitReason = reason; - runLimitTrip?.(); - }); - - try { - const promptText = resolvePromptText(request); - // Cold: replay the full transcript (plan.turnText). Continuation or loaded: send only new text. - const turnText = sendLastMessageOnly(opts) ? promptText : plan.turnText; - - const run = (deps.createOtel ?? createSandboxAgentOtel)({ - harness: plan.harness, - model: env.model, - skills: plan.skillDirs.map((s) => s.name), - traceparent: request.context?.propagation?.traceparent, - baggage: request.context?.propagation?.baggage, - endpoint: request.telemetry?.exporters?.otlp?.endpoint, - authorization: request.telemetry?.exporters?.otlp?.headers?.authorization, - captureContent: request.telemetry?.capture?.content?.enabled, - // Seed from the keys actually APPLIED to this run (`plan.secrets`) plus the mount's STS - // pair — neither lives in the sidecar's process env. - redactor: seedForRun( - { secrets: plan.secrets, telemetry: request.telemetry }, - [ - env.mountCreds?.accessKey, - env.mountCreds?.secretKey, - env.mountCreds?.sessionToken, - ], - ), - emitSpans: !plan.isPi || plan.isDaytona, - // Every emitted event is a progress signal for the idle/TTFB deadlines (message/thought - // deltas, tool calls and results, usage, ...) — the one seam every harness's output flows - // through. Per-tool-call timers are driven separately from `handleUpdate` below. - emit: emit && runLimits.wrapEmit(emit), - }); - otel = run; - - run.start({ - prompt: promptText, - sessionId, - messages: [ - ...priorMessages(request), - { role: "user", content: promptText }, - ], - }); - - const pause = new PendingApprovalPauseController(() => { - // The sibling settle runs UNCONDITIONALLY, park mode or not: latch-loser tool calls - // announced before the winning gate can never execute this turn, and skipping the settle - // here would leave them as orphaned open parts whenever the dispatch later refuses the park - // (multi-gate, pool full) — `env.destroy()` does not re-run it. The exclusion keeps the - // gated (paused) call itself open, so the live resume is untouched. - run.settleOpenToolCalls( - (id) => pause.isPausedToolCall(id), - TOOL_NOT_EXECUTED_PAUSED, - ); - // Park mode: a parkable permission gate (Claude ACP or Pi ACP) recorded - // `env.parkedApproval` BEFORE firing this pause (the onUserApprovalGate hook runs before - // the single-pause latch). Keep the live session — the gated tool runs on the resume — so - // skip ONLY the mcpAbort and the destroySession. The teardown is not lost: the dispatch - // either parks the session or, if it decides not to (multi-gate, pool full), calls - // `env.destroy()` which runs them. A non-parkable pause (keep-alive off, client tool) - // never records `parkedApproval`, so it still tears down here exactly as today. - if (opts.approvalParkMode && env.parkedApproval) return; - // Abort any in-flight loopback `tools/call` (a paused Claude client tool) BEFORE the - // session teardown, so its handler cannot write a result after the turn ends. - env.mcpAbort.abort(); - env.sessionDestroyRequested = true; - return env.sandbox.destroySession?.(env.session.id); - }); - // A human pause resolves this signal exactly once, the moment the turn parks for input — the one - // place every pause path converges, so the one place to retire the run-limits deadlines for good. - void pause.signal.then(() => runLimits.notePaused()); - - // Publish this turn's sink so the session-lifetime listeners route into it. handleUpdate - // reproduces the old per-event routing (suppress paused frames, handleUpdate, pause re-sweep). - const turn: CurrentTurn = { - run, - pause, - toolRelay: undefined, - handleUpdate: (update) => { - // Per-tool-call deadline: starts on the announcement, ends on a terminal status. Tracked - // regardless of the pause-suppression below (a call already timed out must not linger just - // because a later sibling frame gets suppressed). - const rawFrame = update as { - sessionUpdate?: unknown; - toolCallId?: unknown; - status?: unknown; - }; - if (rawFrame?.sessionUpdate === "tool_call" && rawFrame.toolCallId) { - runLimits.noteToolCallStart(String(rawFrame.toolCallId)); - } else if ( - rawFrame?.sessionUpdate === "tool_call_update" && - rawFrame.toolCallId && - (rawFrame.status === "completed" || rawFrame.status === "failed") - ) { - runLimits.noteToolCallEnd(String(rawFrame.toolCallId)); - } - if (!shouldSuppressPausedToolCallUpdate(update, pause)) { - // Record the emitted tool-call ids (unique, first-seen order): the park folds them - // into the expected next-history fingerprint so a tool-using turn continues live. - const frame = update as { - sessionUpdate?: unknown; - toolCallId?: unknown; - }; - if ( - frame?.sessionUpdate === "tool_call" && - typeof frame.toolCallId === "string" && - frame.toolCallId && - !env.lastTurnToolCallIds.includes(frame.toolCallId) - ) { - env.lastTurnToolCallIds.push(frame.toolCallId); - } - run.handleUpdate(update); - // A sibling announced AFTER the pause won the latch can never execute; settle it - // immediately so the client never holds an orphaned part (idempotent re-sweep). - if (pause.active) { - run.settleOpenToolCalls( - (id) => pause.isPausedToolCall(id), - TOOL_NOT_EXECUTED_PAUSED, - ); - } - } - }, - onPermissionRequest: undefined, - }; - activeTurn = turn; - env.currentTurn = turn; - - const permissionPlan = permissionsFromRequest(request); - const storedDecisionMap = extractApprovalDecisions(request); - if (storedDecisionMap.size > 0) { - logger( - `[HITL] resume state: decisions=${JSON.stringify([...storedDecisionMap.keys()])}`, - ); - } - const decisions = new ConversationDecisions( - storedDecisionMap, - extractClientToolOutputs(request), - ); - const executionGrants = new ApprovedExecutionGrants(); - const latch = new PendingApprovalLatch(); - const responder = - deps.responderFactory?.(request) ?? - new ApprovalResponder(permissionPlan, decisions, logger); - // Every pause seeds the durable interactions plane, whichever gate paused. - const recordPendingInteraction = ( - token: string, - toolName: string | undefined, - toolArgs: unknown, - kind: "user_approval" | "client_tool" = "user_approval", - ): void => { - const cred = runCredential(request); - if (!cred) return; - const references = buildWorkflowReferences(request.runContext?.workflow); - if (!references?.workflow_revision) return; - void createInteraction( - sessionId, - request.turnId ?? "", - token, - kind, - { request: { tool: toolName ?? token, args: toolArgs }, references }, - () => cred, - ); - }; - // Transition the durable interaction row to resolved once its gate is answered. Used both by - // the cold decision-map path (via attachPermissionResponder) and the live approval resume, - // which answers the parked gate directly. The turn-start `cancelStaleInteractions` sweep - // (server.ts) cancels only PENDING gates of OTHER turns and spares this gate two ways: an - // interactions-plane answer already transitioned it to responded, and an in-band answer is - // detected at sweep time (`inBandAnswerToken`) and exempted via the sweep's `tokens` — the - // row stays pending until this resolve lands it as resolved, never cancelled. - const resolveInteractionToken = (token: string): void => { - const cred = runCredential(request); - if (!cred) return; - if ( - !buildWorkflowReferences(request.runContext?.workflow) - ?.workflow_revision - ) - return; - void resolveInteraction(sessionId, token, () => cred); - }; - const serverPermissions = serverPermissionsFromRequest(request); - // The SAME name->spec index the relay execute loop hands to the relay execution guard, so - // the approval card and the guard cannot disagree about a tool's permission/readOnly. - const specsByName = toolSpecsByName(plan.toolSpecs); - // Build the per-turn permission handler WITHOUT attaching to the live session: the - // session-lifetime `onPermissionRequest` (in acquireEnvironment) routes into it via - // `currentTurn`. A capturing shim reuses attachPermissionResponder unchanged; its - // respondPermission delegates to the real session. - attachPermissionResponder({ - session: { - onPermissionRequest: (handler: (req: unknown) => void) => { - turn.onPermissionRequest = handler; - }, - respondPermission: (id: string, reply: string) => - env.session.respondPermission(id, reply), - }, - run, - responder, - latch, - serverPermissions, - log: logger, - onPause: () => pause.pause(), - onPausedToolCall: (id) => pause.markPausedToolCall(id), - onCreateInteraction: recordPendingInteraction, - onResolveInteraction: resolveInteractionToken, - toolSpecsByName: specsByName, - // Pi runs only: presence of the specs map turns Pi gate envelope detection on AND is how - // the runner recovers specPermission/readOnlyHint (the envelope carries identity, never - // policy). Absent for Claude, so a title collision there keeps the base path. - piToolSpecsByName: plan.isPi - ? new Map( - plan.toolSpecs.map((spec) => [ - spec.name, - { - permission: spec.permission, - readOnly: spec.readOnly, - // callRef tools only: bound paths are runner-filled at execution, so the - // approval card and decision keys must not carry the model's values for them. - contextBindings: spec.callRef - ? spec.contextBindings - : undefined, - }, - ]), - ) - : undefined, - // A resolved custom-tool allow becomes an execution grant the relay guard consumes, so - // only a dialog-approved (or policy-allowed) call ever executes from the relay dir. - onPiGateAllowed: (info) => - executionGrants.grant(info.toolName, info.args), - // Record the parkable permission gate (only in keep-alive park mode) so the dispatch can - // resume it live. Fires per pending gate (before the latch) so a parallel gate is counted; - // the single-gate resume records only the FIRST gate's answer target. `info.gateType` names - // the plane (Claude ACP vs Pi ACP) so the resume answers on the right one. - onUserApprovalGate: opts.approvalParkMode - ? (info) => { - env.approvalGateCount += 1; - if ( - env.approvalGateCount === 1 && - info.permissionId && - info.toolCallId - ) { - env.parkedApproval = { - gateType: info.gateType, - permissionId: info.permissionId, - toolCallId: info.toolCallId, - toolName: info.toolName, - args: info.args, - interactionToken: info.interactionToken, - }; - } - } - : undefined, - }); - - // Resolve the ONE client-tool seam both delivery paths share. The correlation index is wired - // for Claude only — Pi's relay toolCallId is already exact. - env.clientToolRelayRef.current = buildClientToolRelay({ - responder, - run, - latch, - pause, - recordPendingInteraction, - toolCallIndex: plan.isPi ? undefined : env.toolCallIndex, - log: logger, - }); - - // EVERY harness gets the guard: the relay dir is sandbox-writable, so a forged - // `.req.json` proves nothing about any dialog having run, and this runner-side - // re-check is the only enforcement of the hard deny boundary against forged files. - // `allow` passes and `deny` refuses identically everywhere; `ask` splits by harness — - // Pi consumes a dialog-recorded execution grant (fail-closed parity with the in-sandbox - // confirm), while a non-Pi MCP harness (Claude) passes `ask` because its own harness - // enforces the ask dialog (the rendered `mcp__agenta-tools__` ask rules + the ACP - // permission flow) before a call reaches the shim. See buildRelayExecutionGuard for the - // stated residual (a forged file can still trigger an ask-tool without a dialog there). - const relayGuard: RelayExecutionGuard = buildRelayExecutionGuard({ - isPi: plan.isPi, - permissionPlan, - executionGrants, - }); - - if (plan.useToolRelay) { - turn.toolRelay = (deps.startToolRelay ?? startToolRelay)( - plan.isDaytona - ? (deps.sandboxRelayHost ?? sandboxRelayHost)(env.sandbox, { - log: logger, - }) - : (deps.localRelayHost ?? localRelayHost)(), - plan.relayDir, - plan.toolSpecs, - request.toolCallback as ToolCallbackContext | undefined, - request.runContext, - env.clientToolRelayRef.current, - relayGuard, - { log: logger }, - ); - // Ordering invariant: the relay's stale-file sweep must complete before the - // resume's respondPermission or the fresh prompt below can cause a legitimate - // request, so nothing legitimate can predate the sweep and be swallowed as - // stale. Optional-chained so a fake relay without `ready` is tolerated, and a - // sweep failure never kills the turn. - await turn.toolRelay?.ready?.catch?.(() => {}); - } - - // The prompt promise this turn races against the pause signal. A normal/continuation turn - // sends a fresh prompt; a live approval resume answers the parked gate on the SAME session and - // continues the ORIGINAL, still-pending prompt promise (the tool then runs with its original - // byte-exact args). Either way, on a HITL pause the prompt resolves cancelled or never - // resolves, and the pause signal ends the turn. - let promptPromise: Promise; - if (opts.resume) { - // The new (resume) turn owns streaming + tracing; the environment is already wired to route - // continued events into this turn's sink (env.currentTurn was set above). Seed this run's - // trace with the parked tool call so the completing `tool_call_update` closes it and the FE - // approval part flips to output-available even if the adapter re-announces nothing. Then - // answer the gate on the live session — the original prompt continues from here. - run.handleUpdate({ - sessionUpdate: "tool_call", - toolCallId: opts.resume.toolCallId, - title: opts.resume.toolName, - kind: opts.resume.toolName, - rawInput: opts.resume.args, - }); - promptPromise = Promise.resolve(opts.resume.promptPromise); - promptPromise.catch(() => {}); - // A parked Pi dialog gate resumes on a FRESH turn whose relay and grant ledger are new; - // grant the approved call here so the extension's execute record (written right after the - // confirm resolves) passes the relay guard. Claude resumes grant too — harmlessly, no - // guard consults it. - if (opts.resume.reply === "once") { - executionGrants.grant(opts.resume.toolName, opts.resume.args); - } - await env.session.respondPermission( - opts.resume.permissionId, - opts.resume.reply, - ); - // The gate is answered: resolve the durable interaction row (the parked pending row the cold - // path would otherwise resolve via its decision map). The fresh per-turn pause controller - // starts with an EMPTY pausedToolCallIds set, so the resumed call's `tool_call_update` frames - // are no longer suppressed and stream through — the "clear pausedToolCallIds on resume" step. - resolveInteractionToken(opts.resume.interactionToken); - logger( - `[keepalive] resume answered gate reply=${opts.resume.reply} tool=${opts.resume.toolName ?? "?"}`, - ); - } else { - promptPromise = Promise.resolve( - env.session.prompt([{ type: "text", text: turnText }]), - ); - promptPromise.catch(() => {}); - } - const raced = await Promise.race([ - promptPromise, - pause.signal.then(() => PAUSED), - runLimitTripped.then(() => RUN_LIMIT_TRIPPED), - ]); - // A tripped run-limit ends the turn as an error: throw into the shared catch below so the - // trace is flushed and the caller's teardown reclaims the (wedged) sandbox. - if (raced === RUN_LIMIT_TRIPPED) { - throw new Error(runLimitReason ?? "run limit tripped"); - } - const stopReason = - raced === PAUSED || pause.active ? "paused" : (raced as any)?.stopReason; - // Pause notification is immediate, but terminalization must wait for managed cancellation - // and already-queued ACP updates. Re-sweep after the drain so a sibling announced during - // cancellation receives exactly one deterministic terminal result before `done`. - if (stopReason === "paused") { - await pause.waitForEventDrain(); - run.settleOpenToolCalls( - (id) => pause.isPausedToolCall(id), - TOOL_NOT_EXECUTED_PAUSED, - ); - } - const result = raced === PAUSED ? undefined : raced; - // A parkable pause this turn: hand the still-pending prompt promise to the parked record so a - // later resume can await the same continuation. (Set after the race so `promptPromise` exists. - // The read is asserted because the onUserApprovalGate callback set the field via an async - // mutation TS's flow analysis cannot see, so it would otherwise narrow the reset to `never`.) - const parkedThisTurn = env.parkedApproval as ParkedApproval | undefined; - if (opts.approvalParkMode && pause.active && parkedThisTurn) { - parkedThisTurn.promptPromise = promptPromise; - } - await turn.toolRelay?.stop(); - logger(`prompt stopReason=${stopReason}`); - - const usage = await resolveRunUsage({ - sandbox: env.sandbox, - usageOutPath: plan.usageOutPath, - isDaytona: plan.isDaytona, - promptResult: result, - streamUsage: run.usage(), - }); - run.setUsage(usage); - - const swallowedPiError = - plan.isPi && - !plan.isDaytona && - !run.output().trim() && - !run.events().some((e) => e.type === "tool_call") - ? // The helper derives the transcript location from `piSessionWorkspaceDir(plan.cwd)`, - // the same shared helper `configurePiSessionWorkspace` used to point Pi at it. - findSwallowedPiError(plan.cwd) - : undefined; - let swallowedError: string | undefined; - if (swallowedPiError) { - swallowedError = conciseError( - new Error(swallowedPiError), - plan.harness, - request.provider, - ); - run.recordError(swallowedError, request.provider); - run.emitEvent({ type: "error", message: swallowedError }); - } - - const output = run.finish(); - await run.flush(); - - if (swallowedError) { - // A failed turn may have left a partial turn in the native transcript: the prior record - // is no longer a faithful resume point. - invalidateContinuity(sessionId, plan.harness, deps); - return { ok: false, error: swallowedError }; - } - - // Capture this harness's native session id for the next turn's setup. Only on a turn that - // actually completed (not paused mid-turn — a park has not finished authoring the turn, so - // it must not be marked authoritative) and only when the harness surfaced one. - if ( - stopReason !== "paused" && - env.continuityTurnIndex !== undefined && - sessionId && - env.session?.agentSessionId - ) { - (deps.sessionContinuityStore ?? sessionContinuityStore).record( - sessionId, - plan.harness, - env.session.agentSessionId, - env.continuityTurnIndex, - ); - // Mirror the record durably so it survives a runner restart; fire-and-forget. - const syncCred = runCredential(request); - if (syncCred) { - void (deps.syncHarnessSessionDurable ?? syncHarnessSessionDurable)( - sessionId, - plan.harness, - env.session.agentSessionId, - env.continuityTurnIndex, - { authorization: syncCred, log: logger }, - ); - } - } else if (stopReason === "paused") { - // A pause stopped mid-turn, after the harness may have written a partial turn natively. - invalidateContinuity(sessionId, plan.harness, deps); - } - - return { - ok: true, - output, - messages: output ? [{ role: "assistant", content: output }] : [], - events: emit ? [] : run.events(), - usage, - stopReason, - capabilities: { - ...env.capabilities, - streamingDeltas: !!emit && env.capabilities.streamingDeltas, - }, - sessionId, - model: env.model ?? request.model, - traceId: run.traceId(), - } as AgentRunResult; - } catch (err) { - const error = conciseError(err, plan.harness, request.provider); - otel?.recordError(error, request.provider); - otel?.emitEvent({ type: "error", message: error }); - // An aborted turn may have left a partial turn in the native transcript. - invalidateContinuity(sessionId, plan.harness, deps); - // finish() must not throw uncaught — tracing must not mask the run error. - try { - otel?.finish(); - } catch {} - await otel?.flush().catch(() => {}); - return { ok: false, error }; - } finally { - // Release every run-limits timer (idempotent, never re-arms on a late event) on EVERY path. - runLimits.dispose(); - // This turn owns its relay: stop it on EVERY exit path (the happy path already stopped it - // after the prompt; stop is safe to repeat, matching the old finally). Null it afterwards so - // a later `destroy()` — possibly after the dispatch cleared the sink — cannot double-stop or - // orphan it. - await activeTurn?.toolRelay?.stop().catch(() => {}); - if (activeTurn) activeTurn.toolRelay = undefined; - } -} - -/** - * The cold, one-turn-per-environment entry (also the flag-off path). Acquire an environment, run - * one turn, then tear the environment down — exactly as the single `try/finally` did before the - * split, so behavior here is byte-identical to pre-keep-alive. - */ -/** - * Drop the harness's continuity record after a turn that did not complete. The harness may have - * written a partial turn into its native transcript, so a later `session/load` would resume a - * history the canonical request never sent. Dropping it falls back to cold replay. - */ -function invalidateContinuity( - sessionId: string | undefined, - harness: string, - deps: SandboxAgentDeps, -): void { - if (!sessionId) return; - (deps.sessionContinuityStore ?? sessionContinuityStore).invalidate( - sessionId, - harness, - ); -} - -/** - * Whether a completed turn's environment may be parked: never on abort, client disconnect, - * pause, or failure. Session-owned streams survive disconnect WITHOUT aborting the run signal - * (server policy), so the disconnect check needs the separate `clientGone` flag. A wedged - * sandbox that failed its turn must be destroyed, not reconnected on the next one. - */ -export function shouldPark( - result: AgentRunResult, - signal: AbortSignal | undefined, - clientGone: (() => boolean) | undefined, -): boolean { - if (signal?.aborted) return false; // aborted run: destroy, do not park - if (clientGone?.()) return false; // client disconnected mid-turn: destroy, do not park - if (!result.ok) return false; // failed turn: teardown as today - if (result.stopReason === "paused") return false; // a plain pause never parks - return true; -} - -export async function runSandboxAgent( - request: AgentRunRequest, - emit?: EmitEvent, - signal?: AbortSignal, - deps: SandboxAgentDeps = {}, -): Promise { - const acquired = await acquireEnvironment(request, deps, signal); - if (!acquired.ok) return { ok: false, error: acquired.error }; - const env = acquired.env; - let result: AgentRunResult | undefined; - try { - result = await runTurn(env, request, emit, signal, { - loaded: env.loadedFromContinuity, - }); - return result; - } finally { - // `result` is undefined when runTurn threw: a failed turn, so destroy. - const cleanResumable = - env.resumable && - result !== undefined && - shouldPark(result, signal, undefined); - await env.destroy({ - reason: cleanResumable - ? "clean-resumable" - : signal?.aborted - ? "aborted" - : "failed-turn", - }); - } -} diff --git a/services/runner/src/engines/sandbox_agent/engine.ts b/services/runner/src/engines/sandbox_agent/engine.ts new file mode 100644 index 0000000000..401ed45f5b --- /dev/null +++ b/services/runner/src/engines/sandbox_agent/engine.ts @@ -0,0 +1,62 @@ +import { + type AgentRunRequest, + type AgentRunResult, + type EmitEvent, +} from "../../protocol.ts"; +import { acquireEnvironment } from "./environment.ts"; +import { runTurn } from "./run-turn.ts"; +import { type SandboxAgentDeps } from "./runtime-contracts.ts"; + +/** + * Whether a completed turn's environment may be parked: never on abort, client disconnect, + * pause, or failure. Session-owned streams survive disconnect WITHOUT aborting the run signal + * (server policy), so the disconnect check needs the separate `clientGone` flag. A wedged + * sandbox that failed its turn must be destroyed, not reconnected on the next one. + */ +export function shouldPark( + result: AgentRunResult, + signal: AbortSignal | undefined, + clientGone: (() => boolean) | undefined, +): boolean { + if (signal?.aborted) return false; // aborted run: destroy, do not park + if (clientGone?.()) return false; // client disconnected mid-turn: destroy, do not park + if (!result.ok) return false; // failed turn: teardown as today + if (result.stopReason === "paused") return false; // a plain pause never parks + return true; +} + +/** + * The cold, one-turn-per-environment entry (also the flag-off path). Acquire an environment, run + * one turn, then tear the environment down — exactly as the single `try/finally` did before the + * split, so behavior here is byte-identical to pre-keep-alive. + */ +export async function runSandboxAgent( + request: AgentRunRequest, + emit?: EmitEvent, + signal?: AbortSignal, + deps: SandboxAgentDeps = {}, +): Promise { + const acquired = await acquireEnvironment(request, deps, signal); + if (!acquired.ok) return { ok: false, error: acquired.error }; + const env = acquired.env; + let result: AgentRunResult | undefined; + try { + result = await runTurn(env, request, emit, signal, { + loaded: env.loadedFromContinuity, + }); + return result; + } finally { + // `result` is undefined when runTurn threw: a failed turn, so destroy. + const cleanResumable = + env.resumable && + result !== undefined && + shouldPark(result, signal, undefined); + await env.destroy({ + reason: cleanResumable + ? "clean-resumable" + : signal?.aborted + ? "aborted" + : "failed-turn", + }); + } +} diff --git a/services/runner/src/engines/sandbox_agent/environment-setup.ts b/services/runner/src/engines/sandbox_agent/environment-setup.ts new file mode 100644 index 0000000000..d8ca015109 --- /dev/null +++ b/services/runner/src/engines/sandbox_agent/environment-setup.ts @@ -0,0 +1,386 @@ +import { rmSync } from "node:fs"; + +import { apiBase } from "../../apiBase.ts"; + +import { + resolveRunSessionId, + type AgentRunRequest, +} from "../../protocol.ts"; +import { type ClientToolOutcome } from "../../responder.ts"; +import type { ClientToolRelay } from "../../tools/client-tool-relay.ts"; +import { + agentMountPath, + signAgentMountCredentials, +} from "./agent-mount.ts"; +import { createToolCallCorrelationIndex } from "./client-tools.ts"; +import { buildDaemonEnv, resolveDaemonBinary } from "./daemon.ts"; +import { conciseError } from "./errors.ts"; +import { + signSessionMountCredentials, + type MountCredentials, +} from "./mount.ts"; +import { + buildPiExtensionEnv, + configurePiSessionWorkspace, + configurePiSkillSnapshot, + prepareLocalPiAssets, + resolvePiSkillSnapshot, + writeOtlpAuthFile, +} from "./pi-assets.ts"; +import { + buildPiModelConfigPlan, + type PiModelConfigPlan, +} from "./pi-model-config.ts"; +import { buildRunPlan } from "./run-plan.ts"; +import type { + SandboxAgentDeps, + SessionEnvironment, +} from "./runtime-contracts.ts"; +import { + applyClaudeConnectionEnv, + defaultResolveLocalRunnerOwner, + modelResolutionStrict, + runCredential, +} from "./runtime-policy.ts"; +import { assertLocalRunnerOwnership } from "./session-continuity.ts"; +import { + projectScopeFor, + resolvesToLocalProvider, +} from "./session-identity.ts"; +import { loadRunnerConfig } from "../../config/runner-config.ts"; + +function defaultLog(message: string): void { + process.stderr.write(`[sandbox-agent] ${message}\n`); +} + +export async function prepareEnvironmentSetup( + request: AgentRunRequest, + deps: SandboxAgentDeps = {}, + presignedMount?: MountCredentials | null, +) { + const logger = deps.log ?? defaultLog; + const acquireStartedAt = Date.now(); + const timingLog = (stage: string, startedAt: number, fields = ""): void => { + const sandboxId = environment?.sandbox?.sandboxId ?? "-"; + const sessionId = + environment?.sessionId ?? request.sessionId?.trim() ?? "-"; + logger( + `[timing] stage=${stage} ms=${Math.round(Date.now() - startedAt)} sandbox=${sandboxId} session=${sessionId}${fields}`, + ); + }; + + // Local multi-runner fails loudly. Session-owned + local-sandbox only (a non-session run + // has no cross-replica identity to protect, and a remote sandbox has no runner-local pooled + // state to protect it FROM). The resolver claims the `owner` affinity key and reads the actual + // owner back; a KNOWN different owner throws (never a silent wrong-host cold start). + const continuitySessionForOwnership = request.sessionId?.trim(); + if ( + continuitySessionForOwnership && + resolvesToLocalProvider(request.sandbox) + ) { + const { replicaId, ownerReplicaId } = await ( + deps.resolveLocalRunnerOwner ?? defaultResolveLocalRunnerOwner + )(continuitySessionForOwnership, runCredential(request)); + try { + assertLocalRunnerOwnership( + continuitySessionForOwnership, + replicaId, + ownerReplicaId, + ); + } catch (err) { + return { + ok: false as const, + error: conciseError(err, request.harness ?? ""), + }; + } + } + + // Sign BEFORE buildRunPlan so the prefix is available for the durable cwd derivation. + // Inputs (sessionId, apiBase, credential) are independent of the plan. Best-effort: null on + // failure leaves durableCwd undefined and buildRunPlan falls back to the ephemeral path. + const sessionForMount = request.sessionId?.trim(); + const runCred = runCredential(request); + const signMount = + deps.signSessionMountCredentials ?? signSessionMountCredentials; + let mountCreds: MountCredentials | null = + presignedMount !== undefined + ? presignedMount + : sessionForMount && runCred + ? await signMount(sessionForMount, { + apiBase: apiBase(), + authorization: runCred, + log: logger, + }) + : null; + // A session-owned run expects a durable session cwd mount. When signing returns nothing the run + // still proceeds on an ephemeral cwd (behavior unchanged, RSH-11); emit one structured warning + // keyed by mount kind so durable-to-ephemeral degradation is measurable, not silent. + if (sessionForMount && !mountCreds) { + logger( + `mount degraded kind=session_cwd cause=sign_returned_no_mount session=${sessionForMount}`, + ); + } + + const artifactId = request.runContext?.workflow?.artifact?.id?.trim(); + const signAgentMount = + deps.signAgentMountCredentials ?? signAgentMountCredentials; + const agentMountCreds: MountCredentials | null = + artifactId && runCred + ? await signAgentMount(artifactId, { + apiBase: apiBase(), + authorization: runCred, + log: logger, + }) + : null; + // A workflow-artifact run expects an agent mount; same structured degrade signal when unsigned. + if (artifactId && !agentMountCreds) { + logger( + `mount degraded kind=agent_mount cause=sign_returned_no_mount artifact=${artifactId}`, + ); + } + // Derive the durable cwd from the sign prefix (one source of truth, both providers). + // local: /tmp/agenta/ — daytona: /home/sandbox/agenta/ + // is already "mounts//", so no extra slug is needed. + let durableCwd: string | undefined; + if (mountCreds?.prefix) { + const isDaytonaReq = + (request.sandbox ?? loadRunnerConfig().providers.default) === "daytona"; + durableCwd = isDaytonaReq + ? `/home/sandbox/agenta/${mountCreds.prefix}` + : `/tmp/agenta/${mountCreds.prefix}`; + } + + const planResult = buildRunPlan(request, { + sandboxProvider: deps.sandboxProvider, + createLocalCwd: deps.createLocalCwd, + createDaytonaCwd: deps.createDaytonaCwd, + durableCwd, + resolveSkillDirs: deps.resolveSkillDirs, + log: logger, + }); + if (!planResult.ok) return { ok: false as const, error: planResult.error }; + const plan = planResult.plan; + const piSkillSnapshot = resolvePiSkillSnapshot(plan); + const agentMountDir = agentMountCreds ? agentMountPath(plan.cwd) : undefined; + + // Clear-then-apply (Security rule 5): on a managed run (credentialMode "env") the daemon + // inherits NONE of the sidecar's own provider keys, so only the resolved `plan.secrets` are + // present and an inherited key for another provider cannot leak. For runtime_provided/none/ + // un-migrated runs the harness uses its own login, so the inherited keys stay. + const clearProviderEnv = plan.credentialMode === "env"; + const env = (deps.buildDaemonEnv ?? buildDaemonEnv)(plan.acpAgent, { + clearProviderEnv, + provider: request.provider, + deployment: request.deployment, + }); + Object.assign(env, plan.secrets); // apply only the resolved provider keys + applyClaudeConnectionEnv(env, request, plan.acpAgent, logger); + const piSessionDir = configurePiSessionWorkspace(plan, env); + configurePiSkillSnapshot(piSkillSnapshot, env); + const strictModel = modelResolutionStrict(); + // Pi self-instruments locally: propagate the trace context + public tool metadata into Pi + // via the Agenta extension. Tool execution always relays back to this runner, which keeps + // private specs, scoped env, callback endpoints, and callback auth in memory. + // local Pi's OTLP bearer rides a runner-written 0600 file, never a plain env var — + // Daytona never receives telemetry env here at all (`!plan.isDaytona` gates it off above). + const otlpAuthFilePath = + plan.isPi && !plan.isDaytona ? `${plan.relayDir}.otlp-auth` : undefined; + const otlpAuthorization = + request.telemetry?.exporters?.otlp?.headers?.authorization; + if (otlpAuthFilePath && otlpAuthorization) { + writeOtlpAuthFile(otlpAuthFilePath, otlpAuthorization, logger); + } + const piExtEnv = plan.isPi + ? buildPiExtensionEnv(request, !plan.isDaytona, { + relayDir: plan.relayDir, + usageOutPath: plan.usageOutPath, + otlpAuthFilePath, + builtinGatingActive: plan.builtinGatingActive, + builtinGrants: plan.builtinGrants, + // The materialized skill names (author + forced `_agenta.*`) so Pi's own agent span + // records which skills loaded; local Pi self-instruments, so the runner's sandbox-agent + // otel has no span to stamp here. + skills: plan.skillDirs.map((s) => s.name), + }) + : {}; + // Daytona's provider is built from `piExtEnv` rather than the local daemon env. Keep the + // transcript location in both environment slices so Pi and pi-acp see the same durable path + // regardless of provider. + if (piSessionDir) piExtEnv.PI_CODING_AGENT_SESSION_DIR = piSessionDir; + configurePiSkillSnapshot(piSkillSnapshot, piExtEnv); + Object.assign(env, piExtEnv); // local daemon inherits it; daytona gets it via envVars + logger( + `tools=${plan.toolSpecs.length} executableTools=${plan.executableToolSpecs.length} ` + + `piPublicTools=${piExtEnv.AGENTA_AGENT_TOOLS_PUBLIC_SPECS ? "yes" : "no"}`, + ); + if (!plan.isPi && plan.isDaytona) { + const omittedClientTools = plan.toolSpecs + .filter((spec) => spec.kind === "client") + .map((spec) => spec.name); + if (omittedClientTools.length > 0) { + logger( + `omitting client tools from Daytona stdio MCP shim: ${omittedClientTools.join(", ")}`, + ); + } + } + // Translate a managed OpenAI-compatible custom connection into Pi's native models.json plan + // (design Decision 5). Non-applicable requests yield no plan (current behavior); an applicable + // but incomplete request throws — captured here and re-thrown inside the try below so the + // engine's own catch turns it into `{ ok: false, error }` and a visible error frame (fail loud, + // never a silent fall-back to a default provider). Only the env var NAME enters the plan. + let piModelConfig: PiModelConfigPlan | undefined; + let piModelConfigError: Error | undefined; + if (plan.isPi) { + try { + piModelConfig = buildPiModelConfigPlan(request, plan.secrets); + } catch (err) { + piModelConfigError = err as Error; + } + } + if (piModelConfig) { + logger( + `pi model-config plan provider=${piModelConfig.providerId} api=${piModelConfig.api} ` + + `model=${piModelConfig.models.map((m) => m.id).join(",")}`, + ); + } + + // undefined is fine: the local provider runs its own resolution and errors clearly. + const binaryPath = (deps.resolveDaemonBinary ?? resolveDaemonBinary)(); + const localPiAssets = prepareLocalPiAssets({ + plan, + env, + piModelConfig, + log: logger, + }); + let runAgentDir = localPiAssets.dir; + // Fail closed (Decision 6): a local managed custom run whose models.json could not be written + // must stop rather than run on a default provider. Recorded here (the write ran above) and + // thrown inside the try below, like the permission-extension gate. + const localModelConfigUnwritable = + plan.isPi && + !plan.isDaytona && + !!piModelConfig && + !localPiAssets.modelConfigWritten; + // Fail closed (Decision 2): when the policy could gate a Pi built-in tool but the permission + // extension did not install, the run must stop rather than run those tools unprotected. Recorded + // here (the install ran above) and thrown inside the try below so the engine's own catch turns it + // into `{ ok: false, error }` and a visible error frame. `builtinGatingActive` false means + // allow-everything, where the extension is not needed and a failed install is harmless. + const localBuiltinGatingUnenforceable = + plan.isPi && + !plan.isDaytona && + plan.builtinGatingActive && + !localPiAssets.extensionInstalled; + + // A local Claude subscription run reads and writes the operator's read-write mounted login + // DIRECTLY: `buildDaemonEnv` already carried `CLAUDE_CONFIG_DIR` (the mount) into the daemon env, + // and there is deliberately no per-run copy. Claude refreshes its OAuth token mid-run and writes + // it back to its config dir; copying that dir per run would discard the refresh, so the next run + // would fail as soon as the provider rotated the refresh token. The harness owns its own token + // lifecycle, exactly like a normal local install (interface.md section 6). buildRunPlan already + // rejected a runtime_provided Claude run with no configured CLAUDE_CONFIG_DIR. + + logger(`harness=${plan.harness} sandbox=${plan.sandboxId} cwd=${plan.cwd}`); + + // The resolved model ref as it reaches the runner (key NAMES only, never values) — the one + // line that answers "what model/provider/deployment/credential did this run actually use". + logger( + `resolved model=${request.model ?? ""} provider=${request.provider ?? ""} ` + + `deployment=${request.deployment ?? ""} ` + + `connection=${request.connection ? `${request.connection.mode}:${request.connection.slug ?? "-"}` : ""} ` + + `secretKeys=[${Object.keys(request.secrets ?? {}).join(",")}]`, + ); + + // The shared client-tool relay reference (the deferred ref baked into the MCP server reads it; + // each turn's `runTurn` sets `.current`). A `tools/call` can only arrive during a prompt — + // long after the relay is wired — so the server captures this reference and it resolves to the + // real relay before any call lands. + const clientToolRelayRef: { current?: ClientToolRelay } = {}; + const deferredClientToolRelay: ClientToolRelay = { + onClientTool: (req) => + clientToolRelayRef.current + ? clientToolRelayRef.current.onClientTool(req) + : Promise.resolve("deny" as ClientToolOutcome), + onPause: (req) => clientToolRelayRef.current?.onPause?.(req), + }; + + // Aborts any in-flight loopback `tools/call` (a paused Claude client tool) on pause/teardown, + // so its handler is torn down deterministically and cannot write a result after the turn ends. + const mcpAbort = new AbortController(); + + const environment: SessionEnvironment = { + plan, + logger, + deps, + sandbox: undefined, + session: undefined, + sessionId: resolveRunSessionId(request, ""), + model: undefined, + capabilities: {}, + strictModel, + toolCallIndex: createToolCallCorrelationIndex(), + clientToolRelayRef, + mcpAbort, + runAgentDir, + otlpAuthFilePath, + mountCreds, + agentMountCreds, + mountProjectId: mountCreds?.projectId, + projectScopeId: projectScopeFor(request, mountCreds?.projectId)?.id, + loadedFromContinuity: false, + resumable: false, + continuityTurnIndex: undefined, + sessionDestroyRequested: false, + mountedCwd: undefined, + agentMountedPath: undefined, + durableCwdSafeToDelete: true, + // Local runs get a plain rmSync cleanup for the throwaway cwd; Daytona has none on this host. + workspace: plan.isDaytona + ? undefined + : { + cleanup: async () => + rmSync(plan.cwd, { recursive: true, force: true }), + }, + runtimeRemount: undefined, + closeToolMcp: undefined, + currentTurn: undefined, + lastTurnToolCallIds: [], + parkedApproval: undefined, + approvalGateCount: 0, + destroyed: false, + destroy: async () => {}, + clearTurn: () => {}, + }; + + environment.clearTurn = () => { + environment.currentTurn = undefined; + }; + + return { + ok: true as const, + acquireStartedAt, + agentMountDir, + artifactId, + binaryPath, + deferredClientToolRelay, + env, + environment, + localBuiltinGatingUnenforceable, + logger, + localModelConfigUnwritable, + mcpAbort, + piExtEnv, + piModelConfig, + piModelConfigError, + piSessionDir, + piSkillSnapshot, + plan, + runAgentDir, + runCred, + sessionForMount, + signAgentMount, + signMount, + strictModel, + timingLog, + }; +} diff --git a/services/runner/src/engines/sandbox_agent/environment.ts b/services/runner/src/engines/sandbox_agent/environment.ts new file mode 100644 index 0000000000..f66a87d2ba --- /dev/null +++ b/services/runner/src/engines/sandbox_agent/environment.ts @@ -0,0 +1,1100 @@ +/** + * sandbox-agent harness driver. + * + * Drives a coding harness (Pi, Claude Code, ...) over the Agent Client Protocol (ACP) + * through the `sandbox-agent` daemon, instead of the bespoke Pi SDK calls in the pi + * engine. It serves the same /run contract (AgentRunRequest -> AgentRunResult), so the + * Python side stays thin and the choice of harness/sandbox is config, not new code. + * + * Per invoke (cold), mirroring the shipped code-evaluator DaytonaRunner pattern: + * + * SandboxAgent.start({ sandbox: local({ env }) | daytona({ create }) }) + * -> createSession({ agent: , cwd, model }) + * -> write AGENTS.md into cwd + * -> session.prompt([{ type: "text", text }]) + * -> accumulate ACP `agent_message_chunk` text + build the trace + * -> destroySandbox() + * + * Two orthogonal axes swap independently: the sandbox (where the daemon runs) and the + * harness (which engine). The ACP boundary is daemon-to-harness; the service-to-sandbox-agent + * hop stays harness-agnostic behind the Harness port. + * + * Session keep-alive (flag-gated, off by default) splits the per-invoke work into + * `acquireEnvironment` (session-scoped: sandbox, mount, session, MCP wiring) and `runTurn` + * (per-turn: otel run, prompt, usage, trace). `runSandboxAgent` composes them exactly as + * before (acquire -> runTurn -> destroy), so with the flag off behavior is byte-identical. + * The dispatch in `server.ts` reuses the two halves to continue a live session across a turn + * boundary. See docs/design/agent-workflows/projects/session-keepalive/plan.md. + * + * Tracing is built here from the ACP event stream (see tracing/otel.ts createSandboxAgentOtel), + * so it is uniform across every harness and always nests under the caller's /invoke + * span. stdout is reserved for the JSON result (see cli.ts); logs go to stderr. + */ +import { mkdirSync, rmSync } from "node:fs"; + +import { apiBase } from "../../apiBase.ts"; + +import { + InMemorySessionPersistDriver, + SandboxAgent, + type SessionEvent, + type SessionPermissionRequest, +} from "sandbox-agent"; + +import { + resolveRunSessionId, + type AgentRunRequest, +} from "../../protocol.ts"; +import { advertisedToolSpecs } from "../../tools/public-spec.ts"; +import { createAcpFetch } from "./acp-fetch.ts"; +import { + assert, + assertRequiredCapabilities, + probeCapabilities, +} from "./capabilities.ts"; +import { DaytonaReconnectTerminalError } from "./daytona-provider.ts"; +import { + createCookieFetch, + DAYTONA_PI_DIR, + prepareDaytonaPiAssets, +} from "./daytona.ts"; +import { conciseError } from "./errors.ts"; +import { buildSessionMcpServers } from "./mcp.ts"; +import { applyModel } from "./model.ts"; +import { + discoverTunnelEndpoint, + harnessSessionMounts, + mountHarnessSessionDirs, + mountStorage, + mountStorageRemote, + signSessionMountCredentials, + storeReachableFromSandbox, + unmountStorage, + type MountCredentials, +} from "./mount.ts"; +import { + PI_MODEL_CONFIG_WRITE_FAILED_MESSAGE, + PI_PERMISSION_EXTENSION_UNAVAILABLE_MESSAGE, + prepareLocalPiAssets, + uploadSystemPromptToSandbox, + writeSystemPromptLocal, +} from "./pi-assets.ts"; +import { + AGENT_MOUNT_ENV_VAR, + agentMountPath, + linkAgentFiles, + linkAgentFilesRemote, + seedAgentReadme, + seedAgentReadmeRemote, +} from "./agent-mount.ts"; +import { + AGENT_MOUNT_SYSTEM_PROMPT_SEGMENT, + claudeMountSystemPromptMeta, + combineAppendSystemPrompt, + type ClaudeSystemPromptMeta, +} from "./agent-mount-guidance.ts"; +import { + routePermissionRequestToActiveTurn, + routeSessionEventToActiveTurn, +} from "./session-events.ts"; +import { buildSandboxProvider } from "./provider.ts"; +import { + clearSandboxPointer, + readStoredSandboxPointer, + writeSandboxPointer, +} from "./sandbox-reconnect.ts"; +import type { + AcquireEnvironmentResult, + SandboxAgentDeps, + SessionEnvironment, +} from "./runtime-contracts.ts"; +import { + containsTransportEndpointDisconnected, + isTransportEndpointDisconnected, + runCredential, +} from "./runtime-policy.ts"; +import { hydrateHarnessSessionFromDurable } from "./session-continuity-durable.ts"; +import { + eligibleAgentSessionId, + nextTurnIndex, + sessionContinuityStore, +} from "./session-continuity.ts"; +import { projectScopeFor } from "./session-identity.ts"; +import { + teardownDisposition, + type TeardownReason, +} from "./teardown.ts"; +import { + uploadToolMcpAssets, + type ToolMcpAssets, +} from "./tool-mcp-assets.ts"; +import { prepareWorkspace } from "./workspace.ts"; +import { prepareEnvironmentSetup } from "./environment-setup.ts"; + +function log(message: string): void { + process.stderr.write(`[sandbox-agent] ${message}\n`); +} + +type Log = (message: string) => void; + +const LOCAL_DURABLE_CWD_ENOTCONN_REMOUNT_LIMIT = 1; + +// In-flight sandbox handles, by run. A process KILL (docker stop / SIGTERM / OOM mid-run) skips +// the per-run teardown — so a shutdown signal handler (see `server.ts`) drains this set to +// best-effort delete any still-running sandbox before exit. Remote (Daytona) sandboxes that even a +// signal can never reach (SIGKILL/OOM) self-reap via the lifecycle reapers in `provider.ts`. +const inFlightSandboxes = new Set<{ + destroy: (opts?: { reason?: TeardownReason }) => Promise; + sessionId: string; + mountProjectId?: string; + projectScopeId?: string; +}>(); + +/** + * Best-effort delete every sandbox currently mid-run, bounded so it can never hang shutdown. + * Called from the process signal handler so `docker stop` reaps remote sandboxes instead of + * leaking them. Each delete is independent and its own failure is swallowed; the whole sweep is + * raced against `timeoutMs` so a slow Daytona API call cannot block the exit. + */ +export async function destroyInFlightSandboxes( + timeoutMs = 5000, + reason: TeardownReason = "shutdown-in-flight", +): Promise { + const pending = [...inFlightSandboxes]; + if (pending.length === 0) return; + const sweep = Promise.allSettled( + pending.map((environment) => + Promise.resolve(environment.destroy({ reason })).catch(() => {}), + ), + ); + await Promise.race([ + sweep, + new Promise((resolve) => setTimeout(resolve, timeoutMs)), + ]); +} + +/** + * Same drain as `destroyInFlightSandboxes`, scoped to one session (and, when supplied, its + * owning project). Backs the HTTP `/kill` route so a caller can only tear down its own + * session's in-flight sandbox(es) — the unscoped sweep above stays an in-process-only call + * (the shutdown handler). + * + * Filters on `projectScopeId` (same run-context-preferred, mount-fallback precedence as + * `poolKeyFor`/`projectScopeFor` — never `mountProjectId` alone, which is undefined for a + * mountless run and would make a scoped kill silently match nothing). A sandbox whose run had + * no project scope at all (`projectScopeId` undefined) never matches a scoped `projectId` + * filter: `/kill` requires a non-blank `projectId`, so there is no caller this in-flight entry + * could ever be proven to belong to — the same no-scope-no-park invariant the pool enforces, + * mirrored here as no-scope-no-scoped-kill. It still falls to the unscoped shutdown sweep. + */ +export async function destroyInFlightSandboxesForSession( + sessionId: string, + projectId: string | undefined, + timeoutMs = 5000, + reason: TeardownReason = "kill", +): Promise { + const pending = [...inFlightSandboxes].filter( + (environment) => + environment.sessionId === sessionId && + (!projectId || environment.projectScopeId === projectId), + ); + if (pending.length === 0) return; + const sweep = Promise.allSettled( + pending.map((environment) => + Promise.resolve(environment.destroy({ reason })).catch(() => {}), + ), + ); + await Promise.race([ + sweep, + new Promise((resolve) => setTimeout(resolve, timeoutMs)), + ]); +} + +/** + * Sign the session's durable mount up front so keep-alive can build a pool key (the mount's + * owning `projectId`, the FALLBACK project scope when the run carries no service-stamped + * `runContext.project.id`) and credential epoch without acquiring the whole environment. Returns + * exactly what the sign yielded: `null` when there is no session/credential to sign with, or + * the sign returned no usable mount (store unconfigured, 503, ephemeral fallback). The caller + * threads the result — null included — into `acquireEnvironment` as `presignedMount`, so the + * mount is signed exactly once per run on every path. A null result no longer forces a cold run + * on its own: the request still parks when the run context supplied a project scope, and only + * skips parking when NEITHER source yields one (`poolKeyFor` returns null). + */ +export async function resolveKeepaliveMount( + request: AgentRunRequest, + deps: SandboxAgentDeps = {}, +): Promise { + const logger = deps.log ?? log; + const sessionForMount = request.sessionId?.trim(); + const runCred = runCredential(request); + if (!sessionForMount || !runCred) return null; + const signMount = + deps.signSessionMountCredentials ?? signSessionMountCredentials; + return signMount(sessionForMount, { + apiBase: apiBase(), + authorization: runCred, + log: logger, + }); +} + +/** + * Build the session-scoped environment: sign the mount, build the run plan, start the sandbox, + * mount the durable cwd, prepare the workspace, probe capabilities, wire the internal tool-MCP + * server, and open the ACP session. Session-lifetime `onEvent`/`onPermissionRequest` listeners + * are attached once here and demux into `env.currentTurn`. + * + * Finalizers register incrementally on `env` as each resource is acquired; a mid-acquire failure + * runs `env.destroy()` (which null-checks every resource, so a half-built environment cannot + * leak) and returns `{ ok: false }`, mirroring today's shared teardown. When `presignedMount` is + * supplied (the keep-alive cold path already signed to build the pool key) the initial sign is + * skipped so the mount is signed once per run. + */ +export async function acquireEnvironment( + request: AgentRunRequest, + deps: SandboxAgentDeps = {}, + signal?: AbortSignal, + presignedMount?: MountCredentials | null, +): Promise { + const setup = await prepareEnvironmentSetup(request, deps, presignedMount); + if (!setup.ok) return setup; + const { + acquireStartedAt, + agentMountDir, + artifactId, + binaryPath, + deferredClientToolRelay, + env, + environment, + localBuiltinGatingUnenforceable, + localModelConfigUnwritable, + logger, + mcpAbort, + piExtEnv, + piModelConfig, + piModelConfigError, + piSessionDir, + piSkillSnapshot, + plan, + runCred, + sessionForMount, + signAgentMount, + signMount, + strictModel, + timingLog, + } = setup; + let runAgentDir = setup.runAgentDir; + + // The one complete, idempotent teardown — the same steps the old per-run `finally` ran, in the + // same order. Every resource is null-checked, so it is safe after a partial acquire and safe to + // call twice (the guard returns on a second call). It must never throw. + environment.destroy = async (opts?: { reason?: TeardownReason }) => { + if (environment.destroyed) return; + environment.destroyed = true; + await environment.runtimeRemount?.catch(() => {}); + inFlightSandboxes.delete(environment); + await environment.currentTurn?.toolRelay?.stop().catch(() => {}); + // Teardown backstop: destroy any in-flight loopback `tools/call` before closing the server. + environment.mcpAbort.abort(); + await environment.closeToolMcp?.().catch(() => {}); + // Graceful `session/cancel` BEFORE tearing down the daemon, or the ACP adapter subprocess + // reparents to PID 1 and never exits. Skip if the pause path already sent it. + if (environment.session && !environment.sessionDestroyRequested) + await environment.sandbox + ?.destroySession?.(environment.session.id) + .catch(() => {}); + const disposition = teardownDisposition(opts?.reason ?? "failed-turn"); + let parked = false; + if ( + disposition === "stop" && + plan.isDaytona && + environment.sandbox?.pauseSandbox + ) { + const sandboxLogId = environment.sandbox.sandboxId ?? plan.sandboxId; + try { + await environment.sandbox.pauseSandbox(); + parked = true; + logger(`parked sandbox=${sandboxLogId}`); + } catch (err) { + logger( + `pause failed sandbox=${sandboxLogId}: ${conciseError(err, plan.harness)}`, + ); + } + } + if (!parked) await environment.sandbox?.destroySandbox().catch(() => {}); + await environment.sandbox?.dispose().catch(() => {}); + // Unmount the durable cwd BEFORE removing the dir: data lives in the store, only the host + // mountpoint is torn down. If unmount is not CONFIRMED gone, skip the delete: rmSync must + // never run against a possibly-live FUSE mount into the durable store. + if (environment.mountedCwd) { + environment.durableCwdSafeToDelete = await ( + environment.deps.unmountStorage ?? unmountStorage + )(environment.mountedCwd, { log }).catch(() => false); + } + if (!parked && !plan.isDaytona && environment.agentMountedPath) { + const agentMountSafeToDelete = await ( + environment.deps.unmountStorage ?? unmountStorage + )(environment.agentMountedPath, { log }).catch(() => false); + if (agentMountSafeToDelete) { + try { + rmSync(environment.agentMountedPath, { + recursive: true, + force: true, + }); + } catch (err) { + logger( + `agent mountpoint cleanup failed path=${environment.agentMountedPath}: ${conciseError(err, plan.harness)}`, + ); + } + } + } + if (!environment.durableCwdSafeToDelete) { + logger( + `durable cwd unmount not confirmed, skipping workspace cleanup cwd=${plan.cwd}`, + ); + } else { + await environment.workspace?.cleanup().catch(() => {}); + } + // The per-run Agenta agent dir (skills isolation) is throwaway; remove it too. This is only + // ever a temp dir: a subscription run leaves `runAgentDir` undefined precisely so that the + // operator's mounted login (which the harness runs out of directly) is never deleted here. + if (environment.runAgentDir) + rmSync(environment.runAgentDir, { recursive: true, force: true }); + // Backstop: the extension deletes this on read; remove it here too in case the harness never + // started (or crashed before reading it), so the bearer never lingers. + if (environment.otlpAuthFilePath) + rmSync(environment.otlpAuthFilePath, { force: true }); + // Remove the per-run skills temp root the materializer created (success or error). + plan.skillsCleanup(); + }; + + let agentMountGuidanceActive = false; + const activateAgentMountGuidance = async (): Promise => { + const mountedPath = environment.agentMountedPath; + if (!mountedPath || agentMountGuidanceActive) return; + agentMountGuidanceActive = true; + + // Only advertise durable storage after the mount is confirmed active. Local daemon env is + // still mutable here because local mounts run before SandboxAgent.start below. Daytona cannot + // change daemon env after sandbox creation, so its harness discovers the mount through the + // post-mount system-prompt channel and the cwd-local agent-files symlink instead. + if (!plan.isDaytona) { + env[AGENT_MOUNT_ENV_VAR] = mountedPath; + piExtEnv[AGENT_MOUNT_ENV_VAR] = mountedPath; + } + if (!plan.isPi) return; + + plan.appendSystemPrompt = combineAppendSystemPrompt( + plan.appendSystemPrompt, + AGENT_MOUNT_SYSTEM_PROMPT_SEGMENT, + ); + plan.hasSystemPrompt = true; + if (plan.isDaytona) { + await uploadSystemPromptToSandbox( + environment.sandbox, + DAYTONA_PI_DIR, + plan.systemPrompt, + plan.appendSystemPrompt, + logger, + ); + return; + } + if (environment.runAgentDir) { + writeSystemPromptLocal( + environment.runAgentDir, + plan.systemPrompt, + plan.appendSystemPrompt, + logger, + ); + return; + } + // Discarding `.extensionInstalled` here is safe, and a fail-closed throw here would be + // unsound anyway (both callers wrap this in a mount try/catch that logs and continues, so a + // throw could not stop the run). Reachability: managed/none local Pi runs always created a + // throwaway dir in the first prepareLocalPiAssets call, so `environment.runAgentDir` is set + // for them and they returned above — only the subscription (runtime_provided) path reaches + // this re-prep. That path installs into the SAME operator mount the first call already + // installed into, and the fail-closed gating check right after that first call stopped the + // run when the install was required but failed. So by the time this runs, either enforcement + // is not needed (policy allows everything) or the extension file is already on disk from the + // verified first install; a transient failure here cannot remove it. + runAgentDir = prepareLocalPiAssets({ plan, env, log: logger }).dir; + environment.runAgentDir = runAgentDir; + }; + + // --- local durable cwd mount helpers (session-scoped, close over environment) ------ // + const mountLocalDurableCwd = async (reason: string): Promise => { + if (!environment.mountCreds || plan.isDaytona) return false; + logger( + `local durable cwd mount (${reason}) session=${sessionForMount} cwd=${plan.cwd}`, + ); + environment.durableCwdSafeToDelete = false; + const mounted = await (deps.mountStorage ?? mountStorage)( + plan.cwd, + environment.mountCreds, + { + log: logger, + }, + ); + if (mounted) { + environment.mountedCwd = plan.cwd; + return true; + } + // A false result means mountStorage stopped the attempt and confirmed the path detached. + environment.durableCwdSafeToDelete = true; + return false; + }; + const mountLocalAgentCwd = async (): Promise => { + if (!environment.agentMountCreds || plan.isDaytona) return false; + const mountPath = agentMountPath(plan.cwd); + if (environment.agentMountedPath === mountPath) return true; + try { + mkdirSync(mountPath, { recursive: true }); + if ( + !(await (deps.mountStorage ?? mountStorage)( + mountPath, + environment.agentMountCreds, + { log: logger }, + )) + ) { + // false means mountStorage confirmed detach is safe. This path is a sibling of the + // session cwd, so workspace cleanup cannot remove the failed mountpoint stub. + rmSync(mountPath, { recursive: true, force: true }); + return false; + } + environment.agentMountedPath = mountPath; + await seedAgentReadme(mountPath, { log: logger }); + await linkAgentFiles(plan.cwd, mountPath, { log: logger }); + await activateAgentMountGuidance(); + return true; + } catch (err) { + logger( + `local agent mount failed artifact=${artifactId}: ${conciseError(err, plan.harness)}`, + ); + return false; + } + }; + let localAgentMountEnotconnRemounts = 0; + const reSignAndRemountLocalAgentMount = async (): Promise => { + if (!artifactId || !runCred || plan.isDaytona) return false; + if ( + localAgentMountEnotconnRemounts >= + LOCAL_DURABLE_CWD_ENOTCONN_REMOUNT_LIMIT + ) { + logger( + `local agent mount ENOTCONN remount limit reached artifact=${artifactId} path=${agentMountPath(plan.cwd)}`, + ); + return false; + } + localAgentMountEnotconnRemounts += 1; + logger( + `local agent mount ENOTCONN artifact=${artifactId}; re-signing and remounting`, + ); + const fresh = await signAgentMount(artifactId, { + apiBase: apiBase(), + authorization: runCred, + log: logger, + }); + if (!fresh) { + logger( + `local agent mount re-sign returned no credentials artifact=${artifactId}`, + ); + return false; + } + environment.agentMountCreds = fresh; + // Clear the marker so mountLocalAgentCwd remounts instead of short-circuiting. + environment.agentMountedPath = undefined; + return mountLocalAgentCwd(); + }; + let localDurableCwdEnotconnRemounts = 0; + const reSignAndRemountLocalCwd = async (): Promise => { + if (!sessionForMount || !runCred || plan.isDaytona) return false; + if ( + localDurableCwdEnotconnRemounts >= + LOCAL_DURABLE_CWD_ENOTCONN_REMOUNT_LIMIT + ) { + logger( + `local durable cwd ENOTCONN remount limit reached session=${sessionForMount} cwd=${plan.cwd}`, + ); + return false; + } + localDurableCwdEnotconnRemounts += 1; + logger( + `local durable cwd ENOTCONN session=${sessionForMount} cwd=${plan.cwd}; re-signing and remounting`, + ); + const fresh = await signMount(sessionForMount, { + apiBase: apiBase(), + authorization: runCred, + log: logger, + }); + if (!fresh) { + logger( + `local durable cwd re-sign returned no credentials session=${sessionForMount}`, + ); + return false; + } + environment.mountCreds = fresh; + return mountLocalDurableCwd("enotconn-retry"); + }; + const remountLocalCwdAfterRuntimeEnotconn = (event: unknown): void => { + if (plan.isDaytona) return; + // The event cannot say which mount broke; remount every eligible one (alive mounts no-op). + const cwdEligible = !!environment.mountCreds && !!environment.mountedCwd; + const agentEligible = + !!environment.agentMountCreds && !!environment.agentMountedPath; + if (!cwdEligible && !agentEligible) return; + if ( + environment.runtimeRemount || + !containsTransportEndpointDisconnected(event) + ) + return; + logger( + `local durable mount ENOTCONN observed in ACP event session=${sessionForMount} cwd=${plan.cwd}; re-signing and remounting`, + ); + environment.runtimeRemount = (async () => { + const cwdOk = cwdEligible ? await reSignAndRemountLocalCwd() : true; + const agentOk = agentEligible + ? await reSignAndRemountLocalAgentMount() + : true; + return cwdOk && agentOk; + })().catch((err) => { + logger( + `local durable mount runtime remount failed session=${sessionForMount}: ${conciseError(err, plan.harness)}`, + ); + return false; + }); + }; + + try { + // Fail loud before any sandbox/mount infra spins up: an applicable-but-incomplete + // OpenAI-compatible custom request is a hard error, never a silent fall-back (Decision 5). + if (piModelConfigError) { + throw piModelConfigError; + } + // Fail closed before any sandbox/mount infra spins up: a local Pi run whose policy could gate a + // built-in tool cannot proceed without the permission extension installed (Decision 2). + if (localBuiltinGatingUnenforceable) { + throw new Error(PI_PERMISSION_EXTENSION_UNAVAILABLE_MESSAGE); + } + // Fail closed: a local managed custom run whose models.json could not be materialized cannot + // fall through to a default provider (Decision 6). + if (localModelConfigUnwritable) { + throw new Error(PI_MODEL_CONFIG_WRITE_FAILED_MESSAGE); + } + // Persist events in-process so a follow-up turn can resume by session id. + const persist = + deps.createPersist?.() ?? new InMemorySessionPersistDriver(); + const startSandboxAgent = + deps.startSandboxAgent ?? + ((options: Parameters[0]) => + SandboxAgent.start(options)); + // Local geesefs runs on the host, so mount before spawning the daemon. This lets the + // mount-success path add guidance/env atomically, while a failed mount starts a normal + // scratch-only harness with no false durable-storage signal. + if (environment.mountCreds && !plan.isDaytona) { + await mountLocalDurableCwd("initial"); + } + if (environment.agentMountCreds && !plan.isDaytona) { + await mountLocalAgentCwd(); + } + const sandboxProvider = (deps.buildSandboxProvider ?? buildSandboxProvider)( + plan.sandboxId, + env, + binaryPath, + piExtEnv, + plan.secrets, + plan.sandboxPermission, + ); + const startOptions = { + sandbox: sandboxProvider, + persist, + // Propagate caller cancellation (a client disconnect on the streaming HTTP edge) so an + // in-flight run aborts instead of finishing unobserved. `destroy` still disposes. + ...(signal ? { signal } : {}), + // Long-timeout undici dispatcher so a paused HITL turn is not reaped by undici's default + // headersTimeout; Daytona additionally carries the per-sandbox auth cookie. + fetch: plan.isDaytona + ? (deps.createCookieFetch ?? createCookieFetch)() + : (deps.createAcpFetch ?? createAcpFetch)(), + }; + // A stored sandbox id is trusted: reconnect it by id and let reconnect converge its network + // policy to this run's plan. Any reconnect failure falls through to a fresh create. Snapshot + // and image drift are accepted as per-conversation version pinning, not grounds for a rebuild. + const storedSandboxPointer = + plan.isDaytona && sessionForMount && runCred + ? await (deps.readStoredSandboxPointer ?? readStoredSandboxPointer)( + sessionForMount, + { authorization: runCred, log: logger }, + ) + : undefined; + if (storedSandboxPointer) { + const sandboxStartStartedAt = Date.now(); + try { + environment.sandbox = await startSandboxAgent({ + ...startOptions, + sandboxId: storedSandboxPointer.sandboxId, + }); + logger( + `reconnected sandbox=${storedSandboxPointer.sandboxId} session=${sessionForMount}`, + ); + } catch (err) { + logger( + `reconnect failed sandbox=${storedSandboxPointer.sandboxId}, creating fresh: ${conciseError(err, plan.harness)}`, + ); + if ( + err instanceof DaytonaReconnectTerminalError && + sessionForMount && + runCred + ) { + // The post-hydrate write later in acquire is authoritative. This clear only prevents + // repeated doomed reconnects if acquire fails before reaching that write. Hydrate + // first: after a runner restart the in-memory store is behind the durable + // latest_turn_index, and an unhydrated guard token would be rejected as stale. + await ( + deps.hydrateHarnessSessionFromDurable ?? + hydrateHarnessSessionFromDurable + )( + sessionForMount, + plan.harness, + deps.sessionContinuityStore ?? sessionContinuityStore, + { authorization: runCred, log: logger }, + ); + await (deps.clearSandboxPointer ?? clearSandboxPointer)( + sessionForMount, + nextTurnIndex( + sessionForMount, + deps.sessionContinuityStore ?? sessionContinuityStore, + ), + { authorization: runCred, log: logger }, + ); + } + } finally { + timingLog("sandbox_start", sandboxStartStartedAt, " mode=reconnect"); + } + } + if (!environment.sandbox) { + const sandboxStartStartedAt = Date.now(); + try { + environment.sandbox = await startSandboxAgent(startOptions); + } finally { + timingLog("sandbox_start", sandboxStartStartedAt, " mode=create"); + } + } + environment.resumable = Boolean(plan.isDaytona && sessionForMount); + // Track the live handle so a shutdown signal handler can delete it if `destroy` is skipped by + // a process KILL; removed in `destroy` on every normal exit so it is never double-deleted. + if (environment.sandbox) inFlightSandboxes.add(environment); + + // On Daytona, push the harness login, the extension, and AGENTS.md into the remote sandbox. + // For a non-Pi harness with executable tools, also push the in-sandbox stdio MCP shim + // assets (bundle + public-specs file): a non-Pi harness in the sandbox cannot reach the + // runner-loopback HTTP MCP channel, so the harness's ACP adapter spawns the uploaded shim + // as the internal stdio MCP server instead. Uploaded unconditionally for non-Pi (the + // capability probe runs later; a harness that turns out to lack MCP fails loud in + // `assertRequiredCapabilities` below). Pi delivers via its extension; local non-Pi uses + // the loopback HTTP channel — neither needs this. The upload helper THROWS when the shim + // cannot be delivered (fail loud — this path requires it). + let internalToolMcp: ToolMcpAssets | undefined; + if (plan.isDaytona) { + const daytonaExtensionInstalled = await ( + deps.prepareDaytonaPiAssets ?? prepareDaytonaPiAssets + )({ + sandbox: environment.sandbox, + plan: { ...plan, skillDirs: [] }, + piModelConfig, + log: logger, + }); + // Fail closed (Decision 2): same guarantee as the local path. A genuine upload failure on the + // Daytona sandbox stops the run rather than running Pi's built-in tools unprotected. + if (plan.isPi && plan.builtinGatingActive && !daytonaExtensionInstalled) { + throw new Error(PI_PERMISSION_EXTENSION_UNAVAILABLE_MESSAGE); + } + if (!plan.isPi && plan.executableToolSpecs.length > 0) { + internalToolMcp = await ( + deps.uploadToolMcpAssets ?? uploadToolMcpAssets + )( + environment.sandbox, + plan.toolMcpDir, + advertisedToolSpecs(plan.executableToolSpecs), + logger, + ); + } + } + + // Durable cwd: mount BEFORE createSession (so the session opens inside it) and BEFORE + // workspace materialization (so AGENTS.md, harness files, and skills land in the durable + // prefix instead of being hidden under the FUSE mount). + if (environment.mountCreds && plan.isDaytona) { + const mountsStartedAt = Date.now(); + try { + // Mount against the store's own endpoint when the sandbox can reach it (public S3); fall + // back to the tunnel only for an in-network store. No tunnel + in-network store => skip. + const storeEndpoint = environment.mountCreds.endpoint; + const endpoint = storeReachableFromSandbox(storeEndpoint) + ? undefined + : ((await (deps.discoverTunnelEndpoint ?? discoverTunnelEndpoint)({ + log: logger, + })) ?? undefined); + const canMount = storeReachableFromSandbox(storeEndpoint) || !!endpoint; + if ( + canMount && + (await (deps.mountStorageRemote ?? mountStorageRemote)( + environment.sandbox, + plan.cwd, + environment.mountCreds, + { + endpoint, + log: logger, + }, + )) + ) { + logger(`remote durable cwd active for session=${sessionForMount}`); + } + // Per-harness session/transcript-dir mounts, remote-only by construction (this whole + // branch is `plan.isDaytona`) — local runs never reach here, so they stay mount-free/ + // byte-identical. Transcript mounts derive from the session contract (a durable cwd mount + // is active), with no separate public switch or credential/session-id path. + if (canMount && sessionForMount && runCred) { + const dirs = harnessSessionMounts( + plan.acpAgent, + "/home/sandbox", + DAYTONA_PI_DIR, + ); + await (deps.mountHarnessSessionDirs ?? mountHarnessSessionDirs)( + environment.sandbox, + sessionForMount, + dirs, + endpoint, + { + apiBase: apiBase(), + authorization: runCred, + log: logger, + }, + ); + } + } finally { + timingLog("mounts", mountsStartedAt); + } + } + if ( + environment.agentMountCreds && + agentMountDir && + plan.isDaytona && + !environment.agentMountedPath + ) { + const agentMountStartedAt = Date.now(); + try { + const storeEndpoint = environment.agentMountCreds.endpoint; + const endpoint = storeReachableFromSandbox(storeEndpoint) + ? undefined + : ((await (deps.discoverTunnelEndpoint ?? discoverTunnelEndpoint)({ + log: logger, + })) ?? undefined); + const canMount = storeReachableFromSandbox(storeEndpoint) || !!endpoint; + const mountPath = agentMountDir; + if ( + canMount && + (await (deps.mountStorageRemote ?? mountStorageRemote)( + environment.sandbox, + mountPath, + environment.agentMountCreds, + { endpoint, log: logger }, + )) + ) { + environment.agentMountedPath = mountPath; + await seedAgentReadmeRemote(environment.sandbox, mountPath, { + log: logger, + }); + await linkAgentFilesRemote(environment.sandbox, plan.cwd, mountPath, { + log: logger, + }); + await activateAgentMountGuidance(); + logger(`remote agent mount active for artifact=${artifactId}`); + } + } catch (err) { + logger( + `remote agent mount failed artifact=${artifactId}: ${conciseError(err, plan.harness)}`, + ); + } finally { + timingLog("agent_mount", agentMountStartedAt); + } + } + + const prepareWorkspaceStartedAt = Date.now(); + try { + environment.workspace = await (deps.prepareWorkspace ?? prepareWorkspace)( + { + sandbox: environment.sandbox, + plan, + piSkillSnapshot, + log: logger, + }, + ); + } catch (err) { + if ( + !plan.isDaytona && + environment.mountCreds && + isTransportEndpointDisconnected(err) && + (await reSignAndRemountLocalCwd()) + ) { + logger( + `retrying workspace preparation after local durable cwd remount`, + ); + environment.workspace = await ( + deps.prepareWorkspace ?? prepareWorkspace + )({ + sandbox: environment.sandbox, + plan, + piSkillSnapshot, + log: logger, + }); + } else { + throw err; + } + } finally { + timingLog("prepare_workspace", prepareWorkspaceStartedAt); + } + + // Pi native transcripts belong to the conversation workspace, not the temporary agent + // directory that holds credentials, settings, extensions, skills, and system prompts. + // The cwd mount is already active here on local and Daytona before Pi starts. + if (piSessionDir) { + if (plan.isDaytona) { + await environment.sandbox.mkdirFs({ path: piSessionDir }); + } else { + mkdirSync(piSessionDir, { recursive: true }); + } + } + + // Sandbox-start invariant: `startSandboxAgent` must hand back a usable handle. + assert( + environment.sandbox && + typeof environment.sandbox.createSession === "function", + `sandbox provider '${plan.sandboxId}' returned no usable sandbox handle`, + ); + + // Probe what this harness supports and branch on capabilities, not on the harness name. + const probeCapabilitiesStartedAt = Date.now(); + let probed; + try { + probed = await (deps.probeCapabilities ?? probeCapabilities)( + environment.sandbox, + plan.acpAgent, + ); + } finally { + timingLog("probe_capabilities", probeCapabilitiesStartedAt); + } + const capabilities = probed.capabilities; + environment.capabilities = capabilities; + + // Fail loud (A7): a run that REQUIRES a capability the harness lacks errors specifically + // rather than silently dropping the behavior. + assertRequiredCapabilities({ + harness: plan.harness, + isPi: plan.isPi, + probed, + toolSpecs: plan.toolSpecs, + log: logger, + }); + + const sessionMcp = await buildSessionMcpServers({ + isPi: plan.isPi, + capabilities, + harness: plan.harness, + isDaytona: plan.isDaytona, + toolSpecs: plan.toolSpecs, + userMcpServers: request.mcpServers, + relayDir: plan.relayDir, + clientToolRelay: deferredClientToolRelay, + signal: mcpAbort.signal, + // The uploaded in-sandbox stdio MCP shim assets, set only on Daytona + non-Pi + + // executable-tools; advertises the gateway tools the loopback channel cannot reach + // from inside the sandbox. No server to close for this entry (the harness owns the + // shim process), so `sessionMcp.close` semantics are unchanged. + internalToolMcp, + log: logger, + }); + // Close the internal gateway-tool MCP server (if one started) when the session is destroyed. + environment.closeToolMcp = sessionMcp.close; + + // Shared session-init payload for both the createSession and continuity-resume paths below. + // Built as a plain variable (not an inline object literal at the call site) so the extra + // `_meta` key survives the daemon SDK's narrow `Omit` types — + // the daemon's own runtime forwards `_meta` unconditionally (`normalizeSessionInit` / + // `buildLoadSessionParams` in the vendored `sandbox-agent` patch), only the published types + // are stricter than the wire protocol they describe. + const claudeSystemPromptMeta: ClaudeSystemPromptMeta | undefined = + environment.agentMountedPath && plan.acpAgent === "claude" + ? claudeMountSystemPromptMeta(AGENT_MOUNT_SYSTEM_PROMPT_SEGMENT) + : undefined; + const sessionInit = { + cwd: plan.cwd, + mcpServers: sessionMcp.servers, + ...(claudeSystemPromptMeta ? { _meta: claudeSystemPromptMeta } : {}), + }; + + // If this harness authored the conversation's most recent turn (staleness-guarded) and we + // still remember its native `agentSessionId`, seed the fresh persist driver with a synthetic + // record and resume-by-id so the patched `resumeSession` reaches `session/load` instead of + // `session/new`. Any failure inside `resumeSession` already degrades to a plain new session + // internally (the patch's own `catch {}` around `loadRemoteSession`), so this call is safe to + // attempt unconditionally whenever we have an eligible id — worst case it is exactly today's + // cold `createSession`. + const continuitySessionKey = request.sessionId?.trim(); + const continuityStore = + deps.sessionContinuityStore ?? sessionContinuityStore; + // Seed the in-memory store from the durable row before consulting it, so a resume after a + // runner restart (in-memory map lost) still sees the prior turn's eligibility. No-op (and + // cheap) when the store already has a live in-process record. + if (continuitySessionKey && runCred) { + await ( + deps.hydrateHarnessSessionFromDurable ?? + hydrateHarnessSessionFromDurable + )(continuitySessionKey, plan.harness, continuityStore, { + authorization: runCred, + log: logger, + }); + } + const priorAgentSessionId = continuitySessionKey + ? eligibleAgentSessionId( + continuitySessionKey, + plan.harness, + continuityStore, + ) + : undefined; + const localSessionId = continuitySessionKey + ? `${continuitySessionKey}:${plan.harness}` + : undefined; + // The index THIS turn will occupy once it completes: recorded post-turn against the SAME + // index read here, so a turn that authors turn N leaves the store agreeing with itself. + environment.continuityTurnIndex = continuitySessionKey + ? nextTurnIndex(continuitySessionKey, continuityStore) + : undefined; + // Daytona only: a local run must not overwrite a conversation's remote pointer (switching + // sandboxes mid-conversation would strand the parked Daytona instance). + if (plan.isDaytona && sessionForMount && runCred) { + const liveSandboxId = environment.sandbox?.sandboxId ?? plan.sandboxId; + const pointerWriteOutcome = await ( + deps.writeSandboxPointer ?? writeSandboxPointer + )( + sessionForMount, + { + sandboxId: liveSandboxId, + turnIndex: environment.continuityTurnIndex ?? 0, + }, + { authorization: runCred, log: logger }, + ); + logger( + `sandbox pointer write ${pointerWriteOutcome} session=${sessionForMount} sandbox=${liveSandboxId}`, + ); + } + let loadedFromContinuity = false; + if (priorAgentSessionId && localSessionId) { + await persist.updateSession({ + id: localSessionId, + agent: plan.acpAgent, + agentSessionId: priorAgentSessionId, + lastConnectionId: "", + createdAt: Date.now(), + sessionInit, + }); + const createSessionStartedAt = Date.now(); + try { + environment.session = + await environment.sandbox.resumeSession(localSessionId); + loadedFromContinuity = + environment.session.agentSessionId === priorAgentSessionId; + logger( + `[continuity] session/load attempted session=${continuitySessionKey} ` + + `harness=${plan.harness} loaded=${loadedFromContinuity}`, + ); + } catch (err) { + logger( + `[continuity] resumeSession failed, falling back to cold createSession: ` + + `${conciseError(err, plan.harness)}`, + ); + } finally { + timingLog("create_session", createSessionStartedAt, " mode=load"); + } + } + environment.loadedFromContinuity = loadedFromContinuity; + if (!environment.session) { + const createSessionStartedAt = Date.now(); + try { + environment.session = await environment.sandbox.createSession({ + ...(localSessionId ? { id: localSessionId } : {}), + agent: plan.acpAgent, + cwd: plan.cwd, + sessionInit, + }); + } finally { + timingLog("create_session", createSessionStartedAt, " mode=create"); + } + } + environment.sessionId = resolveRunSessionId( + request, + environment.session.id, + ); + + // Resolve the model first: when the harness rejects the requested id and keeps its own + // default, `model` is undefined and the chat span is labelled "chat". + // + // For a managed OpenAI-compatible custom run, request the FULLY QUALIFIED + // `/` that pi-acp advertises for this provider, not the bare wire + // model id (design Decision 7). `applyModel`/`pickModel` fall back to suffix matching, which + // returns the FIRST advertised id whose suffix matches — so a built-in `openai/` that + // Pi still advertises (the vault key rides in as `OPENAI_API_KEY`, keeping Pi's built-in + // openai provider live) would be selected ahead of the custom `/` when both share + // the model id. That would silently route to api.openai.com instead of the user's endpoint. + // The qualified id is an EXACT match, so it always wins over any bare-suffix collision. + const wantedModel = + piModelConfig && piModelConfig.models.length > 0 + ? `${piModelConfig.providerId}/${piModelConfig.models[0].id}` + : request.model; + environment.model = await (deps.applyModel ?? applyModel)( + environment.session, + wantedModel, + logger, + { strict: strictModel }, + ); + + // Session-lifetime listeners: attach ONCE, each demuxing into the active turn's sink. They + // outlive any single turn, so the routing lives in dedicated non-throwing helpers below. + environment.session.onEvent((event: any) => + routeSessionEventToActiveTurn( + environment, + remountLocalCwdAfterRuntimeEnotconn, + event, + ), + ); + environment.session.onPermissionRequest((req: any) => + routePermissionRequestToActiveTurn(environment, req), + ); + + timingLog("acquire_total", acquireStartedAt); + return { ok: true, env: environment }; + } catch (err) { + const error = conciseError(err, plan.harness, request.provider); + // Mirror today's shared teardown: no otel exists yet during acquire, so there is no partial + // trace to flush — just run the incrementally-registered finalizers and surface the error. + await environment.destroy({ reason: "failed-turn" }); + return { ok: false, error }; + } +} + +/** + * Drop the harness's continuity record after a turn that did not complete. The harness may have + * written a partial turn into its native transcript, so a later `session/load` would resume a + * history the canonical request never sent. Dropping it falls back to cold replay. + */ +export function invalidateContinuity( + sessionId: string | undefined, + harness: string, + deps: SandboxAgentDeps, +): void { + if (!sessionId) return; + (deps.sessionContinuityStore ?? sessionContinuityStore).invalidate( + sessionId, + harness, + ); +} diff --git a/services/runner/src/engines/sandbox_agent/run-turn.ts b/services/runner/src/engines/sandbox_agent/run-turn.ts new file mode 100644 index 0000000000..e044c8d31d --- /dev/null +++ b/services/runner/src/engines/sandbox_agent/run-turn.ts @@ -0,0 +1,608 @@ +import { + PendingApprovalLatch, + permissionsFromRequest, +} from "../../permission-plan.ts"; +import { + resolvePromptText, + type AgentRunRequest, + type AgentRunResult, + type EmitEvent, + type ToolCallbackContext, +} from "../../protocol.ts"; +import { seedForRun } from "../../redaction.ts"; +import { + ApprovalResponder, + ApprovedExecutionGrants, + ConversationDecisions, + extractApprovalDecisions, + extractClientToolOutputs, +} from "../../responder.ts"; +import { + buildWorkflowReferences, + createInteraction, + resolveInteraction, +} from "../../sessions/interactions.ts"; +import { toolSpecsByName } from "../../tools/public-spec.ts"; +import { + localRelayHost, + sandboxRelayHost, + startToolRelay, + type RelayExecutionGuard, +} from "../../tools/relay.ts"; +import { + createSandboxAgentOtel, + TOOL_NOT_EXECUTED_PAUSED, +} from "../../tracing/otel.ts"; +import { attachPermissionResponder } from "./acp-interactions.ts"; +import { buildClientToolRelay } from "./client-tools.ts"; +import { invalidateContinuity } from "./environment.ts"; +import { conciseError } from "./errors.ts"; +import { + PAUSED, + PendingApprovalPauseController, +} from "./pause.ts"; +import { findSwallowedPiError } from "./pi-error.ts"; +import { buildRelayExecutionGuard } from "./relay-guard.ts"; +import { + createRunLimits, + resolveRunLimits, +} from "./run-limits.ts"; +import { + RUN_LIMIT_TRIPPED, + sendLastMessageOnly, + type CurrentTurn, + type ParkedApproval, + type RunTurnOptions, + type SessionEnvironment, +} from "./runtime-contracts.ts"; +import { + runCredential, + serverPermissionsFromRequest, + shouldSuppressPausedToolCallUpdate, +} from "./runtime-policy.ts"; +import { + syncHarnessSessionDurable, +} from "./session-continuity-durable.ts"; +import { sessionContinuityStore } from "./session-continuity.ts"; +import { priorMessages } from "./transcript.ts"; +import { resolveRunUsage } from "./usage.ts"; + +/** + * Run one turn against an acquired environment: start a fresh otel run, wire this turn's pause + * controller / latch / decisions / responder into `env.currentTurn`, restart the tool relay, + * send the prompt, resolve usage, and finish + flush the trace. It does NOT tear down the + * environment (the caller owns `env.destroy`). On a continuation the prompt is only the new user + * text (`buildTurnText` does not run); on a cold turn it is `plan.turnText`, exactly as before. + */ +export async function runTurn( + env: SessionEnvironment, + request: AgentRunRequest, + emit?: EmitEvent, + signal?: AbortSignal, + opts: RunTurnOptions = {}, +): Promise { + const { plan, logger, deps } = env; + const sessionId = env.sessionId; + // Reset the per-turn tool-call id record (the park folds the completed turn's ids into the + // expected next-history fingerprint). + env.lastTurnToolCallIds = []; + // Reset the per-turn approval-park bookkeeping. A fresh turn starts with no parked gate; this + // turn re-records it only if it pauses on a Claude ACP permission gate. (The dispatch has + // already captured any prior park into `opts.resume` before calling us.) + env.parkedApproval = undefined; + env.approvalGateCount = 0; + // Hoisted so the catch can flush a partial trace (mirroring the pre-split `otel?` handling — + // a createOtel throw must still return `{ ok: false }`, not propagate raw) and the finally can + // stop this turn's relay on EVERY exit path (a cleared sink must never orphan it). + let otel: ReturnType | undefined; + let activeTurn: CurrentTurn | undefined; + + // Time-based run deadlines (total/idle/TTFB/per-tool-call) for THIS turn: an idle/wedged harness + // has no deadline anywhere, so a silent or hung turn would hold its sandbox forever. Tripping a + // limit resolves the prompt race with `RUN_LIMIT_TRIPPED`, which ends the turn as an error so the + // caller's teardown (`runSandboxAgent`'s `finally`, or the keep-alive dispatch's evict-on-failure) + // reclaims the sandbox exactly as any other error does. Disposed in the `finally` on every path. + // A human pause retires the deadlines (`notePaused`): a HITL wait is legitimate, not a wedge. + const runLimits = (deps.createRunLimits ?? createRunLimits)( + (deps.resolveRunLimits ?? resolveRunLimits)(logger), + { log: logger }, + ); + let runLimitTrip: (() => void) | undefined; + let runLimitReason: string | undefined; + const runLimitTripped = new Promise((resolve) => { + runLimitTrip = resolve; + }); + runLimits.onTrip((reason) => { + runLimitReason = reason; + runLimitTrip?.(); + }); + + try { + const promptText = resolvePromptText(request); + // Cold: replay the full transcript (plan.turnText). Continuation or loaded: send only new text. + const turnText = sendLastMessageOnly(opts) ? promptText : plan.turnText; + + const run = (deps.createOtel ?? createSandboxAgentOtel)({ + harness: plan.harness, + model: env.model, + skills: plan.skillDirs.map((s) => s.name), + traceparent: request.context?.propagation?.traceparent, + baggage: request.context?.propagation?.baggage, + endpoint: request.telemetry?.exporters?.otlp?.endpoint, + authorization: request.telemetry?.exporters?.otlp?.headers?.authorization, + captureContent: request.telemetry?.capture?.content?.enabled, + // Seed from the keys actually APPLIED to this run (`plan.secrets`) plus the mount's STS + // pair — neither lives in the sidecar's process env. + redactor: seedForRun( + { secrets: plan.secrets, telemetry: request.telemetry }, + [ + env.mountCreds?.accessKey, + env.mountCreds?.secretKey, + env.mountCreds?.sessionToken, + ], + ), + emitSpans: !plan.isPi || plan.isDaytona, + // Every emitted event is a progress signal for the idle/TTFB deadlines (message/thought + // deltas, tool calls and results, usage, ...) — the one seam every harness's output flows + // through. Per-tool-call timers are driven separately from `handleUpdate` below. + emit: emit && runLimits.wrapEmit(emit), + }); + otel = run; + + run.start({ + prompt: promptText, + sessionId, + messages: [ + ...priorMessages(request), + { role: "user", content: promptText }, + ], + }); + + const pause = new PendingApprovalPauseController(() => { + // The sibling settle runs UNCONDITIONALLY, park mode or not: latch-loser tool calls + // announced before the winning gate can never execute this turn, and skipping the settle + // here would leave them as orphaned open parts whenever the dispatch later refuses the park + // (multi-gate, pool full) — `env.destroy()` does not re-run it. The exclusion keeps the + // gated (paused) call itself open, so the live resume is untouched. + run.settleOpenToolCalls( + (id) => pause.isPausedToolCall(id), + TOOL_NOT_EXECUTED_PAUSED, + ); + // Park mode: a parkable permission gate (Claude ACP or Pi ACP) recorded + // `env.parkedApproval` BEFORE firing this pause (the onUserApprovalGate hook runs before + // the single-pause latch). Keep the live session — the gated tool runs on the resume — so + // skip ONLY the mcpAbort and the destroySession. The teardown is not lost: the dispatch + // either parks the session or, if it decides not to (multi-gate, pool full), calls + // `env.destroy()` which runs them. A non-parkable pause (keep-alive off, client tool) + // never records `parkedApproval`, so it still tears down here exactly as today. + if (opts.approvalParkMode && env.parkedApproval) return; + // Abort any in-flight loopback `tools/call` (a paused Claude client tool) BEFORE the + // session teardown, so its handler cannot write a result after the turn ends. + env.mcpAbort.abort(); + env.sessionDestroyRequested = true; + return env.sandbox.destroySession?.(env.session.id); + }); + // A human pause resolves this signal exactly once, the moment the turn parks for input — the one + // place every pause path converges, so the one place to retire the run-limits deadlines for good. + void pause.signal.then(() => runLimits.notePaused()); + + // Publish this turn's sink so the session-lifetime listeners route into it. handleUpdate + // reproduces the old per-event routing (suppress paused frames, handleUpdate, pause re-sweep). + const turn: CurrentTurn = { + run, + pause, + toolRelay: undefined, + handleUpdate: (update) => { + // Per-tool-call deadline: starts on the announcement, ends on a terminal status. Tracked + // regardless of the pause-suppression below (a call already timed out must not linger just + // because a later sibling frame gets suppressed). + const rawFrame = update as { + sessionUpdate?: unknown; + toolCallId?: unknown; + status?: unknown; + }; + if (rawFrame?.sessionUpdate === "tool_call" && rawFrame.toolCallId) { + runLimits.noteToolCallStart(String(rawFrame.toolCallId)); + } else if ( + rawFrame?.sessionUpdate === "tool_call_update" && + rawFrame.toolCallId && + (rawFrame.status === "completed" || rawFrame.status === "failed") + ) { + runLimits.noteToolCallEnd(String(rawFrame.toolCallId)); + } + if (!shouldSuppressPausedToolCallUpdate(update, pause)) { + // Record the emitted tool-call ids (unique, first-seen order): the park folds them + // into the expected next-history fingerprint so a tool-using turn continues live. + const frame = update as { + sessionUpdate?: unknown; + toolCallId?: unknown; + }; + if ( + frame?.sessionUpdate === "tool_call" && + typeof frame.toolCallId === "string" && + frame.toolCallId && + !env.lastTurnToolCallIds.includes(frame.toolCallId) + ) { + env.lastTurnToolCallIds.push(frame.toolCallId); + } + run.handleUpdate(update); + // A sibling announced AFTER the pause won the latch can never execute; settle it + // immediately so the client never holds an orphaned part (idempotent re-sweep). + if (pause.active) { + run.settleOpenToolCalls( + (id) => pause.isPausedToolCall(id), + TOOL_NOT_EXECUTED_PAUSED, + ); + } + } + }, + onPermissionRequest: undefined, + }; + activeTurn = turn; + env.currentTurn = turn; + + const permissionPlan = permissionsFromRequest(request); + const storedDecisionMap = extractApprovalDecisions(request); + if (storedDecisionMap.size > 0) { + logger( + `[HITL] resume state: decisions=${JSON.stringify([...storedDecisionMap.keys()])}`, + ); + } + const decisions = new ConversationDecisions( + storedDecisionMap, + extractClientToolOutputs(request), + ); + const executionGrants = new ApprovedExecutionGrants(); + const latch = new PendingApprovalLatch(); + const responder = + deps.responderFactory?.(request) ?? + new ApprovalResponder(permissionPlan, decisions, logger); + // Every pause seeds the durable interactions plane, whichever gate paused. + const recordPendingInteraction = ( + token: string, + toolName: string | undefined, + toolArgs: unknown, + kind: "user_approval" | "client_tool" = "user_approval", + ): void => { + const cred = runCredential(request); + if (!cred) return; + const references = buildWorkflowReferences(request.runContext?.workflow); + if (!references?.workflow_revision) return; + void createInteraction( + sessionId, + request.turnId ?? "", + token, + kind, + { request: { tool: toolName ?? token, args: toolArgs }, references }, + () => cred, + ); + }; + // Transition the durable interaction row to resolved once its gate is answered. Used both by + // the cold decision-map path (via attachPermissionResponder) and the live approval resume, + // which answers the parked gate directly. The turn-start `cancelStaleInteractions` sweep + // (server.ts) cancels only PENDING gates of OTHER turns and spares this gate two ways: an + // interactions-plane answer already transitioned it to responded, and an in-band answer is + // detected at sweep time (`inBandAnswerToken`) and exempted via the sweep's `tokens` — the + // row stays pending until this resolve lands it as resolved, never cancelled. + const resolveInteractionToken = (token: string): void => { + const cred = runCredential(request); + if (!cred) return; + if ( + !buildWorkflowReferences(request.runContext?.workflow) + ?.workflow_revision + ) + return; + void resolveInteraction(sessionId, token, () => cred); + }; + const serverPermissions = serverPermissionsFromRequest(request); + // The SAME name->spec index the relay execute loop hands to the relay execution guard, so + // the approval card and the guard cannot disagree about a tool's permission/readOnly. + const specsByName = toolSpecsByName(plan.toolSpecs); + // Build the per-turn permission handler WITHOUT attaching to the live session: the + // session-lifetime `onPermissionRequest` (in acquireEnvironment) routes into it via + // `currentTurn`. A capturing shim reuses attachPermissionResponder unchanged; its + // respondPermission delegates to the real session. + attachPermissionResponder({ + session: { + onPermissionRequest: (handler: (req: unknown) => void) => { + turn.onPermissionRequest = handler; + }, + respondPermission: (id: string, reply: string) => + env.session.respondPermission(id, reply), + }, + run, + responder, + latch, + serverPermissions, + log: logger, + onPause: () => pause.pause(), + onPausedToolCall: (id) => pause.markPausedToolCall(id), + onCreateInteraction: recordPendingInteraction, + onResolveInteraction: resolveInteractionToken, + toolSpecsByName: specsByName, + // Pi runs only: presence of the specs map turns Pi gate envelope detection on AND is how + // the runner recovers specPermission/readOnlyHint (the envelope carries identity, never + // policy). Absent for Claude, so a title collision there keeps the base path. + piToolSpecsByName: plan.isPi + ? new Map( + plan.toolSpecs.map((spec) => [ + spec.name, + { + permission: spec.permission, + readOnly: spec.readOnly, + // callRef tools only: bound paths are runner-filled at execution, so the + // approval card and decision keys must not carry the model's values for them. + contextBindings: spec.callRef + ? spec.contextBindings + : undefined, + }, + ]), + ) + : undefined, + // A resolved custom-tool allow becomes an execution grant the relay guard consumes, so + // only a dialog-approved (or policy-allowed) call ever executes from the relay dir. + onPiGateAllowed: (info) => + executionGrants.grant(info.toolName, info.args), + // Record the parkable permission gate (only in keep-alive park mode) so the dispatch can + // resume it live. Fires per pending gate (before the latch) so a parallel gate is counted; + // the single-gate resume records only the FIRST gate's answer target. `info.gateType` names + // the plane (Claude ACP vs Pi ACP) so the resume answers on the right one. + onUserApprovalGate: opts.approvalParkMode + ? (info) => { + env.approvalGateCount += 1; + if ( + env.approvalGateCount === 1 && + info.permissionId && + info.toolCallId + ) { + env.parkedApproval = { + gateType: info.gateType, + permissionId: info.permissionId, + toolCallId: info.toolCallId, + toolName: info.toolName, + args: info.args, + interactionToken: info.interactionToken, + }; + } + } + : undefined, + }); + + // Resolve the ONE client-tool seam both delivery paths share. The correlation index is wired + // for Claude only — Pi's relay toolCallId is already exact. + env.clientToolRelayRef.current = buildClientToolRelay({ + responder, + run, + latch, + pause, + recordPendingInteraction, + toolCallIndex: plan.isPi ? undefined : env.toolCallIndex, + log: logger, + }); + + // EVERY harness gets the guard: the relay dir is sandbox-writable, so a forged + // `.req.json` proves nothing about any dialog having run, and this runner-side + // re-check is the only enforcement of the hard deny boundary against forged files. + // `allow` passes and `deny` refuses identically everywhere; `ask` splits by harness — + // Pi consumes a dialog-recorded execution grant (fail-closed parity with the in-sandbox + // confirm), while a non-Pi MCP harness (Claude) passes `ask` because its own harness + // enforces the ask dialog (the rendered `mcp__agenta-tools__` ask rules + the ACP + // permission flow) before a call reaches the shim. See buildRelayExecutionGuard for the + // stated residual (a forged file can still trigger an ask-tool without a dialog there). + const relayGuard: RelayExecutionGuard = buildRelayExecutionGuard({ + isPi: plan.isPi, + permissionPlan, + executionGrants, + }); + + if (plan.useToolRelay) { + turn.toolRelay = (deps.startToolRelay ?? startToolRelay)( + plan.isDaytona + ? (deps.sandboxRelayHost ?? sandboxRelayHost)(env.sandbox, { + log: logger, + }) + : (deps.localRelayHost ?? localRelayHost)(), + plan.relayDir, + plan.toolSpecs, + request.toolCallback as ToolCallbackContext | undefined, + request.runContext, + env.clientToolRelayRef.current, + relayGuard, + { log: logger }, + ); + // Ordering invariant: the relay's stale-file sweep must complete before the + // resume's respondPermission or the fresh prompt below can cause a legitimate + // request, so nothing legitimate can predate the sweep and be swallowed as + // stale. Optional-chained so a fake relay without `ready` is tolerated, and a + // sweep failure never kills the turn. + await turn.toolRelay?.ready?.catch?.(() => {}); + } + + // The prompt promise this turn races against the pause signal. A normal/continuation turn + // sends a fresh prompt; a live approval resume answers the parked gate on the SAME session and + // continues the ORIGINAL, still-pending prompt promise (the tool then runs with its original + // byte-exact args). Either way, on a HITL pause the prompt resolves cancelled or never + // resolves, and the pause signal ends the turn. + let promptPromise: Promise; + if (opts.resume) { + // The new (resume) turn owns streaming + tracing; the environment is already wired to route + // continued events into this turn's sink (env.currentTurn was set above). Seed this run's + // trace with the parked tool call so the completing `tool_call_update` closes it and the FE + // approval part flips to output-available even if the adapter re-announces nothing. Then + // answer the gate on the live session — the original prompt continues from here. + run.handleUpdate({ + sessionUpdate: "tool_call", + toolCallId: opts.resume.toolCallId, + title: opts.resume.toolName, + kind: opts.resume.toolName, + rawInput: opts.resume.args, + }); + promptPromise = Promise.resolve(opts.resume.promptPromise); + promptPromise.catch(() => {}); + // A parked Pi dialog gate resumes on a FRESH turn whose relay and grant ledger are new; + // grant the approved call here so the extension's execute record (written right after the + // confirm resolves) passes the relay guard. Claude resumes grant too — harmlessly, no + // guard consults it. + if (opts.resume.reply === "once") { + executionGrants.grant(opts.resume.toolName, opts.resume.args); + } + await env.session.respondPermission( + opts.resume.permissionId, + opts.resume.reply, + ); + // The gate is answered: resolve the durable interaction row (the parked pending row the cold + // path would otherwise resolve via its decision map). The fresh per-turn pause controller + // starts with an EMPTY pausedToolCallIds set, so the resumed call's `tool_call_update` frames + // are no longer suppressed and stream through — the "clear pausedToolCallIds on resume" step. + resolveInteractionToken(opts.resume.interactionToken); + logger( + `[keepalive] resume answered gate reply=${opts.resume.reply} tool=${opts.resume.toolName ?? "?"}`, + ); + } else { + promptPromise = Promise.resolve( + env.session.prompt([{ type: "text", text: turnText }]), + ); + promptPromise.catch(() => {}); + } + const raced = await Promise.race([ + promptPromise, + pause.signal.then(() => PAUSED), + runLimitTripped.then(() => RUN_LIMIT_TRIPPED), + ]); + // A tripped run-limit ends the turn as an error: throw into the shared catch below so the + // trace is flushed and the caller's teardown reclaims the (wedged) sandbox. + if (raced === RUN_LIMIT_TRIPPED) { + throw new Error(runLimitReason ?? "run limit tripped"); + } + const stopReason = + raced === PAUSED || pause.active ? "paused" : (raced as any)?.stopReason; + // Pause notification is immediate, but terminalization must wait for managed cancellation + // and already-queued ACP updates. Re-sweep after the drain so a sibling announced during + // cancellation receives exactly one deterministic terminal result before `done`. + if (stopReason === "paused") { + await pause.waitForEventDrain(); + run.settleOpenToolCalls( + (id) => pause.isPausedToolCall(id), + TOOL_NOT_EXECUTED_PAUSED, + ); + } + const result = raced === PAUSED ? undefined : raced; + // A parkable pause this turn: hand the still-pending prompt promise to the parked record so a + // later resume can await the same continuation. (Set after the race so `promptPromise` exists. + // The read is asserted because the onUserApprovalGate callback set the field via an async + // mutation TS's flow analysis cannot see, so it would otherwise narrow the reset to `never`.) + const parkedThisTurn = env.parkedApproval as ParkedApproval | undefined; + if (opts.approvalParkMode && pause.active && parkedThisTurn) { + parkedThisTurn.promptPromise = promptPromise; + } + await turn.toolRelay?.stop(); + logger(`prompt stopReason=${stopReason}`); + + const usage = await resolveRunUsage({ + sandbox: env.sandbox, + usageOutPath: plan.usageOutPath, + isDaytona: plan.isDaytona, + promptResult: result, + streamUsage: run.usage(), + }); + run.setUsage(usage); + + const swallowedPiError = + plan.isPi && + !plan.isDaytona && + !run.output().trim() && + !run.events().some((e) => e.type === "tool_call") + ? // The helper derives the transcript location from `piSessionWorkspaceDir(plan.cwd)`, + // the same shared helper `configurePiSessionWorkspace` used to point Pi at it. + findSwallowedPiError(plan.cwd) + : undefined; + let swallowedError: string | undefined; + if (swallowedPiError) { + swallowedError = conciseError( + new Error(swallowedPiError), + plan.harness, + request.provider, + ); + run.recordError(swallowedError, request.provider); + run.emitEvent({ type: "error", message: swallowedError }); + } + + const output = run.finish(); + await run.flush(); + + if (swallowedError) { + // A failed turn may have left a partial turn in the native transcript: the prior record + // is no longer a faithful resume point. + invalidateContinuity(sessionId, plan.harness, deps); + return { ok: false, error: swallowedError }; + } + + // Capture this harness's native session id for the next turn's setup. Only on a turn that + // actually completed (not paused mid-turn — a park has not finished authoring the turn, so + // it must not be marked authoritative) and only when the harness surfaced one. + if ( + stopReason !== "paused" && + env.continuityTurnIndex !== undefined && + sessionId && + env.session?.agentSessionId + ) { + (deps.sessionContinuityStore ?? sessionContinuityStore).record( + sessionId, + plan.harness, + env.session.agentSessionId, + env.continuityTurnIndex, + ); + // Mirror the record durably so it survives a runner restart; fire-and-forget. + const syncCred = runCredential(request); + if (syncCred) { + void (deps.syncHarnessSessionDurable ?? syncHarnessSessionDurable)( + sessionId, + plan.harness, + env.session.agentSessionId, + env.continuityTurnIndex, + { authorization: syncCred, log: logger }, + ); + } + } else if (stopReason === "paused") { + // A pause stopped mid-turn, after the harness may have written a partial turn natively. + invalidateContinuity(sessionId, plan.harness, deps); + } + + return { + ok: true, + output, + messages: output ? [{ role: "assistant", content: output }] : [], + events: emit ? [] : run.events(), + usage, + stopReason, + capabilities: { + ...env.capabilities, + streamingDeltas: !!emit && env.capabilities.streamingDeltas, + }, + sessionId, + model: env.model ?? request.model, + traceId: run.traceId(), + } as AgentRunResult; + } catch (err) { + const error = conciseError(err, plan.harness, request.provider); + otel?.recordError(error, request.provider); + otel?.emitEvent({ type: "error", message: error }); + // An aborted turn may have left a partial turn in the native transcript. + invalidateContinuity(sessionId, plan.harness, deps); + // finish() must not throw uncaught — tracing must not mask the run error. + try { + otel?.finish(); + } catch {} + await otel?.flush().catch(() => {}); + return { ok: false, error }; + } finally { + // Release every run-limits timer (idempotent, never re-arms on a late event) on EVERY path. + runLimits.dispose(); + // This turn owns its relay: stop it on EVERY exit path (the happy path already stopped it + // after the prompt; stop is safe to repeat, matching the old finally). Null it afterwards so + // a later `destroy()` — possibly after the dispatch cleared the sink — cannot double-stop or + // orphan it. + await activeTurn?.toolRelay?.stop().catch(() => {}); + if (activeTurn) activeTurn.toolRelay = undefined; + } +} diff --git a/services/runner/src/engines/sandbox_agent/runtime-contracts.ts b/services/runner/src/engines/sandbox_agent/runtime-contracts.ts new file mode 100644 index 0000000000..01e82d9b1e --- /dev/null +++ b/services/runner/src/engines/sandbox_agent/runtime-contracts.ts @@ -0,0 +1,246 @@ +import { InMemorySessionPersistDriver, SandboxAgent } from "sandbox-agent"; + +import { type AgentRunRequest, type HarnessCapabilities } from "../../protocol.ts"; +import { type Responder } from "../../responder.ts"; +import type { ClientToolRelay } from "../../tools/client-tool-relay.ts"; +import { localRelayHost, sandboxRelayHost, startToolRelay } from "../../tools/relay.ts"; +import { createSandboxAgentOtel } from "../../tracing/otel.ts"; +import { createAcpFetch } from "./acp-fetch.ts"; +import { type ParkedApprovalGateType } from "./acp-interactions.ts"; +import { signAgentMountCredentials } from "./agent-mount.ts"; +import { probeCapabilities } from "./capabilities.ts"; +import { createToolCallCorrelationIndex } from "./client-tools.ts"; +import { buildDaemonEnv, resolveDaemonBinary } from "./daemon.ts"; +import { createCookieFetch, prepareDaytonaPiAssets } from "./daytona.ts"; +import { applyModel } from "./model.ts"; +import { discoverTunnelEndpoint, mountHarnessSessionDirs, mountStorage, mountStorageRemote, signSessionMountCredentials, unmountStorage, type MountCredentials } from "./mount.ts"; +import { PendingApprovalPauseController } from "./pause.ts"; +import { buildSandboxProvider } from "./provider.ts"; +import { createRunLimits, resolveRunLimits } from "./run-limits.ts"; +import { type BuildRunPlanDeps, type RunPlan } from "./run-plan.ts"; +import { clearSandboxPointer, readStoredSandboxPointer, writeSandboxPointer } from "./sandbox-reconnect.ts"; +import { hydrateHarnessSessionFromDurable, syncHarnessSessionDurable } from "./session-continuity-durable.ts"; +import { type SessionContinuityStore } from "./session-continuity.ts"; +import { type TeardownReason } from "./teardown.ts"; +import { uploadToolMcpAssets } from "./tool-mcp-assets.ts"; +import { prepareWorkspace } from "./workspace.ts"; + +type Log = (message: string) => void; + +export interface SandboxAgentDeps extends BuildRunPlanDeps { + startSandboxAgent?: typeof SandboxAgent.start; + createPersist?: () => InMemorySessionPersistDriver; + createOtel?: typeof createSandboxAgentOtel; + buildDaemonEnv?: typeof buildDaemonEnv; + resolveDaemonBinary?: typeof resolveDaemonBinary; + buildSandboxProvider?: typeof buildSandboxProvider; + createCookieFetch?: typeof createCookieFetch; + createAcpFetch?: typeof createAcpFetch; + prepareWorkspace?: typeof prepareWorkspace; + prepareDaytonaPiAssets?: typeof prepareDaytonaPiAssets; + uploadToolMcpAssets?: typeof uploadToolMcpAssets; + probeCapabilities?: typeof probeCapabilities; + applyModel?: typeof applyModel; + startToolRelay?: typeof startToolRelay; + localRelayHost?: typeof localRelayHost; + sandboxRelayHost?: typeof sandboxRelayHost; + signSessionMountCredentials?: typeof signSessionMountCredentials; + signAgentMountCredentials?: typeof signAgentMountCredentials; + mountStorage?: typeof mountStorage; + mountStorageRemote?: typeof mountStorageRemote; + unmountStorage?: typeof unmountStorage; + discoverTunnelEndpoint?: typeof discoverTunnelEndpoint; + /** Per-harness transcript mounts (remote only; see mount.ts). */ + mountHarnessSessionDirs?: typeof mountHarnessSessionDirs; + responderFactory?: (request: AgentRunRequest) => Responder; + resolveRunLimits?: typeof resolveRunLimits; + createRunLimits?: typeof createRunLimits; + /** Session-continuity store override (tests inject their own; default is the process singleton). */ + sessionContinuityStore?: SessionContinuityStore; + /** Durable read-back/write-forward of the continuity store (tests inject fakes). */ + hydrateHarnessSessionFromDurable?: typeof hydrateHarnessSessionFromDurable; + syncHarnessSessionDurable?: typeof syncHarnessSessionDurable; + /** Durable read/write of the sandbox pointer, for the remote reconnect ladder. */ + readStoredSandboxPointer?: typeof readStoredSandboxPointer; + clearSandboxPointer?: typeof clearSandboxPointer; + writeSandboxPointer?: typeof writeSandboxPointer; + /** + * Resolve `{replicaId, ownerReplicaId}` for a session-owned local-sandbox run, so + * `acquireEnvironment` can fail loudly instead of silently cold-starting on a non-owner + * replica. The default claims the `owner` affinity key via the coordination plane and reads + * back the actual owner (`claimSessionOwnership`); tests inject their own. `authorization` is + * the run credential (the claim authenticates as the invoke caller). + */ + resolveLocalRunnerOwner?: ( + sessionId: string, + authorization: string, + ) => Promise<{ replicaId: string; ownerReplicaId: string | undefined }>; + log?: Log; +} + +/** + * Race sentinel: a run-limits deadline (total/idle/TTFB/per-tool-call) tripped mid-turn. Distinct + * from `PAUSED` so the prompt race can tell a human pause (keep the session) from a wedge deadline + * (end the turn as an error, letting the caller's teardown reclaim the sandbox). + */ +export const RUN_LIMIT_TRIPPED = Symbol("run-limit-tripped"); + +/** + * The per-turn sink the session-lifetime listeners demux into. `runTurn` swaps a fresh one in + * at turn start (`env.currentTurn`) and the dispatch clears it at turn end. The `sandbox-agent` + * listener registries are plain Sets — an event with no listener is dropped and a permission + * request with no listener is CANCELLED — so the listeners stay attached for the session's whole + * life and route into whichever turn is active, with no detach/attach window between turns. + */ +export interface CurrentTurn { + run: ReturnType; + pause: PendingApprovalPauseController; + toolRelay?: { ready?: Promise; stop: () => Promise }; + /** Route a session/update for the active turn (suppress + handleUpdate + pause re-sweep). */ + handleUpdate: (update: unknown) => void; + /** Route a permission reverse-RPC for the active turn (built by attachPermissionResponder). */ + onPermissionRequest?: (req: unknown) => void; +} + +/** + * A permission gate that paused the turn and can be answered later on the SAME live session. + * Recorded for a Claude ACP permission gate (keep-alive slice 2) or a Pi ACP permission gate + * (Pi approval parking: the gate rides the extension's `ctx.ui.confirm` onto the same ACP + * permission plane). NOT recorded for a client-tool MCP pause — that cannot be answered across + * a turn boundary and stays on the cold path. Existence of this record is what makes the + * dispatch park a paused session in `awaiting_approval` instead of tearing it down. + */ +export interface ParkedApproval { + /** Which gate paused; the dispatch resumes only a recognized type and treats others as cold. */ + gateType: ParkedApprovalGateType; + /** The ACP permission-request id, answered later via `session.respondPermission`. */ + permissionId: string; + /** The gated tool call's id — matched against the incoming approval envelope's toolCallId. */ + toolCallId: string; + /** The gated tool name (logging + the durable interaction row); never its args, in logs. */ + toolName: string | undefined; + /** The gated call's original args, used to seed the resume turn's trace/egress tool span. */ + args: unknown; + /** The durable interaction row token, resolved on the answer via the onResolveInteraction hook. */ + interactionToken: string; + /** The held original `prompt()` promise; the resume awaits it after `respondPermission`. */ + promptPromise?: Promise; +} + +/** Answer a parked Claude ACP permission gate on the live session (the keep-alive resume input). */ +export interface ResumeApprovalInput { + permissionId: string; + reply: "once" | "reject"; + toolCallId: string; + toolName: string | undefined; + args: unknown; + interactionToken: string; + promptPromise?: Promise; +} + +/** Per-turn options for `runTurn`. Absent (flag off / cold) means today's byte-identical path. */ +export interface RunTurnOptions { + /** A live continuation: send only the new user text instead of the full cold transcript. */ + continuation?: boolean; + /** + * The session was rehydrated via `session/load` (the patched `resumeSession`), so the harness + * already holds the prior turns natively. Like `continuation`, the prompt is only the new user + * text; `buildTurnText` must not run. Distinct field from `continuation` because the two arrive + * through different acquire paths (live pool checkout vs a fresh cold acquire that loaded an + * old session) — `runTurn` treats them identically for the text-selection decision. + */ + loaded?: boolean; + /** + * Keep-alive approval park mode: on a Claude ACP permission gate the pause keeps the session + * alive (no settle/abort/destroy) so a later resume can answer it. A non-parkable pause (Pi + * relay, client tool) still tears down exactly as today, so this is safe to set on any eligible + * keep-alive turn. + */ + approvalParkMode?: boolean; + /** A live approval resume: answer the parked gate and stream the continued prompt's events. */ + resume?: ResumeApprovalInput; +} + +/** + * Send only the new user text (not the full cold transcript) when the harness already holds the + * prior turns: a live continuation, or a session rehydrated via `session/load`. `runTurn` calls + * this, so a test that pins it pins the shipped decision. + */ +export function sendLastMessageOnly(opts: RunTurnOptions): boolean { + return Boolean(opts.continuation || opts.loaded); +} + +/** + * A session-scoped environment that can serve many turns. Everything expensive to build lives + * here (sandbox, session, internal tool-MCP server, mounted cwd, relay/temp dirs); `destroy()` + * is the one complete idempotent teardown the pool, the shutdown handler, and the cold path all + * call. Per-turn state rides `currentTurn`, swapped in by `runTurn`. + */ +export interface SessionEnvironment { + plan: RunPlan; + logger: Log; + deps: SandboxAgentDeps; + sandbox: any; + session: any; + sessionId: string; + model: string | undefined; + capabilities: HarnessCapabilities; + strictModel: boolean; + toolCallIndex: ReturnType; + /** The current turn's client-tool relay, read by the deferred ref baked into the MCP server. */ + clientToolRelayRef: { current?: ClientToolRelay }; + mcpAbort: AbortController; + runAgentDir: string | undefined; + otlpAuthFilePath: string | undefined; + mountCreds: MountCredentials | null; + agentMountCreds?: MountCredentials | null; + /** The mount's owning project id (keep-alive pool key FALLBACK scope, preferred is + * `runContext.project.id`); undefined when there is no mount. */ + mountProjectId?: string; + /** This run's resolved project scope (`projectScopeFor`: run-context preferred, mount + * fallback) — the same scope `poolKeyFor` keys on. Undefined when neither source yields + * one; a scoped `/kill` can then never claim this sandbox (see `destroyInFlightSandboxesForSession`). */ + projectScopeId?: string; + /** This acquire resumed the harness's native session via `session/load` (not cold). */ + loadedFromContinuity: boolean; + /** A remote, session-owned run whose sandbox can be parked (warm) rather than deleted at end. */ + resumable: boolean; + /** The conversation turn index this acquire's continuity record was read/written at. */ + continuityTurnIndex: number | undefined; + // Mutable teardown/turn state shared across acquire, runTurn, and destroy. + sessionDestroyRequested: boolean; + mountedCwd: string | undefined; + agentMountedPath?: string; + durableCwdSafeToDelete: boolean; + workspace: { cleanup: () => Promise } | undefined; + runtimeRemount: Promise | undefined; + closeToolMcp: (() => Promise) | undefined; + currentTurn?: CurrentTurn; + /** + * The unique ACP tool-call ids the LAST completed turn emitted (reset at each turn start). + * The keep-alive dispatch folds them into the expected next-history fingerprint at park time, + * so a tool-using turn still matches its own continuation (the FE keeps assistant tool parts). + */ + lastTurnToolCallIds: string[]; + /** + * The Claude ACP permission gate the LAST turn paused on, or undefined. Set only for a harness + * ACP permission gate, reset at each turn start; the dispatch reads it after a paused turn to + * decide whether to park in `awaiting_approval` and, on the next request, how to resume. + */ + parkedApproval?: ParkedApproval; + /** + * How many Claude ACP permission gates resolved to pendingApproval THIS turn (reset at turn + * start). More than one means parallel gates the single-gate resume cannot answer, so the + * dispatch does not park (tears down cold as today). + */ + approvalGateCount: number; + destroyed: boolean; + /** Complete, idempotent teardown selected from the typed teardown reason. */ + destroy: (opts?: { reason?: TeardownReason }) => Promise; + /** End the active turn: clear the current-turn sink (called before a park). */ + clearTurn: () => void; +} + +export type AcquireEnvironmentResult = + | { ok: true; env: SessionEnvironment } + | { ok: false; error: string }; diff --git a/services/runner/src/engines/sandbox_agent/runtime-policy.ts b/services/runner/src/engines/sandbox_agent/runtime-policy.ts new file mode 100644 index 0000000000..7293324a24 --- /dev/null +++ b/services/runner/src/engines/sandbox_agent/runtime-policy.ts @@ -0,0 +1,167 @@ +import { + type AgentRunRequest, + type ToolPermission, +} from "../../protocol.ts"; +import { claimSessionOwnership, REPLICA_ID } from "../../sessions/alive.ts"; +import { PendingApprovalPauseController } from "./pause.ts"; + +type Log = (message: string) => void; + +/** Extract the run credential from the OTLP export headers (initial value, constant for the run). */ +export function runCredential(request: AgentRunRequest): string { + const headers = (request.telemetry?.exporters?.otlp?.headers ?? {}) as Record< + string, + string + >; + return (headers["authorization"] ?? headers["Authorization"] ?? "").trim(); +} + +export function serverPermissionsFromRequest( + request: AgentRunRequest, +): ReadonlyMap { + const permissions = new Map(); + for (const server of request.mcpServers ?? []) { + if (server.policy?.permission !== undefined) { + permissions.set(server.name, server.policy.permission); + } + } + return permissions; +} + +export function shouldSuppressPausedToolCallUpdate( + update: unknown, + pause: PendingApprovalPauseController, +): boolean { + const frame = update as + | { sessionUpdate?: unknown; toolCallId?: unknown } + | undefined; + const kind = frame?.sessionUpdate; + if (kind !== "tool_call" && kind !== "tool_call_update") return false; + const toolCallId = + typeof frame?.toolCallId === "string" ? frame.toolCallId : undefined; + return pause.isPausedToolCall(toolCallId); +} + +const CLAUDE_STRICT_DEPLOYMENTS = new Set([ + "custom", + "bedrock", + "vertex", + "vertex_ai", +]); + +export function applyClaudeConnectionEnv( + env: Record, + request: AgentRunRequest, + acpAgent: string, + logger: Log, +): void { + if (acpAgent !== "claude") return; + + // Disable the Claude Agent SDK's Tool-Search feature for every Claude run. The bundled + // SDK defaults Tool-Search ON, which makes Claude DEFER the `agenta-tools` MCP tools and + // call them before their `inputSchema` is loaded — so it emits an empty `input: {}` and + // tools-with-args (reference workflows, commit_revision) never receive their arguments. + // Our tool count is small, so deferral buys nothing and only strips the schema. The SDK + // treats only `false`/`0`/`no`/`off` as off, so the string must be "false" (not "0"/"100"). + // This is applied after `buildDaemonEnv`'s clear and is not in `KNOWN_PROVIDER_ENV_VARS`, + // so it is never stripped, and it reaches the Daytona sandbox like `ANTHROPIC_BASE_URL`. + env.ENABLE_TOOL_SEARCH = "false"; + + const deployment = request.deployment; + const selectedModel = request.model; + const baseUrl = request.endpoint?.baseUrl; + if (baseUrl) { + env.ANTHROPIC_BASE_URL = baseUrl; + logger(`claude base_url: ${baseUrl}`); + } + + if (deployment === "bedrock") { + env.CLAUDE_CODE_USE_BEDROCK = "1"; + const region = request.endpoint?.region; + if (region) { + env.AWS_REGION = region; + env.AWS_DEFAULT_REGION ??= region; + } + } else if (deployment === "vertex" || deployment === "vertex_ai") { + env.CLAUDE_CODE_USE_VERTEX = "1"; + } + + if ( + selectedModel && + (baseUrl || (deployment && CLAUDE_STRICT_DEPLOYMENTS.has(deployment))) + ) { + env.ANTHROPIC_MODEL = selectedModel; + env.ANTHROPIC_CUSTOM_MODEL_OPTION = selectedModel; + logger( + `claude model=${selectedModel} deployment=${deployment ?? ""}`, + ); + } +} + +/** + * Whether a requested-but-unsettable model fails the run (F-007). Strict by default on every + * harness path: a user who picks a model either runs that model or sees a loud error, never a + * silent (often pricier) fallback to the harness default. `AGENTA_AGENT_MODEL_STRICT=false` is + * the explicit opt-out that restores the legacy warn-and-fallback behavior. A run that requests + * no model is unaffected either way — it keeps the harness default. + */ +export function modelResolutionStrict(): boolean { + return process.env.AGENTA_AGENT_MODEL_STRICT !== "false"; +} + +export async function defaultResolveLocalRunnerOwner( + sessionId: string, + authorization: string, +): Promise<{ replicaId: string; ownerReplicaId: string | undefined }> { + // No credential ⇒ the claim would 401; treat as "no known owner" (pass), never worse than today. + if (!authorization) { + return { replicaId: REPLICA_ID, ownerReplicaId: undefined }; + } + return claimSessionOwnership(sessionId, authorization); +} + +export function isTransportEndpointDisconnected(err: unknown): boolean { + const message = String(err instanceof Error ? err.message : err); + const code = + typeof err === "object" && err !== null && "code" in err + ? String((err as { code?: unknown }).code) + : ""; + return ( + code === "ENOTCONN" || + message.includes("ENOTCONN") || + message.includes("Transport endpoint is not connected") + ); +} + +export function containsTransportEndpointDisconnected(value: unknown): boolean { + const seen = new Set(); + + const visit = (current: unknown): boolean => { + if (typeof current === "string") { + return isTransportEndpointDisconnected(current); + } + if (current instanceof Error) { + return isTransportEndpointDisconnected(current); + } + if (!current || typeof current !== "object") { + return false; + } + if (seen.has(current)) { + return false; + } + seen.add(current); + + const code = + "code" in current ? String((current as { code?: unknown }).code) : ""; + if (code === "ENOTCONN") { + return true; + } + + if (Array.isArray(current)) { + return current.some(visit); + } + return Object.values(current as Record).some(visit); + }; + + return visit(value); +} diff --git a/services/runner/src/engines/sandbox_agent/session-events.ts b/services/runner/src/engines/sandbox_agent/session-events.ts new file mode 100644 index 0000000000..6c6c66cf5c --- /dev/null +++ b/services/runner/src/engines/sandbox_agent/session-events.ts @@ -0,0 +1,79 @@ +import { conciseError } from "./errors.ts"; +import type { SessionEnvironment } from "./runtime-contracts.ts"; + +/** + * Route one harness event into the active turn's sink. + * + * Data flow: the ACP session emits an event -> we demux it -> the active turn + * (`environment.currentTurn`) consumes the update. The session listener is attached ONCE and + * outlives every turn, so this must never throw: the sandbox-agent registries are plain Sets and a + * thrown handler would corrupt the event stream, so any error is swallowed and logged. + * + * Steps: let the ENOTCONN watcher observe the raw event, extract the update payload (dropping events + * that carry none), record live tool_call ids for client-tool correlation, then hand the update to + * the active turn — or, between turns when no turn owns it, log and drop it. + */ +function routeSessionEventToActiveTurn( + environment: SessionEnvironment, + remountLocalCwdAfterRuntimeEnotconn: (event: unknown) => void, + event: any, +): void { + const { logger, plan } = environment; + try { + remountLocalCwdAfterRuntimeEnotconn(event); + const payload = event?.payload; + const update = payload?.params?.update ?? payload?.update; + if (!update) return; + // Record live ACP tool_call ids so a paused client_tool can correlate to Claude's bubble + // (session-scoped; a lookup CONSUMES its matched id). + environment.toolCallIndex.record(update); + const turn = environment.currentTurn; + if (turn) { + turn.handleUpdate(update); + } else { + // Between turns (parked/idle): no turn owns this event. Log and drop by decision. + logger(`[keepalive] between-turns event dropped`); + } + } catch (err) { + logger(`session onEvent handler error: ${conciseError(err, plan.harness)}`); + } +} + +/** + * Route one permission gate into the active turn's approval handler. + * + * Data flow: the harness raises a permission request -> the active turn (`environment.currentTurn`) + * decides it. Like the event listener this is attached ONCE and must never throw (a thrown handler + * would corrupt the sandbox-agent registries), so errors are swallowed and logged. + * + * Between turns no turn owns the gate. An approval park is always recorded DURING the active turn + * (the gate fires while a prompt runs, routing through currentTurn), and a parked-on-approval + * session leaves its harness suspended on that gate, so nothing new fires while parked. A gate that + * reaches here is therefore a genuine stray (e.g. a late teardown artifact): reject it by policy so + * it cannot hang. + */ +function routePermissionRequestToActiveTurn( + environment: SessionEnvironment, + req: any, +): void { + const { logger, plan } = environment; + try { + const turn = environment.currentTurn; + if (turn?.onPermissionRequest) { + turn.onPermissionRequest(req); + return; + } + logger( + `[keepalive] between-turns permission request, cancelling by policy id=${req?.id}`, + ); + void Promise.resolve( + environment.session?.respondPermission?.(req?.id, "reject"), + ).catch(() => {}); + } catch (err) { + logger( + `session onPermissionRequest handler error: ${conciseError(err, plan.harness)}`, + ); + } +} + +export { routeSessionEventToActiveTurn, routePermissionRequestToActiveTurn }; diff --git a/services/runner/src/engines/sandbox_agent/session-identity.ts b/services/runner/src/engines/sandbox_agent/session-identity.ts new file mode 100644 index 0000000000..27a4a90f39 --- /dev/null +++ b/services/runner/src/engines/sandbox_agent/session-identity.ts @@ -0,0 +1,441 @@ +import { createHash } from "node:crypto"; + +import { + type AgentRunRequest, + type ChatMessage, + type ContentBlock, + messageText, +} from "../../protocol.ts"; +import { approvalDecisionOf } from "../../responder.ts"; +import type { TeardownReason } from "./teardown.ts"; +import { loadRunnerConfig } from "../../config/runner-config.ts"; + +function log(message: string): void { + process.stderr.write(`[keepalive] ${message}\n`); +} + +// --- Config (read once, in one place; mirrors server.ts's env reads) --------- // + +export interface KeepaliveConfig { + enabled: boolean; + ttlMs: number; + approvalTtlMs: number; + poolMax: number; +} + +export type KeepaliveProviderName = "local" | "daytona"; + +const KEEPALIVE_ENV = "AGENTA_RUNNER_SESSION_KEEPALIVE"; +const TTL_ENV = "AGENTA_RUNNER_SESSION_TTL_MS"; +const APPROVAL_TTL_ENV = "AGENTA_RUNNER_SESSION_APPROVAL_TTL_MS"; +const POOL_MAX_ENV = "AGENTA_RUNNER_SESSION_POOL_MAX"; + +const DEFAULT_TTL_MS = 60_000; +const DEFAULT_APPROVAL_TTL_MS = 300_000; +const DEFAULT_POOL_MAX = 8; +const DAYTONA_TTL_ENV = "AGENTA_RUNNER_DAYTONA_SESSION_IDLE_TTL_MS"; +const DAYTONA_POOL_MAX_ENV = "AGENTA_RUNNER_DAYTONA_SESSION_MAX_WARM"; +// Two minutes: the shipping default decided in the plan (about half a cent per parked turn), +// enabled after the E3 live verification. 0 disables keeping Daytona sandboxes running. +const DEFAULT_DAYTONA_TTL_MS = 120_000; +const DEFAULT_DAYTONA_POOL_MAX = 20; + +function positiveIntEnv(name: string, fallback: number): number { + const raw = process.env[name]; + const parsed = raw ? Number(raw) : NaN; + return Number.isFinite(parsed) && parsed > 0 ? Math.floor(parsed) : fallback; +} + +/** + * Like `positiveIntEnv` but zero is a VALID value, not a fallback trigger. The Daytona idle + * TTL uses this because 0 is its documented off switch; with a nonzero shipping default, a + * positive-only parse would silently turn "0" back into the default. + */ +function nonNegativeIntEnv(name: string, fallback: number): number { + const raw = process.env[name]; + if (raw === undefined || raw.trim() === "") return fallback; + const parsed = Number(raw); + return Number.isFinite(parsed) && parsed >= 0 ? Math.floor(parsed) : fallback; +} + +function boolEnv(name: string, fallback: boolean): boolean { + const raw = (process.env[name] ?? "").trim().toLowerCase(); + if (!raw) return fallback; + if (raw === "1" || raw === "true" || raw === "yes" || raw === "on") return true; + if (raw === "0" || raw === "false" || raw === "no" || raw === "off") return false; + return fallback; +} + +/** Read one provider's keep-alive config from the environment. */ +export function readKeepaliveConfig( + provider: KeepaliveProviderName, +): KeepaliveConfig { + if (provider === "daytona") { + const ttlMs = nonNegativeIntEnv(DAYTONA_TTL_ENV, DEFAULT_DAYTONA_TTL_MS); + // Keep this live window comfortably below the signed mount-credential lifetime. The + // existing credential-epoch check evicts to cold when those credentials expire. + return { + enabled: ttlMs > 0, + ttlMs, + // Pending approvals on Daytona take the cold path until the F-018 gate plan lands; the + // pool never sees an awaiting_approval park for Daytona today because parkedApproval is + // only set by ACP gates. + approvalTtlMs: ttlMs, + // This budgets billed compute (idle warm sandboxes), deliberately separate from the local + // pool's host-memory budget; Slice 4 adds the strict warm-slot accounting semantics. + poolMax: positiveIntEnv( + DAYTONA_POOL_MAX_ENV, + DEFAULT_DAYTONA_POOL_MAX, + ), + }; + } + return { + enabled: boolEnv(KEEPALIVE_ENV, true), + ttlMs: positiveIntEnv(TTL_ENV, DEFAULT_TTL_MS), + approvalTtlMs: positiveIntEnv(APPROVAL_TTL_ENV, DEFAULT_APPROVAL_TTL_MS), + poolMax: positiveIntEnv(POOL_MAX_ENV, DEFAULT_POOL_MAX), + }; +} + +/** + * `poolMax` (and the LRU/TTL eviction it drives) is a LOCAL-provider parameter — "how many + * ~300 MB hot Claude trees fit on this runner host" — never a global one. Mirrors `run-plan.ts`'s + * own sandbox-id resolution (`request.sandbox || configured default provider`). The pool dispatch + * (`server.ts` `isLocalSandbox`) and the continuity module's own local/remote framing both + * resolve through this one function, so the "local-only" invariant has a single source of truth. + */ +export function resolvesToLocalProvider( + requestSandbox: string | undefined, + defaultProvider: string = loadRunnerConfig().providers.default, +): boolean { + return (requestSandbox || defaultProvider) === "local"; +} + +// --- Fingerprints and the pool key ------------------------------------------ // + +function sha256(value: string): string { + return createHash("sha256").update(value).digest("hex"); +} + +/** Deterministic JSON: object keys sorted recursively so equal values hash equal. */ +function canonicalJson(value: unknown): string { + if (value === null || typeof value !== "object") return JSON.stringify(value); + if (Array.isArray(value)) { + return `[${value.map(canonicalJson).join(",")}]`; + } + const entries = Object.entries(value as Record) + .filter(([, v]) => v !== undefined) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)); + return `{${entries + .map(([k, v]) => `${JSON.stringify(k)}:${canonicalJson(v)}`) + .join(",")}}`; +} + +/** + * A canonical hash over the config-bearing request fields (the continuation-versus-cold + * decision). Per-turn volatiles are excluded: `messages`, `turnId`, trace propagation + * (`context`), the rotating telemetry headers, and secret VALUES (`secrets` — the credential + * epoch covers rotation, and values must never enter any hash used for logging). The + * tool-callback ENDPOINT is included (routing config); its authorization is a credential and + * lives in the credential epoch instead. + */ +export function configFingerprint(request: AgentRunRequest): string { + const workflow = request.runContext?.workflow; + const shape = { + harness: request.harness ?? null, + sandbox: request.sandbox ?? null, + model: request.model ?? null, + provider: request.provider ?? null, + connection: request.connection ?? null, + deployment: request.deployment ?? null, + endpoint: request.endpoint ?? null, + credentialMode: request.credentialMode ?? null, + agentsMd: request.agentsMd ?? null, + systemPrompt: request.systemPrompt ?? null, + appendSystemPrompt: request.appendSystemPrompt ?? null, + tools: request.tools ?? null, + skills: request.skills ?? null, + customTools: request.customTools ?? null, + mcpServers: request.mcpServers ?? null, + toolCallbackEndpoint: request.toolCallback?.endpoint ?? null, + permissions: request.permissions ?? null, + sandboxPermission: request.sandboxPermission ?? null, + harnessFiles: request.harnessFiles ?? null, + workflowRevision: workflow?.revision + ? { + id: workflow.revision.id ?? null, + version: workflow.revision.version ?? null, + } + : null, + isDraft: workflow?.is_draft ?? null, + }; + return sha256(canonicalJson(shape)); +} + +function collectToolCallIds( + content: string | ContentBlock[] | undefined, + into: string[], + seen: Set, +): void { + if (!Array.isArray(content)) return; + for (const block of content) { + if (!block) continue; + if ( + (block.type === "tool_call" || block.type === "tool_result") && + typeof block.toolCallId === "string" && + block.toolCallId && + !seen.has(block.toolCallId) + ) { + seen.add(block.toolCallId); + into.push(block.toolCallId); + } + } +} + +/** + * A hash over the conversation the server received (the FE's pruned array): the ordered user + * message texts, the ordered tool-call ids across every message, and the user-message count. + * Assistant TEXT is deliberately ignored, so a live session that has already answered a plain + * user turn matches the next request's prefix (the FE's assistant turn contributes nothing). + * Tool-call ids ARE included, so an edited history trips a mismatch and degrades to cold + * replay rather than continuing wrongly. Ids are DEDUPED (unique, first-seen order): a resolved + * tool call rides the wire as a `tool_call` block PLUS a `tool_result` block sharing one id + * (vercel `messages.py` `_tool_part_blocks`), and the park-time prediction + * (`expectedNextHistoryFingerprint`) folds each emitted id in once — dedupe makes the two agree + * while a genuinely different id SET still mismatches. + * + * The dispatch stores the fingerprint the next request is EXPECTED to hash to (see + * `expectedNextHistoryFingerprint`), and checks the next request against the fingerprint of its + * PRIOR messages (everything before the new user tail), so a plain conversational continuation + * matches and any divergence falls to cold. + */ +export function historyFingerprint(messages: readonly ChatMessage[]): string { + const userTexts: string[] = []; + const toolCallIds: string[] = []; + const seenIds = new Set(); + let promptCount = 0; + for (const message of messages) { + if (message.role === "user") { + promptCount += 1; + userTexts.push(messageText(message.content)); + } + collectToolCallIds(message.content, toolCallIds, seenIds); + } + return sha256(canonicalJson({ userTexts, toolCallIds, promptCount })); +} + +/** + * The fingerprint a park should record so the NEXT request's prior conversation matches it: + * the full messages this turn ran, plus the tool-call ids the turn itself emitted, folded in + * as one synthetic trailing assistant message. + * + * Why: the FE keeps an assistant turn iff it has an answer part (`agentRequest.ts` + * `isAnswerPart`: non-empty text, `tool-*`/`dynamic-tool`, or file). So a tool-calling turn's + * ids ALWAYS appear in the next request's prior messages, and a fully empty assistant turn is + * pruned but contributes neither text nor ids — the prediction is deterministic either way + * (assistant text is not hashed). An id divergence still trips a mismatch and falls to cold. + */ +export function expectedNextHistoryFingerprint( + messages: readonly ChatMessage[], + emittedToolCallIds: readonly string[], +): string { + if (emittedToolCallIds.length === 0) return historyFingerprint(messages); + const syntheticAssistantTurn: ChatMessage = { + role: "assistant", + content: emittedToolCallIds.map((id) => ({ + type: "tool_call", + toolCallId: id, + })), + }; + return historyFingerprint([...messages, syntheticAssistantTurn]); +} + +/** + * The prior conversation for a continuation check: everything before the request's new user + * tail. Mirrors `transcript.priorMessages` for the trailing-user case (the playground always + * sends the new turn as the last user message), without importing that do-not-touch module. + */ +export function priorConversation(request: AgentRunRequest): ChatMessage[] { + const messages = request.messages ?? []; + if (messages.length && messages[messages.length - 1].role === "user") { + return messages.slice(0, -1); + } + return messages.slice(); +} + +/** + * The approval decision (allow/deny) the incoming request carries for a specific parked gate's + * tool-call id, or undefined when the request has no approval envelope for that id. Reuses the + * cold path's `approvalDecisionOf` (responder.ts) to parse the `{approved}` envelope, and matches + * strictly by toolCallId (the parked gate's id) — never by name+args — so a live resume answers + * exactly the gate that parked. An incoming reply for a different id, or a plain user message, + * yields undefined and the dispatch degrades to cold. + */ +export function approvalDecisionForToolCall( + request: AgentRunRequest, + toolCallId: string, +): "allow" | "deny" | undefined { + if (!toolCallId) return undefined; + for (const message of request.messages ?? []) { + const content = message?.content; + if (!Array.isArray(content)) continue; + for (const block of content) { + if (block?.type !== "tool_result" || block.toolCallId !== toolCallId) { + continue; + } + const decision = approvalDecisionOf(block); + if (decision !== undefined) return decision; + } + } + return undefined; +} + +/** + * True when the request's tail is a fresh user message with text and NOT an approval envelope. + * A continuation only takes the live path for a plain new user turn; an approval reply (a + * trailing tool-role message, or a user turn carrying a tool_result) stays cold here. + */ +export function tailIsFreshUserMessage(request: AgentRunRequest): boolean { + const messages = request.messages ?? []; + const tail = messages[messages.length - 1]; + if (!tail || tail.role !== "user") return false; + if (!messageText(tail.content).trim()) return false; + if (Array.isArray(tail.content)) { + const carriesToolTurn = tail.content.some( + (block) => block?.type === "tool_result" || block?.type === "tool_call", + ); + if (carriesToolTurn) return false; + } + return true; +} + +/** + * The credential epoch bounds how long a parked session may reuse its baked credentials. It is + * a PROCESS-LOCAL hash over the actual resolved secret VALUES (held only in runner memory — + * never logged, persisted, or emitted), combined with the mount credential expiry. A rotated + * same-slug secret changes the hash; an elapsed expiry invalidates the epoch. Either way the + * dispatch evicts and cold-starts with fresh credentials. + * + * The tool-callback bearer is deliberately EXCLUDED: it is per-turn material the backend + * re-mints on its auth-cache cadence (~60s), and every turn — continuation included — starts + * its tool relay from the INCOMING request's `toolCallback`, so the parked copy is never used + * to execute anything. Hashing it made warm sessions evict as "credentials-rotated" on every + * cache rollover for no protective value. Only material actually BAKED into the parked + * environment (the sandbox env secrets) belongs in the hash; the mount expiry bounds the rest. + */ +export interface CredentialEpoch { + /** sha256 over canonical(secrets). In-memory only; never surfaced. */ + secretsHash: string; + /** Mount credential expiry as epoch millis, or undefined when the sign response had none. */ + mountExpiresAtMs?: number; +} + +export function computeCredentialEpoch( + request: AgentRunRequest, + mountExpiresAt?: string, +): CredentialEpoch { + const material = canonicalJson({ + secrets: request.secrets ?? {}, + }); + const parsed = mountExpiresAt ? Date.parse(mountExpiresAt) : NaN; + return { + secretsHash: sha256(material), + mountExpiresAtMs: Number.isFinite(parsed) ? parsed : undefined, + }; +} + +/** + * Whether a parked session's MOUNT credentials have already expired, ignoring the secret material + * hash entirely. This answers only "can the parked environment still write its durable cwd?". + * + * The approval-resume path uses this instead of `credentialEpochValid`: a resume must NOT require + * the resume request's re-minted credentials to MATCH the parked ones (a fresh /run mints fresh + * short-lived material every time, so they practically never match), but an expired mount means + * the parked cwd can no longer be written, so it must still evict to cold. + */ +export function mountCredentialsExpired( + epoch: CredentialEpoch, + now = Date.now(), +): boolean { + return epoch.mountExpiresAtMs !== undefined && now >= epoch.mountExpiresAtMs; +} + +/** + * Why a parked epoch is no longer usable for an incoming request's epoch, or undefined when it + * still is. The two failure modes are distinguished so diagnosis works from logs: + * - `credentials-expired` — the mount credential's lifetime elapsed (time bound). + * - `credentials-rotated` — the resolved secret material changed (a rotated same-slug secret). + */ +export function credentialEpochMismatch( + parked: CredentialEpoch, + incoming: CredentialEpoch, + now = Date.now(), +): "credentials-expired" | "credentials-rotated" | undefined { + if (mountCredentialsExpired(parked, now)) return "credentials-expired"; + if (parked.secretsHash !== incoming.secretsHash) return "credentials-rotated"; + return undefined; +} + +/** + * Whether a parked epoch is still valid for an incoming request's epoch. Invalid (evict, cold) + * when the mount credential expired, or the resolved secret material changed. Thin + * wrapper over `credentialEpochMismatch` for callers that only need the boolean. + */ +export function credentialEpochValid( + parked: CredentialEpoch, + incoming: CredentialEpoch, + now = Date.now(), +): boolean { + return credentialEpochMismatch(parked, incoming, now) === undefined; +} + +/** Which project-scope source produced a pool key: the service-stamped run context, or the mount. */ +export type PoolScopeSource = "run-context" | "mount"; + +/** A pool key plus the scope source that produced it (for the greppable `[keepalive] scope=` log). */ +export interface PoolScope { + key: string; + source: PoolScopeSource; +} + +/** + * The project scope for a run: PREFERRED from the run context the service stamps server-side + * (`runContext.project.id`), FALLING BACK to the mount's owning project id when the run context + * carries none. The run-context id is the trustworthy source: the service derives it from its own + * request state (never from a caller-supplied wire field), so it does not depend on a durable + * mount existing. The mount scope stays as the fallback for the transition and for runs without a + * stamped project. Returns undefined when NEITHER source yields a scope. + * + * This is the single precedence rule other project-scoped decisions (the pool key, the in-flight + * sandbox kill filter) must reuse rather than re-deriving, so they agree by construction. + */ +export function projectScopeFor( + request: Pick, + mountProjectId: string | undefined, +): { id: string; source: PoolScopeSource } | undefined { + const runContextProject = request.runContext?.project?.id?.trim(); + if (runContextProject) return { id: runContextProject, source: "run-context" }; + const mount = mountProjectId?.trim(); + if (mount) return { id: mount, source: "mount" }; + return undefined; +} + +/** + * The pool key: `:`. Provider separation does not need another key segment: + * providers have separate pools, and `configFingerprint` includes `request.sandbox`. + * + * Returns null when there is no session id, or when `projectScopeFor` yields no project scope — + * such a request MUST NOT park (there is no safe key that separates callers), and the dispatch + * runs it fully cold. This no-scope-no-park rule is the keep-alive safety invariant and is + * unchanged. + */ +export function poolKeyFor( + request: AgentRunRequest, + mountProjectId: string | undefined, +): PoolScope | null { + const sessionId = request.sessionId?.trim(); + if (!sessionId) return null; + const scope = projectScopeFor(request, mountProjectId); + if (!scope) return null; + return { key: `${scope.id}:${sessionId}`, source: scope.source }; +} diff --git a/services/runner/src/engines/sandbox_agent/session-pool.ts b/services/runner/src/engines/sandbox_agent/session-pool.ts index ea138b5d18..3aee84efaf 100644 --- a/services/runner/src/engines/sandbox_agent/session-pool.ts +++ b/services/runner/src/engines/sandbox_agent/session-pool.ts @@ -12,448 +12,13 @@ * never imports the engine, so it stays a pure map + timer + policy unit. Operators can disable * it explicitly with `AGENTA_RUNNER_SESSION_KEEPALIVE=off`. */ -import { createHash } from "node:crypto"; - -import { - type AgentRunRequest, - type ChatMessage, - type ContentBlock, - messageText, -} from "../../protocol.ts"; -import { approvalDecisionOf } from "../../responder.ts"; +import type { CredentialEpoch, KeepaliveConfig } from "./session-identity.ts"; import type { TeardownReason } from "./teardown.ts"; -import { loadRunnerConfig } from "../../config/runner-config.ts"; function log(message: string): void { process.stderr.write(`[keepalive] ${message}\n`); } -// --- Config (read once, in one place; mirrors server.ts's env reads) --------- // - -export interface KeepaliveConfig { - enabled: boolean; - ttlMs: number; - approvalTtlMs: number; - poolMax: number; -} - -export type KeepaliveProviderName = "local" | "daytona"; - -const KEEPALIVE_ENV = "AGENTA_RUNNER_SESSION_KEEPALIVE"; -const TTL_ENV = "AGENTA_RUNNER_SESSION_TTL_MS"; -const APPROVAL_TTL_ENV = "AGENTA_RUNNER_SESSION_APPROVAL_TTL_MS"; -const POOL_MAX_ENV = "AGENTA_RUNNER_SESSION_POOL_MAX"; - -const DEFAULT_TTL_MS = 60_000; -const DEFAULT_APPROVAL_TTL_MS = 300_000; -const DEFAULT_POOL_MAX = 8; -const DAYTONA_TTL_ENV = "AGENTA_RUNNER_DAYTONA_SESSION_IDLE_TTL_MS"; -const DAYTONA_POOL_MAX_ENV = "AGENTA_RUNNER_DAYTONA_SESSION_MAX_WARM"; -// Two minutes: the shipping default decided in the plan (about half a cent per parked turn), -// enabled after the E3 live verification. 0 disables keeping Daytona sandboxes running. -const DEFAULT_DAYTONA_TTL_MS = 120_000; -const DEFAULT_DAYTONA_POOL_MAX = 20; - -function positiveIntEnv(name: string, fallback: number): number { - const raw = process.env[name]; - const parsed = raw ? Number(raw) : NaN; - return Number.isFinite(parsed) && parsed > 0 ? Math.floor(parsed) : fallback; -} - -/** - * Like `positiveIntEnv` but zero is a VALID value, not a fallback trigger. The Daytona idle - * TTL uses this because 0 is its documented off switch; with a nonzero shipping default, a - * positive-only parse would silently turn "0" back into the default. - */ -function nonNegativeIntEnv(name: string, fallback: number): number { - const raw = process.env[name]; - if (raw === undefined || raw.trim() === "") return fallback; - const parsed = Number(raw); - return Number.isFinite(parsed) && parsed >= 0 ? Math.floor(parsed) : fallback; -} - -function boolEnv(name: string, fallback: boolean): boolean { - const raw = (process.env[name] ?? "").trim().toLowerCase(); - if (!raw) return fallback; - if (raw === "1" || raw === "true" || raw === "yes" || raw === "on") return true; - if (raw === "0" || raw === "false" || raw === "no" || raw === "off") return false; - return fallback; -} - -/** Read one provider's keep-alive config from the environment. */ -export function readKeepaliveConfig( - provider: KeepaliveProviderName, -): KeepaliveConfig { - if (provider === "daytona") { - const ttlMs = nonNegativeIntEnv(DAYTONA_TTL_ENV, DEFAULT_DAYTONA_TTL_MS); - // Keep this live window comfortably below the signed mount-credential lifetime. The - // existing credential-epoch check evicts to cold when those credentials expire. - return { - enabled: ttlMs > 0, - ttlMs, - // Pending approvals on Daytona take the cold path until the F-018 gate plan lands; the - // pool never sees an awaiting_approval park for Daytona today because parkedApproval is - // only set by ACP gates. - approvalTtlMs: ttlMs, - // This budgets billed compute (idle warm sandboxes), deliberately separate from the local - // pool's host-memory budget; Slice 4 adds the strict warm-slot accounting semantics. - poolMax: positiveIntEnv( - DAYTONA_POOL_MAX_ENV, - DEFAULT_DAYTONA_POOL_MAX, - ), - }; - } - return { - enabled: boolEnv(KEEPALIVE_ENV, true), - ttlMs: positiveIntEnv(TTL_ENV, DEFAULT_TTL_MS), - approvalTtlMs: positiveIntEnv(APPROVAL_TTL_ENV, DEFAULT_APPROVAL_TTL_MS), - poolMax: positiveIntEnv(POOL_MAX_ENV, DEFAULT_POOL_MAX), - }; -} - -/** - * `poolMax` (and the LRU/TTL eviction it drives) is a LOCAL-provider parameter — "how many - * ~300 MB hot Claude trees fit on this runner host" — never a global one. Mirrors `run-plan.ts`'s - * own sandbox-id resolution (`request.sandbox || configured default provider`). The pool dispatch - * (`server.ts` `isLocalSandbox`) and the continuity module's own local/remote framing both - * resolve through this one function, so the "local-only" invariant has a single source of truth. - */ -export function resolvesToLocalProvider( - requestSandbox: string | undefined, - defaultProvider: string = loadRunnerConfig().providers.default, -): boolean { - return (requestSandbox || defaultProvider) === "local"; -} - -// --- Fingerprints and the pool key ------------------------------------------ // - -function sha256(value: string): string { - return createHash("sha256").update(value).digest("hex"); -} - -/** Deterministic JSON: object keys sorted recursively so equal values hash equal. */ -function canonicalJson(value: unknown): string { - if (value === null || typeof value !== "object") return JSON.stringify(value); - if (Array.isArray(value)) { - return `[${value.map(canonicalJson).join(",")}]`; - } - const entries = Object.entries(value as Record) - .filter(([, v]) => v !== undefined) - .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)); - return `{${entries - .map(([k, v]) => `${JSON.stringify(k)}:${canonicalJson(v)}`) - .join(",")}}`; -} - -/** - * A canonical hash over the config-bearing request fields (the continuation-versus-cold - * decision). Per-turn volatiles are excluded: `messages`, `turnId`, trace propagation - * (`context`), the rotating telemetry headers, and secret VALUES (`secrets` — the credential - * epoch covers rotation, and values must never enter any hash used for logging). The - * tool-callback ENDPOINT is included (routing config); its authorization is a credential and - * lives in the credential epoch instead. - */ -export function configFingerprint(request: AgentRunRequest): string { - const workflow = request.runContext?.workflow; - const shape = { - harness: request.harness ?? null, - sandbox: request.sandbox ?? null, - model: request.model ?? null, - provider: request.provider ?? null, - connection: request.connection ?? null, - deployment: request.deployment ?? null, - endpoint: request.endpoint ?? null, - credentialMode: request.credentialMode ?? null, - agentsMd: request.agentsMd ?? null, - systemPrompt: request.systemPrompt ?? null, - appendSystemPrompt: request.appendSystemPrompt ?? null, - tools: request.tools ?? null, - skills: request.skills ?? null, - customTools: request.customTools ?? null, - mcpServers: request.mcpServers ?? null, - toolCallbackEndpoint: request.toolCallback?.endpoint ?? null, - permissions: request.permissions ?? null, - sandboxPermission: request.sandboxPermission ?? null, - harnessFiles: request.harnessFiles ?? null, - workflowRevision: workflow?.revision - ? { - id: workflow.revision.id ?? null, - version: workflow.revision.version ?? null, - } - : null, - isDraft: workflow?.is_draft ?? null, - }; - return sha256(canonicalJson(shape)); -} - -function collectToolCallIds( - content: string | ContentBlock[] | undefined, - into: string[], - seen: Set, -): void { - if (!Array.isArray(content)) return; - for (const block of content) { - if (!block) continue; - if ( - (block.type === "tool_call" || block.type === "tool_result") && - typeof block.toolCallId === "string" && - block.toolCallId && - !seen.has(block.toolCallId) - ) { - seen.add(block.toolCallId); - into.push(block.toolCallId); - } - } -} - -/** - * A hash over the conversation the server received (the FE's pruned array): the ordered user - * message texts, the ordered tool-call ids across every message, and the user-message count. - * Assistant TEXT is deliberately ignored, so a live session that has already answered a plain - * user turn matches the next request's prefix (the FE's assistant turn contributes nothing). - * Tool-call ids ARE included, so an edited history trips a mismatch and degrades to cold - * replay rather than continuing wrongly. Ids are DEDUPED (unique, first-seen order): a resolved - * tool call rides the wire as a `tool_call` block PLUS a `tool_result` block sharing one id - * (vercel `messages.py` `_tool_part_blocks`), and the park-time prediction - * (`expectedNextHistoryFingerprint`) folds each emitted id in once — dedupe makes the two agree - * while a genuinely different id SET still mismatches. - * - * The dispatch stores the fingerprint the next request is EXPECTED to hash to (see - * `expectedNextHistoryFingerprint`), and checks the next request against the fingerprint of its - * PRIOR messages (everything before the new user tail), so a plain conversational continuation - * matches and any divergence falls to cold. - */ -export function historyFingerprint(messages: readonly ChatMessage[]): string { - const userTexts: string[] = []; - const toolCallIds: string[] = []; - const seenIds = new Set(); - let promptCount = 0; - for (const message of messages) { - if (message.role === "user") { - promptCount += 1; - userTexts.push(messageText(message.content)); - } - collectToolCallIds(message.content, toolCallIds, seenIds); - } - return sha256(canonicalJson({ userTexts, toolCallIds, promptCount })); -} - -/** - * The fingerprint a park should record so the NEXT request's prior conversation matches it: - * the full messages this turn ran, plus the tool-call ids the turn itself emitted, folded in - * as one synthetic trailing assistant message. - * - * Why: the FE keeps an assistant turn iff it has an answer part (`agentRequest.ts` - * `isAnswerPart`: non-empty text, `tool-*`/`dynamic-tool`, or file). So a tool-calling turn's - * ids ALWAYS appear in the next request's prior messages, and a fully empty assistant turn is - * pruned but contributes neither text nor ids — the prediction is deterministic either way - * (assistant text is not hashed). An id divergence still trips a mismatch and falls to cold. - */ -export function expectedNextHistoryFingerprint( - messages: readonly ChatMessage[], - emittedToolCallIds: readonly string[], -): string { - if (emittedToolCallIds.length === 0) return historyFingerprint(messages); - const syntheticAssistantTurn: ChatMessage = { - role: "assistant", - content: emittedToolCallIds.map((id) => ({ - type: "tool_call", - toolCallId: id, - })), - }; - return historyFingerprint([...messages, syntheticAssistantTurn]); -} - -/** - * The prior conversation for a continuation check: everything before the request's new user - * tail. Mirrors `transcript.priorMessages` for the trailing-user case (the playground always - * sends the new turn as the last user message), without importing that do-not-touch module. - */ -export function priorConversation(request: AgentRunRequest): ChatMessage[] { - const messages = request.messages ?? []; - if (messages.length && messages[messages.length - 1].role === "user") { - return messages.slice(0, -1); - } - return messages.slice(); -} - -/** - * The approval decision (allow/deny) the incoming request carries for a specific parked gate's - * tool-call id, or undefined when the request has no approval envelope for that id. Reuses the - * cold path's `approvalDecisionOf` (responder.ts) to parse the `{approved}` envelope, and matches - * strictly by toolCallId (the parked gate's id) — never by name+args — so a live resume answers - * exactly the gate that parked. An incoming reply for a different id, or a plain user message, - * yields undefined and the dispatch degrades to cold. - */ -export function approvalDecisionForToolCall( - request: AgentRunRequest, - toolCallId: string, -): "allow" | "deny" | undefined { - if (!toolCallId) return undefined; - for (const message of request.messages ?? []) { - const content = message?.content; - if (!Array.isArray(content)) continue; - for (const block of content) { - if (block?.type !== "tool_result" || block.toolCallId !== toolCallId) { - continue; - } - const decision = approvalDecisionOf(block); - if (decision !== undefined) return decision; - } - } - return undefined; -} - -/** - * True when the request's tail is a fresh user message with text and NOT an approval envelope. - * A continuation only takes the live path for a plain new user turn; an approval reply (a - * trailing tool-role message, or a user turn carrying a tool_result) stays cold here. - */ -export function tailIsFreshUserMessage(request: AgentRunRequest): boolean { - const messages = request.messages ?? []; - const tail = messages[messages.length - 1]; - if (!tail || tail.role !== "user") return false; - if (!messageText(tail.content).trim()) return false; - if (Array.isArray(tail.content)) { - const carriesToolTurn = tail.content.some( - (block) => block?.type === "tool_result" || block?.type === "tool_call", - ); - if (carriesToolTurn) return false; - } - return true; -} - -/** - * The credential epoch bounds how long a parked session may reuse its baked credentials. It is - * a PROCESS-LOCAL hash over the actual resolved secret VALUES (held only in runner memory — - * never logged, persisted, or emitted), combined with the mount credential expiry. A rotated - * same-slug secret changes the hash; an elapsed expiry invalidates the epoch. Either way the - * dispatch evicts and cold-starts with fresh credentials. - * - * The tool-callback bearer is deliberately EXCLUDED: it is per-turn material the backend - * re-mints on its auth-cache cadence (~60s), and every turn — continuation included — starts - * its tool relay from the INCOMING request's `toolCallback`, so the parked copy is never used - * to execute anything. Hashing it made warm sessions evict as "credentials-rotated" on every - * cache rollover for no protective value. Only material actually BAKED into the parked - * environment (the sandbox env secrets) belongs in the hash; the mount expiry bounds the rest. - */ -export interface CredentialEpoch { - /** sha256 over canonical(secrets). In-memory only; never surfaced. */ - secretsHash: string; - /** Mount credential expiry as epoch millis, or undefined when the sign response had none. */ - mountExpiresAtMs?: number; -} - -export function computeCredentialEpoch( - request: AgentRunRequest, - mountExpiresAt?: string, -): CredentialEpoch { - const material = canonicalJson({ - secrets: request.secrets ?? {}, - }); - const parsed = mountExpiresAt ? Date.parse(mountExpiresAt) : NaN; - return { - secretsHash: sha256(material), - mountExpiresAtMs: Number.isFinite(parsed) ? parsed : undefined, - }; -} - -/** - * Whether a parked session's MOUNT credentials have already expired, ignoring the secret material - * hash entirely. This answers only "can the parked environment still write its durable cwd?". - * - * The approval-resume path uses this instead of `credentialEpochValid`: a resume must NOT require - * the resume request's re-minted credentials to MATCH the parked ones (a fresh /run mints fresh - * short-lived material every time, so they practically never match), but an expired mount means - * the parked cwd can no longer be written, so it must still evict to cold. - */ -export function mountCredentialsExpired( - epoch: CredentialEpoch, - now = Date.now(), -): boolean { - return epoch.mountExpiresAtMs !== undefined && now >= epoch.mountExpiresAtMs; -} - -/** - * Why a parked epoch is no longer usable for an incoming request's epoch, or undefined when it - * still is. The two failure modes are distinguished so diagnosis works from logs: - * - `credentials-expired` — the mount credential's lifetime elapsed (time bound). - * - `credentials-rotated` — the resolved secret material changed (a rotated same-slug secret). - */ -export function credentialEpochMismatch( - parked: CredentialEpoch, - incoming: CredentialEpoch, - now = Date.now(), -): "credentials-expired" | "credentials-rotated" | undefined { - if (mountCredentialsExpired(parked, now)) return "credentials-expired"; - if (parked.secretsHash !== incoming.secretsHash) return "credentials-rotated"; - return undefined; -} - -/** - * Whether a parked epoch is still valid for an incoming request's epoch. Invalid (evict, cold) - * when the mount credential expired, or the resolved secret material changed. Thin - * wrapper over `credentialEpochMismatch` for callers that only need the boolean. - */ -export function credentialEpochValid( - parked: CredentialEpoch, - incoming: CredentialEpoch, - now = Date.now(), -): boolean { - return credentialEpochMismatch(parked, incoming, now) === undefined; -} - -/** Which project-scope source produced a pool key: the service-stamped run context, or the mount. */ -export type PoolScopeSource = "run-context" | "mount"; - -/** A pool key plus the scope source that produced it (for the greppable `[keepalive] scope=` log). */ -export interface PoolScope { - key: string; - source: PoolScopeSource; -} - -/** - * The project scope for a run: PREFERRED from the run context the service stamps server-side - * (`runContext.project.id`), FALLING BACK to the mount's owning project id when the run context - * carries none. The run-context id is the trustworthy source: the service derives it from its own - * request state (never from a caller-supplied wire field), so it does not depend on a durable - * mount existing. The mount scope stays as the fallback for the transition and for runs without a - * stamped project. Returns undefined when NEITHER source yields a scope. - * - * This is the single precedence rule other project-scoped decisions (the pool key, the in-flight - * sandbox kill filter) must reuse rather than re-deriving, so they agree by construction. - */ -export function projectScopeFor( - request: Pick, - mountProjectId: string | undefined, -): { id: string; source: PoolScopeSource } | undefined { - const runContextProject = request.runContext?.project?.id?.trim(); - if (runContextProject) return { id: runContextProject, source: "run-context" }; - const mount = mountProjectId?.trim(); - if (mount) return { id: mount, source: "mount" }; - return undefined; -} - -/** - * The pool key: `:`. Provider separation does not need another key segment: - * providers have separate pools, and `configFingerprint` includes `request.sandbox`. - * - * Returns null when there is no session id, or when `projectScopeFor` yields no project scope — - * such a request MUST NOT park (there is no safe key that separates callers), and the dispatch - * runs it fully cold. This no-scope-no-park rule is the keep-alive safety invariant and is - * unchanged. - */ -export function poolKeyFor( - request: AgentRunRequest, - mountProjectId: string | undefined, -): PoolScope | null { - const sessionId = request.sessionId?.trim(); - if (!sessionId) return null; - const scope = projectScopeFor(request, mountProjectId); - if (!scope) return null; - return { key: `${scope.id}:${sessionId}`, source: scope.source }; -} - // --- The pool --------------------------------------------------------------- // export type SessionState = "busy" | "idle" | "awaiting_approval" | "destroyed"; diff --git a/services/runner/src/server.ts b/services/runner/src/server.ts index 20cdfa9981..d2ddc2987e 100644 --- a/services/runner/src/server.ts +++ b/services/runner/src/server.ts @@ -54,10 +54,12 @@ import { priorConversation, readKeepaliveConfig, resolvesToLocalProvider, - SessionPool, tailIsFreshUserMessage, type KeepaliveConfig, type KeepaliveProviderName, +} from "./engines/sandbox_agent/session-identity.ts"; +import { + SessionPool, type LiveSession, } from "./engines/sandbox_agent/session-pool.ts"; import { runnerInfo } from "./version.ts"; diff --git a/services/runner/tests/unit/session-keepalive-approval.test.ts b/services/runner/tests/unit/session-keepalive-approval.test.ts index d9e7ea0acb..2ad6362a19 100644 --- a/services/runner/tests/unit/session-keepalive-approval.test.ts +++ b/services/runner/tests/unit/session-keepalive-approval.test.ts @@ -23,10 +23,8 @@ import { type KeepaliveContext, type KeepaliveEngine, } from "../../src/server.ts"; -import { - SessionPool, - type KeepaliveConfig, -} from "../../src/engines/sandbox_agent/session-pool.ts"; +import { SessionPool } from "../../src/engines/sandbox_agent/session-pool.ts"; +import type { KeepaliveConfig } from "../../src/engines/sandbox_agent/session-identity.ts"; import type { MountCredentials } from "../../src/engines/sandbox_agent/mount.ts"; import { acquireEnvironment, diff --git a/services/runner/tests/unit/session-keepalive-dispatch.test.ts b/services/runner/tests/unit/session-keepalive-dispatch.test.ts index 7c3dcb58ca..74c40f4f40 100644 --- a/services/runner/tests/unit/session-keepalive-dispatch.test.ts +++ b/services/runner/tests/unit/session-keepalive-dispatch.test.ts @@ -23,10 +23,8 @@ import { type KeepaliveContext, type KeepaliveEngine, } from "../../src/server.ts"; -import { - SessionPool, - type KeepaliveConfig, -} from "../../src/engines/sandbox_agent/session-pool.ts"; +import { SessionPool } from "../../src/engines/sandbox_agent/session-pool.ts"; +import type { KeepaliveConfig } from "../../src/engines/sandbox_agent/session-identity.ts"; import type { MountCredentials } from "../../src/engines/sandbox_agent/mount.ts"; import type { SessionEnvironment } from "../../src/engines/sandbox_agent.ts"; diff --git a/services/runner/tests/unit/session-pool.test.ts b/services/runner/tests/unit/session-pool.test.ts index 1d8cf921c6..6465bcae89 100644 --- a/services/runner/tests/unit/session-pool.test.ts +++ b/services/runner/tests/unit/session-pool.test.ts @@ -20,10 +20,10 @@ import { priorConversation, readKeepaliveConfig, resolvesToLocalProvider, - SessionPool, tailIsFreshUserMessage, type CredentialEpoch, -} from "../../src/engines/sandbox_agent/session-pool.ts"; +} from "../../src/engines/sandbox_agent/session-identity.ts"; +import { SessionPool } from "../../src/engines/sandbox_agent/session-pool.ts"; describe("resolvesToLocalProvider (local/remote gate)", () => { it("is true when the request explicitly asks for local", () => {