diff --git a/codev/plans/1198-shellper-reconnect-error-is-sw.md b/codev/plans/1198-shellper-reconnect-error-is-sw.md new file mode 100644 index 000000000..51a3da14f --- /dev/null +++ b/codev/plans/1198-shellper-reconnect-error-is-sw.md @@ -0,0 +1,128 @@ +# PIR Plan: Fix swallowed shellper reconnect error (silent zombie terminals) + +Issue: #1198 + +## Understanding + +After a Tower restart, a persistent (shellper-backed) terminal can become a silent zombie: blank in every viewer, keystrokes and `afx send` messages dropped, yet reported as `running` with "Message sent" logged. The shellper and its Claude child stay healthy throughout. + +I verified every claim in the issue against the code. The root cause chain is confirmed: + +1. **The client swallows its own close event on error paths.** `packages/codev/src/terminal/shellper-client.ts:266-272`: `cleanup()` sets `this._connected = false` before destroying the socket. The socket's subsequent `'close'` event (`shellper-client.ts:153-163`) reads `wasConnected = this._connected`, which is already false, so `this.emit('close')` is skipped. Any post-handshake socket or parser error (`shellper-client.ts:131-145`) runs `safeEmitError(err)` then `cleanup()`, hitting exactly this path. +2. **Everything downstream hangs off that missing emission.** `session-manager.ts:291-300` (createSession) and `session-manager.ts:407-413` (reconnectSession) contain the ONLY code that logs `shellper disconnected unexpectedly` and calls `removeDeadSession`. `pty-session.ts:198-205` is the only code that transitions the PtySession to exited on socket death. Neither runs. The session stays in every map as a healthy-looking zombie. +3. **The error that WAS emitted goes nowhere.** session-manager re-emits it as `'session-error'` (`session-manager.ts:287,403`), which has zero consumers anywhere in the Tower layer (verified by grep across `packages/codev` and `packages/core`). Not logged, not acted on. +4. **Writes are silent no-ops by design.** `shellper-client.ts:274-282`: `write()`/`resize()` return void and do nothing when `!this._connected`. `pty-session.ts:306-316` propagates the void. The message router (`tower-routes.ts:1374-1380`) then logs "Message sent" for frames that were dropped. +5. **Contributing race:** `towerStop` (`packages/codev/src/agent-farm/commands/tower.ts:346-358`) sends SIGTERM and returns immediately, so `afx tower stop && afx tower start` overlaps the old Tower's teardown with the new Tower's adoption pass. + +### Two additional findings from investigation + +**(a) Fixing the close emission alone is hazardous.** Once `'close'` fires on error paths, the unexpected-close path runs `removeDeadSession` (which unlinks the shellper's socket file at `session-manager.ts:704-712`, even if the shellper is alive and listening) and PtySession emits `'exit'`, whose tower-side handler calls `deleteTerminalSession` (`tower-terminals.ts:803`, deletes the SQLite row). Net effect: a transient one-shot socket error against a healthy shellper would permanently orphan a live Claude conversation (no socket file, no DB row, no adoption on next restart). The fix must attempt reconnection before declaring death. This is the issue's item 5 ("re-run the reconnect for that session"), and it turns out to be a correctness requirement of item 1, not optional hardening. + +**(b) Adoption races the REPLAY frame** (explains the architect's fleet-sweep comment: 6 healthy terminals blank at `total=0`). Both adoption call sites (`tower-terminals.ts:750` and `tower-terminals.ts:962`) call `client.getReplayData()` synchronously right after `connect()` resolves. The shellper sends REPLAY after WELCOME, often in a separate socket read, so the replay is frequently not there yet and the ring buffer starts empty. `waitForReplay()` exists on the client (`shellper-client.ts:310-325`) and is unused at both sites. + +## Proposed Change + +Six pieces, in the issue's priority order plus the two findings above. + +### 1. Fix the swallowed close (shellper-client.ts) + +Add two private flags: + +- `_everConnected`: set true at handshake success (next to `_connected = true`), never cleared by `cleanup()`. +- `_intentionalDisconnect`: set true in `disconnect()` before calling `cleanup()`. + +The socket `'close'` handler emits `'close'` when `_everConnected && !_intentionalDisconnect` (then clears `_everConnected` so the emission is one-shot). + +Semantics preserved exactly where they matter: + +- Intentional `disconnect()` (Tower shutdown, killSession, detach) emits nothing, same as today. `SessionManager.shutdown()` and `killSession()` stay safe without relying on handler-ordering luck. +- Remote hangup post-handshake emits `'close'`, same as today. +- Error-then-cleanup now also emits `'close'`. This is the fix. +- Handshake-phase failures (version mismatch, invalid WELCOME) still only reject the connect promise, no `'close'` emission, since `_everConnected` was never set. + +`afx attach` (`commands/attach.ts:196`) already listens for `'close'` and currently hangs silently on an error-path close. It gets fixed for free. + +### 2. Reconnect-in-place before declaring death (session-manager.ts) + +Refactor the duplicated client wiring in `createSession` (lines 274-300) and `reconnectSession` (lines 395-413) into one private `wireClientEvents(sessionId, session)` method, and change the unexpected-close behavior: + +On `'close'` with the session still in the map: + +1. Log `Session shellper connection lost unexpectedly` (the trace that was missing). +2. If the shellper process is still alive (`isProcessAlive(pid)` plus start-time match, the same PID-reuse guard `reconnectSession` already uses) and the socket file still exists: attempt a bounded reconnect, up to 3 attempts with 500ms / 1s / 2s backoff. On success: replace `session.client`, re-wire events via `wireClientEvents`, log success, and emit a new event `'session-reconnected', sessionId, client`. +3. If the process is dead, the socket is gone, or all attempts fail: the existing dead path, unchanged (`removeDeadSession` + `'session-error'`). + +Loop guard: a per-session `reconnectAttemptCount` plus `lastReconnectAt` timestamp. The count resets when a reconnected client survives longer than 30s; if a session burns through 3 consecutive recovery rounds without a 30s-stable connection, take the dead path. This prevents a pathological shellper (accepts connects, then drops them) from spinning forever. + +### 3. Re-attach the reconnected client to the PtySession (pty-session.ts + tower-server.ts) + +Two coordinated edits: + +- **pty-session.ts:198-205**: on client `'close'` (with no EXIT seen), do not tear down immediately. Start a grace timer (15s, longer than the session-manager retry window of ~3.5s plus connect timeouts) before setting `exitCode = -1` / emitting `'exit'` / `cleanupShellper()`. `attachShellper()` cancels the pending timer when a new client arrives. This mirrors the existing `_restartCleanupTimeout` pattern in the same file (lines 158-189), and prevents the destructive `deleteTerminalSession` from racing a successful reconnect. +- **tower-server.ts** (next to the SessionManager construction at line 373): subscribe to `'session-reconnected'` and re-attach: look up the PtySession by sessionId and call `attachShellper(newClient, Buffer.alloc(0), pid, shellperSessionId)`. Replay data is deliberately empty on re-attach: the ring buffer already holds the session history, and re-pushing the shellper's replay would duplicate output for connected viewers. `attachShellper` is already idempotent for re-attach (Issue #1047 Fix E removes stale listeners). + +### 4. Consume and log 'session-error' (tower-server.ts) + +Same place: `shellperManager.on('session-error', (sessionId, err) => log('ERROR', ...))`. A reconnect that dies permanently now always leaves an ERROR line in the Tower log. + +### 5. Make towerStop wait (commands/tower.ts) + +After the SIGTERM loop (`tower.ts:346-353`): poll each PID with `process.kill(pid, 0)` every 200ms, up to 8s. Any survivor gets SIGKILL. Print "Tower stopped" only once all PIDs are gone (with a note if escalation happened). `afx tower stop && afx tower start` then serializes instead of overlapping, removing the adoption-vs-teardown race that likely minted the observed zombie. + +### 6. Make write/resize failure observable (shellper-client.ts, pty-session.ts, tower-routes.ts) + +- `ShellperClient.write()/resize()` return `boolean` (false when not connected). `IShellperClient` interface updated accordingly. +- `PtySession.write()/resize()` return `boolean`, propagating the client's return (pty-backed path: true when the pty write happens). +- New `PtySession.writable` getter: shellper-backed sessions report `shellperClient.connected && status === 'running'`; pty-backed report `pty != null && status === 'running'`. +- Message router (`tower-routes.ts` handleSendMessage): after resolving the session, if `!session.writable`, respond 503 `{ error: 'TERMINAL_NOT_WRITABLE' }` and log at ERROR (`Message DROPPED: ...`) instead of writing into the void and logging "Message sent". The deferred-delivery flush path (`tower-routes.ts:111`) gets the same guard, logging a drop instead of silently flushing to a dead session. + +With piece 2 in place a dropped message should be rare (the client self-heals), but when it does happen the sender now gets a hard error instead of a false success, and `afx send` surfaces it to the caller. + +### 7. Wait for REPLAY during adoption (tower-terminals.ts) + +Both call sites (`tower-terminals.ts:750` and `:962`): replace `client.getReplayData() ?? Buffer.alloc(0)` with `await client.waitForReplay()`. Resolves immediately when the frame already arrived; waits up to 500ms otherwise; returns an empty buffer when the shellper genuinely has nothing to replay (it only sends REPLAY for non-empty buffers). Fixes the blank-until-poked terminals from the fleet sweep and makes `total=0` a much stronger zombie signal for future diagnosis. Worst-case adoption cost: +500ms per batch of 5 for terminals with empty replay buffers. + +### Out of scope (proposed follow-up issue) + +PING/PONG heartbeat-based zombie detection (first half of the issue's item 5). Reconnect-on-close covers the entire observed failure class (the client always sees an error or close when its socket dies; the heartbeat only adds coverage for a hung-but-open socket). Worth its own issue rather than riding this one. + +## Files to Change + +- `packages/codev/src/terminal/shellper-client.ts:131-163, 262-297` — close-emission flags; boolean returns for write/resize; interface update +- `packages/codev/src/terminal/session-manager.ts:240-307, 350-421, 704-712` — extract shared `wireClientEvents`; bounded reconnect-in-place on unexpected close; `'session-reconnected'` event +- `packages/codev/src/terminal/pty-session.ts:119-206, 305-331` — close grace timer cancelled by `attachShellper`; boolean write/resize; `writable` getter +- `packages/codev/src/agent-farm/servers/tower-server.ts:~373` — subscribe to `'session-error'` (ERROR log) and `'session-reconnected'` (re-attach + INFO log) +- `packages/codev/src/agent-farm/servers/tower-terminals.ts:750, 962` — `await client.waitForReplay()` in both adoption paths +- `packages/codev/src/agent-farm/servers/tower-routes.ts:~111, ~1307-1389` — `writable` guard, 503 + ERROR-log on drop, in both immediate and deferred-flush paths +- `packages/codev/src/agent-farm/commands/tower.ts:~346-358` — poll-for-exit with SIGKILL escalation in `towerStop` +- `packages/codev/src/terminal/__tests__/shellper-client.test.ts` — new close-emission cases +- `packages/codev/src/terminal/__tests__/session-manager.test.ts` — reconnect-in-place cases +- `packages/codev/src/terminal/__tests__/pty-session-attach.test.ts` — grace-timer and re-attach cases; mock client updates for the interface change + +## Risks & Alternatives Considered + +- **Risk: new `'close'` emissions reach existing listeners on paths that never saw them before** (shutdown, killSession, detach). Mitigated by the `_intentionalDisconnect` flag: `disconnect()` keeps today's emit-nothing behavior, so only genuinely unexpected closes gain the emission. Consumers audited: session-manager (guards on `sessions.has`), pty-session (grace timer added), attach.ts (wants the event; currently bug-hangs without it). +- **Risk: reconnect loop against a flapping shellper.** Mitigated by the attempt cap plus 30s-stability reset described in piece 2; exhaustion falls through to the existing dead path. +- **Risk: re-attach duplicates output.** Mitigated by passing empty replay on re-attach (ring buffer already has history). The 15s PtySession grace timer only delays the *unexpected-death* teardown; real EXIT frames are unaffected (separate event). +- **Risk: `waitForReplay` slows adoption.** Bounded at 500ms per empty-replay terminal, batched 5-way; ~2s worst case for a 20-terminal fleet. Acceptable against the alternative (blank terminals). +- **Alternative: emit `'close'` unconditionally from `cleanup()`.** Rejected: fires on intentional disconnects and handshake failures, breaking shutdown()/killSession() semantics and double-firing with connect() rejections. +- **Alternative: heartbeat-driven zombie detection instead of reconnect-on-close.** Rejected for this PR: heavier (periodic timers per session, threshold tuning) and it detects the state instead of preventing it. Proposed as follow-up. +- **Alternative: have the message router probe liveness end-to-end (write + echo check).** Rejected: intrusive (injects bytes), racy, and the `writable` check plus self-healing reconnect covers the realistic failure class. + +## Test Plan + +Unit tests (vitest, existing harnesses with mini-shellper socket servers): + +- shellper-client: error-path close DOES emit `'close'` (post-handshake socket error, then assert `'close'` fired); `disconnect()` does NOT emit `'close'`; handshake failure does not emit `'close'`; `write()` returns false when disconnected, true when connected. +- session-manager: unexpected socket close with shellper alive → client reconnects in place, `'session-reconnected'` emitted, session stays in map, socket file NOT unlinked; unexpected close with shellper dead → existing dead path (`removeDeadSession` + `'session-error'`); reconnect exhaustion → dead path. +- pty-session: unexpected client close → no immediate 'exit'; `attachShellper` with new client within grace window cancels teardown and I/O resumes; grace expiry without re-attach → 'exit' emitted as before; `writable` reflects client connectivity. + +Build + full test suite: `pnpm --filter @cluesmith/codev build && pnpm --filter @cluesmith/codev test` (run from the worktree). + +Manual verification for the dev-approval gate (reviewer, on the running worktree build): + +1. `pnpm build && pnpm -w run local-install` to run the patched Tower. +2. Baseline: every persistent terminal's `GET /api/terminals/:id/output` returns `total>=1` right after restart (piece 7: no more blank-until-poked terminals with non-empty shellper buffers). +3. Simulated transient error: pick a session's shellper socket, kill the Tower-side TCP connection (e.g. `gdb`/`lsof` fd close, or easier: send a malformed frame by connecting a second raw client and observing; simplest reliable repro is SIGSTOP the shellper, write to it until the socket errors, SIGCONT). Expected: Tower log shows `connection lost unexpectedly` then `re-established`; terminal keeps working; no restart needed. +4. Zombie-drop observability: with a session forced dead (SIGKILL its shellper), `afx send` to it must return an error, and the Tower log must show `Message DROPPED` / ERROR, never "Message sent". +5. `afx tower stop` returns only after the process exits: `afx tower stop && lsof -i :4100` shows nothing; `afx tower stop && afx tower start` no longer overlaps (no port-in-use retry lines in the new log). diff --git a/codev/projects/1198-shellper-reconnect-error-is-sw/1198-review-iter1-rebuttals.md b/codev/projects/1198-shellper-reconnect-error-is-sw/1198-review-iter1-rebuttals.md new file mode 100644 index 000000000..fd23708f7 --- /dev/null +++ b/codev/projects/1198-shellper-reconnect-error-is-sw/1198-review-iter1-rebuttals.md @@ -0,0 +1,29 @@ +# Consultation rebuttals — PIR #1198 + +## Iteration 2 (human-requested round, post-incident-fix diff) + +**claude: APPROVE** — no findings. + +**codex: COMMENT** (advisory, MEDIUM confidence), two notes: + +1. *Creation sites still race REPLAY* (tower-routes.ts:609/2050, tower-instances.ts:581/1059). Accepted — real for fast-starting children whose first bytes land in the REPLAY frame. Fixed without adding creation latency: new shellpers now always send a REPLAY frame even when empty (so waiters resolve in milliseconds; creation always talks to a freshly spawned new-binary shellper), and all four creation sites await `waitForReplay()`. Test added: shellper sends an empty REPLAY when its buffer is empty. Old clients treat an empty REPLAY as no replay data (wire-compatible). +2. *Plan file lacks approval frontmatter*. Rebutted: the frontmatter convention exists for architect-pre-created artifacts committed to main before spawn, so porch can skip the authoring phase. This plan was authored inside porch's PIR run and approved at the `plan-approval` gate; the approval record lives in `codev/projects/1198-*/status.yaml` (gate history), which is the source of truth for porch-driven artifacts. Adding retroactive frontmatter would duplicate state the state machine already owns. + +## Iteration 1 rebuttals — PIR #1198 + +## codex: REQUEST_CHANGES + +**Finding**: `attachShellper()` (pty-session.ts) unconditionally opened `this.logPath` and assigned `this.logFd` without closing the previous handle. With #1198 making re-attach the routine recovery step after `'session-reconnected'`, each successful in-place reconnect leaked one append fd when disk logging is enabled. + +**Assessment**: Real defect, accepted in full. A regression introduced by this PR's recovery flow (pre-#1198, `attachShellper` ran once per session lifetime, so the unconditional open was safe). + +**Change made** (commit `0126d5d3`): +- The open is now guarded: `if (this.diskLogEnabled && this.logFd === null)`. A recovery re-attach reuses the existing handle; `cleanupShellper()` closes and nulls the fd, so a fresh attach after a genuine teardown still reopens it. +- Regression test added as requested: `pty-session-attach.test.ts` — "does not reopen the disk log when a recovery re-attach arrives" (attach → re-attach asserts exactly one open of the log path; detach → attach asserts the reopen). The test fails without the guard. +- Documented in the review file's "Things to Look At During PR Review" with an explicit note that PIR's single-pass consultation did not re-review the fix, so the human at the `pr` gate is the remaining reviewer of the guard. + +Full suite after the fix: 3533 passed, 0 failed. + +## claude: APPROVE + +No findings to address. The review independently verified the `_closePending` simplification against the plan's original two-flag design and confirmed handshake-phase failures still only reject the connect promise. diff --git a/codev/projects/1198-shellper-reconnect-error-is-sw/status.yaml b/codev/projects/1198-shellper-reconnect-error-is-sw/status.yaml new file mode 100644 index 000000000..dba50db22 --- /dev/null +++ b/codev/projects/1198-shellper-reconnect-error-is-sw/status.yaml @@ -0,0 +1,30 @@ +id: '1198' +title: shellper-reconnect-error-is-sw +protocol: pir +phase: verified +plan_phases: [] +current_plan_phase: null +gates: + plan-approval: + status: approved + requested_at: '2026-07-18T10:24:16.064Z' + approved_at: '2026-07-18T10:25:46.731Z' + dev-approval: + status: approved + requested_at: '2026-07-18T14:59:12.299Z' + approved_at: '2026-07-19T01:43:07.193Z' + pr: + status: approved + requested_at: '2026-07-19T01:51:34.055Z' + approved_at: '2026-07-21T11:13:39.356Z' +iteration: 1 +build_complete: true +history: [] +started_at: '2026-07-18T10:17:40.159Z' +updated_at: '2026-07-21T11:15:14.553Z' +pr_history: + - phase: review + pr_number: 1204 + branch: builder/pir-1198 + created_at: '2026-07-19T01:45:52.590Z' +pr_ready_for_human: false diff --git a/codev/resources/arch.md b/codev/resources/arch.md index c85d13cda..4857cf357 100644 --- a/codev/resources/arch.md +++ b/codev/resources/arch.md @@ -206,6 +206,19 @@ Shell / Claude / Builder process 5. **Kill**: Tower sends SIGTERM via SIGNAL frame, waits 5s, SIGKILL if needed. Cleans up socket file. 6. **Graceful degradation**: If shellper spawn fails, Tower falls back to direct node-pty (non-persistent). SQLite row has `shellper_socket = NULL`. Dashboard shows "Session persistence unavailable" warning. +#### Connection-loss recovery (#1198) + +A Tower↔shellper connection that dies **unexpectedly** (post-handshake socket error, protocol/parser error, or remote hangup without an EXIT frame) self-heals instead of zombifying or tearing down: + +- **Close-emission contract** (`shellper-client.ts`): `cleanup(intentional)` records `_closePending` when it tears down a live connection that was not asked to die via `disconnect()`; the socket `'close'` event consumes it and emits `'close'` exactly once. The decision is captured at teardown time because error paths run `cleanup()` *before* the close event fires — reading `_connected` inside the close handler is the bug that swallowed the emission (silent zombie terminals). Intentional teardown (`disconnect()`, `killSession`, `detachShellper`, `shutdown()`) never emits `'close'`. +- **In-place reconnect** (`session-manager.ts`): on unexpected close, `recoverSession` retries the connection against the still-alive shellper (3 attempts at 500ms/1s/2s, preflighted by PID-alive + start-time + socket-file checks; capped at 3 consecutive rounds with a 30s stability reset), replacing `session.client` and emitting `'session-reconnected'` on success. Only exhaustion or an unreachable process takes the historical dead path (`removeDeadSession` + `'session-error'`). +- **Deferred teardown** (`pty-session.ts`): PtySession's close handler arms a grace timer (`SHELLPER_CLOSE_GRACE_MS`, 15s — sized above a full recovery round) instead of tearing down immediately; `attachShellper()` cancels it. The tower layer subscribes to `'session-reconnected'` and re-attaches the replacement client (empty replay — the ring buffer already holds history), and logs `'session-error'` at ERROR. +- **Write observability**: `ShellperClient.write()/resize()` and `PtySession.write()/resize()` return delivery booleans; `PtySession.writable` reports live-socket connectivity (a dying session still shows `status: running` until teardown). The send router returns 503 `TERMINAL_NOT_WRITABLE` and logs `Message DROPPED` instead of `Message sent`; the deferred-send buffer holds messages while a session is unwritable and drops loudly only at max age. + +**Oversized frames are dropped, not fatal** (#1198 incident): a long-lived full-screen TUI session's replay buffer (newline-free, unbounded partial — see #1047) can exceed `MAX_FRAME_SIZE` (16MB). Old shellper binaries send it anyway, and treating it as a fatal parser error is what zombified the same terminals on every reconnect — and, once recovery retried a *deterministic* failure to exhaustion, escalated to killing live sessions. The `FrameParser` therefore discards an oversized frame incrementally (emitting `'frame-skipped'`) and keeps the stream alive; `ShellperClient` resolves replay waiters with an empty replay (viewers repaint via the post-connect resize nudge); new shellpers cap the replay they send to `REPLAY_PAYLOAD_MAX` (8MB tail). This must stay Tower-side-tolerant: running shellpers are old binaries that survive Tower upgrades. + +**Invariants**: (1) never tear down a terminal's registry/DB state (socket unlink, `deleteTerminalSession`) on a raw socket close without first checking whether the shellper process is alive — the close path's cleanup is destructive, and the shellper protocol exists precisely so a lost connection can be re-established; (2) `removeDeadSession` never unlinks the socket of a still-live shellper process — an unlinked socket makes a live shellper unreachable *and* flags it for `killOrphanedShellpers`' kill path, converting a connect failure into a killed conversation; (3) the shellper never emits a frame larger than `MAX_FRAME_SIZE`. Relatedly, `towerStop` polls until the SIGTERMed process exits (SIGKILL after 8s) so `afx tower stop && afx tower start` cannot overlap the old Tower's teardown with the new Tower's adoption pass, and both adoption call sites use `waitForReplay()` rather than reading replay data synchronously (the REPLAY frame usually arrives in a later socket read than WELCOME). + #### Terminal reconnect/replay contract (#1047) When a WebSocket client attaches, Tower replays the ring buffer as a binary DATA frame **bracketed by `pause`/`resume` control frames** so the client can render the (potentially large) snapshot without counting it against its live-backpressure budget. Clients pass `?resume=` to request only the bytes after a sequence number (a *delta* reconnect) instead of the full buffer — the dominant cost saver when Tower is hosted remotely and reconnects are frequent. The ring-buffer `partial` (incomplete trailing line) is kept **whole and unbounded** so a full-screen TUI's alt-screen state replays faithfully; per-frame CPU is bounded instead by `pushData` scanning only the new chunk (not re-splitting the accumulated buffer). On the client side, a connecting terminal must **force a redraw** shortly after attach (a size-delta resize → SIGWINCH) — a full-screen TUI only repaints on a size *change*, and the connect-time resize can be a same-size no-op; both the web dashboard (`Terminal.tsx`) and the VSCode adapter (`terminal-adapter.ts`) do this. Under genuine *live* overload, clients **drop** ephemeral output (the app repaints) rather than reconnecting — reconnecting to relieve backpressure re-pulls the same payload and storms. diff --git a/codev/resources/lessons-learned.md b/codev/resources/lessons-learned.md index f7dea5ef1..a37f5f8b7 100644 --- a/codev/resources/lessons-learned.md +++ b/codev/resources/lessons-learned.md @@ -444,6 +444,9 @@ Generalizable wisdom extracted from review documents, ordered by impact. Updated - [From 0116] Tower's `server.listen()` callback runs after TCP bind, but the callback itself is `async` and may not complete Shellper manager initialization before the first request arrives. E2E tests need explicit readiness probes (e.g., `waitForShellperReady()`). - [From 0116] Shellper requires both `persistent: true` and `cwd` in terminal creation API requests. Without both, the handler falls through to the non-Shellper path. Easy to miss in E2E tests. - [From 0118] `socket.write()` returning `false` means kernel buffer is full, not that the write failed. The aggressive approach (destroy on `false`) is correct for broadcast scenarios where slow clients should not degrade output to others. +- [From #1198] Decide lifecycle emissions from state captured **at the transition**, not state re-read later — and before un-swallowing a long-suppressed event, audit every consumer of it. ShellperClient's close handler read `_connected` to decide whether to emit `'close'`, but error paths ran `cleanup()` (clearing the flag) before the socket's close event fired, so every error-path death was silent: the terminal stayed registered as `running` with all input dropped until the next Tower restart. The fix records the owed emission (`_closePending`) inside `cleanup()` itself, where the live→dead transition happens. Crucially, the ten-line emission fix alone would have been a *regression*: the downstream close handlers were written for "shellper crashed" and destructively unlink the live socket and delete the SQLite row — a transient error against a healthy shellper would have gone from "zombie until restart heals it" to "healthy Claude conversation permanently orphaned". Un-suppressing the event required pairing it with in-place reconnection and a grace-deferred teardown. +- [From #1198] A retry loop assumes the failure is transient — against a *deterministic* failure it amplifies the blast radius, so the give-up path must degrade to a state-preserving mode, never a destructive one. The in-place-reconnect loop was built for transient socket errors; the real trigger turned out to be deterministic (a >16MB replay frame that errored the parser on *every* connect), so recovery retried, exhausted its rounds, and the dead path unlinked a live shellper's socket — which `killOrphanedShellpers` then read as "orphan with unresponsive socket" and killed two architects' live Claude conversations (strictly worse than the zombie being fixed). Two durable rules: classify the *first* failure before retrying identically (same error text on every attempt = deterministic, stop early), and audit what your failure path *destroys* — cleanup written for "process is dead" must verify the process is dead. The layered fix: tolerate the poison input (parser discards oversized frames), cap it at the source (shellper truncates replay), and make the destructive path conditional (`removeDeadSession` keeps a live process's socket). +- [From #1198] Success-shaped logging makes an outage invisible: the message router logged `"Message sent"` unconditionally because `write()` returned void and dropped silently when disconnected — so three messages vanished into a zombie terminal with success logged for each. A delivery path needs an observable failure contract at the API boundary (boolean returns, a `writable` probe, an error response), not just log lines written before the outcome is known. Corollary: a state that reports `running` while its transport is dead needs a distinct probe (`writable`) — status alone attested health the connection didn't have. - [From 627] When multiple bugfix patches compete over the same state, consolidate them into a single owner (e.g., a state machine) rather than adding more guards. Three scroll mechanisms fighting each other created more bugs than they fixed. - [From 627] Lifecycle phases (initial-load → buffer-replay → interactive) eliminate magic thresholds. Instead of asking "is this scroll event real?", ask "what phase am I in?" to determine behavior. diff --git a/codev/reviews/1198-shellper-reconnect-error-is-sw.md b/codev/reviews/1198-shellper-reconnect-error-is-sw.md new file mode 100644 index 000000000..b0eddb512 --- /dev/null +++ b/codev/reviews/1198-shellper-reconnect-error-is-sw.md @@ -0,0 +1,75 @@ +# PIR Review: Shellper reconnect error swallowed — silent zombie terminals + +Fixes #1198 + +## Summary + +A post-handshake socket or parser error on a shellper connection ran `cleanup()` before the socket's `'close'` event fired, so the `'close'` emission (decided by reading `_connected`, already cleared) was swallowed — and with it every downstream recovery path. The terminal became a silent zombie: reported `running`, all input dropped, `"Message sent"` logged for messages that went nowhere, until the next Tower restart. This PR makes the client record the owed `'close'` at teardown time (`_closePending`, captured inside `cleanup()` where the knowledge exists), and — because emitting `'close'` alone would have made a transient error *permanently orphan* a healthy shellper (the historical close path unlinks the live socket and deletes the SQLite row) — pairs it with in-place recovery: SessionManager reconnects to the still-alive shellper (bounded retry with PID/start-time/socket preflight), PtySession defers its destructive teardown behind a 15s grace timer that a successful re-attach cancels, and the Tower layer re-attaches the replacement client. Supporting fixes: `'session-error'` finally has a consumer (ERROR-logged), adoption awaits the REPLAY frame instead of racing it (blank-until-poked terminals), sends to a dead terminal return 503 `TERMINAL_NOT_WRITABLE` and log `Message DROPPED` instead of false success, and `afx tower stop` polls until the process exits (SIGKILL after 8s) so `stop && start` serializes instead of overlapping adoption with teardown. + +## Files Changed + +- `packages/codev/src/terminal/shellper-client.ts` (+31 / −8) +- `packages/codev/src/terminal/session-manager.ts` (+147 / −45) +- `packages/codev/src/terminal/pty-session.ts` (+65 / −11) +- `packages/codev/src/terminal/pty-manager.ts` (+15 / −0) +- `packages/codev/src/agent-farm/servers/tower-server.ts` (+27 / −0) +- `packages/codev/src/agent-farm/servers/tower-terminals.ts` (+7 / −2) +- `packages/codev/src/agent-farm/servers/tower-routes.ts` (+13 / −0) +- `packages/codev/src/agent-farm/servers/send-buffer.ts` (+14 / −0) +- `packages/codev/src/agent-farm/commands/tower.ts` (+35 / −0) +- `packages/codev/src/terminal/__tests__/shellper-client.test.ts` (+71 / −0) +- `packages/codev/src/terminal/__tests__/session-manager.test.ts` (+130 / −0) +- `packages/codev/src/terminal/__tests__/pty-session-attach.test.ts` (+34 / −1) +- `packages/codev/src/terminal/__tests__/tower-shellper-integration.test.ts` (+42 / −3) +- `packages/codev/src/agent-farm/__tests__/send-buffer.test.ts` (+41 / −1) +- `packages/codev/src/agent-farm/__tests__/tower-routes.test.ts` (+33 / −5) +- `packages/codev/src/agent-farm/__tests__/tower-terminals.test.ts` (+2 / −0) + +## Commits + +- `adb335eb` [PIR #1198] Fix swallowed error-path close and make write/resize observable in ShellperClient +- `00afce9f` [PIR #1198] Reconnect in place on unexpected shellper socket close +- `8e224038` [PIR #1198] Grace-timer teardown on unexpected disconnect; observable write/resize in PtySession +- `f951a338` [PIR #1198] Tower layer: log session errors, re-attach reconnected clients, wait for replay, fail dropped sends loudly +- `fb61c59d` [PIR #1198] towerStop waits for process exit, escalating to SIGKILL after 8s +- `58fc3526` [PIR #1198] Tests: close emission, in-place reconnect, grace window, writable guards +- `41398bff` [PIR #1198] Collapse close-emission flags into a single _closePending recorded by cleanup +- `54ef5bd4` [PIR #1198] Extract towerStop wait timing into named constants + +## Test Results + +- `pnpm build` (workspace root): pass +- `pnpm test` (packages/codev): pass — 3532 tests, 0 failures (13 new tests) +- New coverage: error-path close emission (black-box: oversized frame header → parser error → `'close'` fires), intentional disconnect stays silent, write/resize delivery booleans, in-place reconnect success / dead-path / exhaustion (real Unix sockets against an in-process shellper), grace-window teardown and re-attach cancellation, `writable` guard, 503 send path, send-buffer hold-then-drop. +- One pre-existing test (`tower-shellper-integration`: "emits exit with code -1 on unexpected disconnect") asserted the old immediate-teardown behavior and was rewritten for the grace-window semantics. +- Manual verification: reviewed and dev-approval-gated by the human on the running worktree; the definitive live test (Tower restart on the patched build restoring the currently zombified architects) runs at deploy time — see How to Test Locally. + +## Architecture Updates + +Routed to **COLD** (`codev/resources/arch.md`, Shellper Process Architecture): a new "Connection-loss recovery (#1198)" subsection documenting the recovery pipeline (close-emission contract, in-place reconnect, grace-timer teardown, re-attach, write observability) and its invariant: never tear down a terminal's registry/DB state on a raw socket close without first checking whether the shellper process is alive. Nothing routed to HOT — `arch-critical.md` is at its 10-fact cap and this is subsystem-internal behavior, not a cross-cutting decision-changing fact; no existing hot fact is weaker than it. + +## Lessons Learned Updates + +Routed to **COLD** (`codev/resources/lessons-learned.md`, Debugging and Root Cause Analysis): two entries — (1) lifecycle emissions must be decided from state captured at the teardown transition, not re-read later, and un-swallowing a long-suppressed event requires auditing every consumer first (the naive fix would have converted zombies into permanently orphaned sessions); (2) success-shaped logging ("Message sent" with no delivery signal) turns an outage invisible — delivery paths need an observable failure contract at the API boundary, not just logs. Nothing routed to HOT (cap full; these are narrower than the current ten). + +## Things to Look At During PR Review + +- **Live-deploy incident (2026-07-19, fixed in this branch — read this first).** The first install of this branch surfaced the *true* trigger of the whole issue and a critical amplification in my recovery design. Two long-lived architects (shannon/app, codev architect) had **replay buffers over MAX_FRAME_SIZE (17.6MB / 17.7MB > 16MB)**. The shellper sends that as one REPLAY frame; the parser treated it as a fatal stream error — **deterministically, on every connect**. On the old code that error was swallowed → the original zombies (the stop/start-overlap theory was wrong; same terminal zombified every restart because the failure is deterministic). On this branch pre-fix, recovery retried the deterministic failure, exhausted its rounds, took the dead path, **unlinked the live sockets** → `killOrphanedShellpers` then killed both live shellpers and their Claude conversations. Fix, three layers: (1) `FrameParser` now **discards** oversized frames incrementally (`'frame-skipped'` event) and keeps the stream alive — mandatory Tower-side because running shellpers are old binaries; (2) `removeDeadSession` **never unlinks a live process's socket**, so any future deterministic failure degrades to a recoverable orphan instead of feeding the kill sweeper; (3) new shellpers cap replay to `REPLAY_PAYLOAD_MAX` (8MB tail; client resolves skipped replay as empty and viewers repaint via the post-connect resize nudge). Regression tests at all three layers (client-level oversized-REPLAY survival test reproduces the incident shape byte-for-byte). Verified post-fix on the live fleet: all 20 surviving sessions show live-Tower fds and non-empty output buffers. +- **Post-incident follow-ups (both fixed in-branch).** (1) *UI interactivity regression*: restoring the replay path meant adoption seeded up to 16MB of newline-free history into each ring buffer, and every viewer attach shipped that whole payload to xterm.js in one bracketed write (multi-second parse; webview stalls needing a reload). Adoption now caps the ring seed to the most recent 1MB (`RING_SEED_MAX_BYTES`, logged when trimming); the shellper retains full history and TUIs repaint via the resize nudge. (2) *Architect review finding (PID-reuse leak, Medium)*: `removeDeadSession` gated socket unlink on `isProcessAlive` alone, so a reused PID would leak a dead shellper's socket/log. The unlink decision is now async best-effort (`cleanupDeadSessionSocket`) using the same alive + start-time guard as reconnection (`isShellperProcessCurrent`, shared with `canReachShellper`); `removeDeadSession`'s map mutation stays synchronous. +- **Consultation round 2 (human-requested, on the post-incident diff)**: claude APPROVE; codex COMMENT with one accepted note — creation-time attach also raced the REPLAY frame (real for fast-starting children). Fixed with zero creation latency: new shellpers always send a REPLAY frame even when empty, and all four creation sites await it. Codex's second note (plan frontmatter) rebutted in the rebuttals file: porch-driven artifacts carry their approval in `status.yaml` gate history, not retroactive frontmatter. +- **Consultation finding (codex, REQUEST_CHANGES — fixed)**: `attachShellper()` opened the disk-log fd unconditionally; with re-attach now a routine recovery step, every reconnect leaked one append handle when disk logging is enabled. Fixed by guarding the open on `logFd === null` (`pty-session.ts`; `cleanupShellper()` closes and nulls the fd, so post-teardown attaches still reopen). Regression test pins it: `pty-session-attach.test.ts` "does not reopen the disk log when a recovery re-attach arrives" (attach → re-attach → one open; detach → attach → second open). PIR's consultation is single-pass, so this fix was not independently re-reviewed — worth a human glance at the guard. The claude consultation verdict was APPROVE. +- **The recovery loop guards** (`session-manager.ts`, `recoverSession`): the 3-attempt/3-round caps with the 30s stability reset, and the `session.client !== client` stale-wiring checks that prevent a replaced client's events from re-triggering recovery. +- **Grace-timer interplay** (`pty-session.ts`): the 15s `SHELLPER_CLOSE_GRACE_MS` teardown deferral is sized above SessionManager's worst-case recovery round (~4s); `attachShellper` cancels it. A genuinely dead terminal now reports `exited` up to 15s later than before (writes to it fail loudly during the window via the `writable` guard). +- **Re-attach passes empty replay deliberately** (`tower-server.ts`): the ring buffer already holds session history; replaying would duplicate output. Known limitation: bytes the PTY emitted during the disconnection window (seconds) are not spliced in — a full-screen TUI repaints past this; a plain shell's scrollback genuinely misses those lines. +- **`findByShellperSessionId`** (`pty-manager.ts`): needed because the create flow keys SessionManager by a fresh UUID while adoption keys by terminal id. +- **Known gaps, deliberately out of scope, for follow-up issues**: (a) PING/PONG heartbeat detection for a socket that hangs open without erroring (issue #1198 item 5, first half); (b) `connect()` has no handshake timeout — a shellper that accepts but never sends WELCOME would hang a recovery attempt (pre-existing exposure, shared with the adoption path; fails visible via the grace timer, not as a zombie); (c) possible enum consolidation of the client's connection-lifecycle state, best done together with (a). + +## How to Test Locally + +- **View diff**: VSCode sidebar → right-click builder pir-1198 → **Review Diff** +- **Run dev**: `afx dev pir-1198`, or for the definitive live test install this branch's build: `cd .builders/pir-1198 && pnpm build && pnpm -w run local-install` +- **What to verify** (maps to the plan's Test Plan): + - After the Tower restart, every persistent terminal's `GET /api/terminals/:id/output` returns `total >= 1` immediately (previously blank-until-poked); the previously zombified architects respond to typed input and `afx send`. + - `afx tower stop` returns only after the process exits (`lsof -i :4100` is empty when it returns); `stop && start` shows no port-in-use retries. + - Kill a disposable terminal's shellper with SIGKILL, then `afx send` to it: the send errors with `TERMINAL_NOT_WRITABLE` and the Tower log shows `Message DROPPED`, never `Message sent`. + - Tower log after any connection blip shows `connection lost unexpectedly` followed by `re-established` / `re-attached` (recovery) or an ERROR line (genuine death) — never silence. diff --git a/codev/state/pir-1198_thread.md b/codev/state/pir-1198_thread.md new file mode 100644 index 000000000..9aa55bbe6 --- /dev/null +++ b/codev/state/pir-1198_thread.md @@ -0,0 +1,54 @@ +# Builder thread: pir-1198 + +PIR builder for issue #1198 (shellper reconnect error swallowed, terminal becomes silent zombie). + +## Plan phase + +- Verified every claim in the issue against the code. All confirmed: + - `shellper-client.ts` `cleanup()` clears `_connected` before the socket `'close'` event fires, so the `wasConnected` guard suppresses `'close'` on every error-path close. + - `session-manager.ts` close handlers (both createSession and reconnectSession paths) are the only code that logs "shellper disconnected unexpectedly" and removes the dead session. They never run on error-path closes. + - `'session-error'` has zero consumers outside session-manager.ts (grep across packages/codev and packages/core). + - `write()`/`resize()` are silent no-ops when `_connected` is false; the message router logs "Message sent" regardless. + - `towerStop` sends SIGTERM and returns without waiting. +- Important extra finding: fixing the close emission ALONE is hazardous. The unexpected-close path calls `removeDeadSession` (unlinks the LIVE shellper's socket file) and PtySession's close handler emits 'exit', whose tower-side handler calls `deleteTerminalSession` (deletes the SQLite row). A transient socket error against a healthy shellper would permanently orphan a live Claude conversation. So the plan includes reconnect-in-place (bounded retry) before declaring the session dead. This is also what the issue's item 5 suggests ("re-run the reconnect for that session"). +- Second extra finding (matches architect's fleet-sweep comment): both adoption call sites read `client.getReplayData()` synchronously right after `connect()` resolves, racing the REPLAY frame which often arrives in a later socket read. `waitForReplay()` exists and is unused there. This explains the blank-until-poked healthy terminals (6 false positives in the sweep). +- attach.ts (terminal-mode client) has the same swallowed-close symptom and gets fixed for free. + +Plan written to codev/plans/1198-shellper-reconnect-error-is-sw.md. Waiting at plan-approval gate. + +## Implement phase + +Plan approved as written (architect confirmed the core = swallowed close + session-error consumer; heartbeat detection stays a follow-up). Implementation landed in six commits: + +1. shellper-client.ts: `_everConnected` + `_intentionalDisconnect` flags; error-path closes now emit 'close'; write/resize return boolean. +2. session-manager.ts: wiring unified into `wireClientEvents`; unexpected close triggers `recoverSession` (up to 3 attempts, 500ms/1s/2s backoff, PID + start-time + socket-file preflight, 3-round cap with 30s stability reset) before the historical dead path; new 'session-reconnected' event. pty-manager.ts gained `findByShellperSessionId` (create flow keys SessionManager by a UUID distinct from the terminal id; adoption keys by terminal id — the helper handles both). +3. pty-session.ts: unexpected-close teardown deferred behind a 15s grace timer (`SHELLPER_CLOSE_GRACE_MS`), cancelled by attachShellper; `writable` getter; boolean write/resize. +4. Tower: session-error now logged at ERROR; session-reconnected re-attaches the replacement client (empty replay to avoid duplicating ring-buffer content); both adoption sites use `waitForReplay()` instead of racing the REPLAY frame; send router returns 503 TERMINAL_NOT_WRITABLE + logs "Message DROPPED" instead of "Message sent"; send-buffer holds messages while a session is unwritable and only drops (loudly) at max age. +5. tower.ts: towerStop polls for process exit (8s), SIGKILL escalation. + +Notable test change: tower-shellper-integration's "emits exit with code -1 on unexpected disconnect" was asserting the OLD immediate-teardown behavior; rewritten for the grace-window semantics plus a re-attach-cancels-teardown case. + +## Dev-approval gate iteration + +Architect probed the design hard at the gate. Two code changes came out of it: +- The `_connected`/`_everConnected`/`_intentionalDisconnect` trio collapsed into a single `_closePending` recorded inside `cleanup(intentional)` — the reviewer's "these feel like duplicates" instinct was right; capturing the decision at the transition point is both smaller and clearer (41398bff). +- towerStop's 8000/200 literals extracted to STOP_EXIT_TIMEOUT_MS / STOP_CHECK_INTERVAL_MS (54ef5bd4). +Also surfaced honestly at the gate: replay-gap loss on re-attach, no handshake timeout in connect() (pre-existing), tuning numbers are heuristics, incident trigger (stop/start overlap) is probable-not-proven. + +## Review phase + +Review file written; arch.md gained "Connection-loss recovery (#1198)" under Shellper Process Architecture; lessons-learned.md gained two entries (transition-time state capture / audit consumers before un-swallowing an event; success-shaped logging). HOT tier untouched (at cap; nothing here displaces the current ten). Follow-up candidates for the architect to file: heartbeat detection, connect() handshake timeout, lifecycle enum consolidation. + +## Live-deploy incident at the pr gate (2026-07-19) + +First install of the branch KILLED two architects (shannon/app pid 4509, codev architect pid 6348). Log-driven RCA: their replay buffers exceeded MAX_FRAME_SIZE (17.6/17.7MB > 16MB) — the parser threw on the REPLAY frame deterministically on every connect. THIS was the original zombie trigger all along (not the stop/start overlap; explains why the same terminal re-zombified every restart). My recovery loop retried the deterministic failure, gave up after 3 rounds, dead path unlinked the LIVE sockets, and killOrphanedShellpers killed both processes. Fix: parser discards oversized frames ('frame-skipped') instead of erroring (required Tower-side: running shellpers are old binaries); removeDeadSession preserves a live process's socket; new shellpers cap replay to an 8MB tail. Lesson recorded: retry loops assume transience — classify the first failure, and the give-up path must preserve state, not destroy it. Post-fix fleet check: all 20 surviving sessions have live fds and non-empty buffers. The two killed conversations are unrecoverable. + +PR #1204 opened. Consult (single pass): claude=APPROVE, codex=REQUEST_CHANGES. Codex's finding was a real regression: attachShellper() leaked one disk-log fd per recovery re-attach. Fixed (guard open on logFd===null) + regression test, documented in review + rebuttal file; consult validated the consult-trusting lesson once again. Also caught at final worktree check: the exitInfo.code ?? -1 type fix had been verified by every build but never staged — committed in 0db7c7b7. Waiting at pr gate. + +## Post-incident hardening at the pr gate + +Two more findings, both fixed in 65526295: +- Human-reported interactivity regression: restoring the replay path seeded up to 16MB of newline-free history into ring buffers; every viewer attach shipped it to xterm in one write (multi-second parses, webview restarts). Adoption now caps the ring seed to a 1MB tail (RING_SEED_MAX_BYTES, logged when trimming). The old "fast" behavior was the replay-drop bug acting as accidental UI protection. +- Architect review finding (Medium, PID reuse): removeDeadSession gated unlink on isProcessAlive alone. Now an async best-effort cleanupDeadSessionSocket using the shared alive+start-time guard (isShellperProcessCurrent); map mutation stays sync. + +Follow-up filed at the user's direction: #1205 (replay buffer redesign: store the screen, not the stream — the root condition behind the 16MB+ frames). PR #1204's caps documented there as containment, not cure. diff --git a/packages/codev/src/agent-farm/__tests__/send-buffer.test.ts b/packages/codev/src/agent-farm/__tests__/send-buffer.test.ts index a313547d0..e3c174d86 100644 --- a/packages/codev/src/agent-farm/__tests__/send-buffer.test.ts +++ b/packages/codev/src/agent-farm/__tests__/send-buffer.test.ts @@ -27,10 +27,11 @@ function makeMsg(sessionId: string, overrides?: Partial): Buffe }; } -function makeSession(idle: boolean, composing = false): PtySession { +function makeSession(idle: boolean, composing = false, writable = true): PtySession { return { isUserIdle: () => idle, composing, + writable, write: vi.fn(), } as unknown as PtySession; } @@ -57,6 +58,45 @@ describe('SendBuffer', () => { expect(buf.sessionCount).toBe(2); }); + it('holds messages for an unwritable session, then drops loudly at max age (#1198)', () => { + const session = makeSession(true, false, false); // idle but unwritable + const deliver = vi.fn().mockReturnValue(0); + const log = vi.fn(); + + buf.start(() => session, deliver, log); + buf.enqueue(makeMsg('sess-1')); + + // Idle would normally deliver, but the shellper connection is down: + // the message is held, not written into the void. + vi.advanceTimersByTime(500); + expect(deliver).not.toHaveBeenCalled(); + expect(buf.pendingCount).toBe(1); + + // Still down at max age: dropped with an ERROR, never "delivered". + vi.advanceTimersByTime(10_000); + expect(deliver).not.toHaveBeenCalled(); + expect(buf.pendingCount).toBe(0); + expect(log).toHaveBeenCalledWith('ERROR', expect.stringContaining('Dropping')); + }); + + it('delivers held messages once the session becomes writable again (#1198)', () => { + const session = makeSession(true, false, false) as PtySession & { writable: boolean }; + const deliver = vi.fn().mockReturnValue(0); + const log = vi.fn(); + + buf.start(() => session, deliver, log); + buf.enqueue(makeMsg('sess-1')); + + vi.advanceTimersByTime(500); + expect(deliver).not.toHaveBeenCalled(); + + // In-place reconnect landed: connection is back before max age. + session.writable = true; + vi.advanceTimersByTime(500); + expect(deliver).toHaveBeenCalledTimes(1); + expect(buf.pendingCount).toBe(0); + }); + it('delivers messages when session is idle', () => { const session = makeSession(true); const deliver = vi.fn().mockReturnValue(0); diff --git a/packages/codev/src/agent-farm/__tests__/tower-routes.test.ts b/packages/codev/src/agent-farm/__tests__/tower-routes.test.ts index a82b236c2..35f9fa209 100644 --- a/packages/codev/src/agent-farm/__tests__/tower-routes.test.ts +++ b/packages/codev/src/agent-farm/__tests__/tower-routes.test.ts @@ -1241,7 +1241,7 @@ describe('tower-routes', () => { agent: 'architect', }); mockGetTerminalManager.mockReturnValue({ - getSession: () => ({ write: vi.fn(), pid: 1234, isUserIdle: () => true, composing: false }), + getSession: () => ({ write: vi.fn(), pid: 1234, writable: true, isUserIdle: () => true, composing: false }), listSessions: () => [], }); const req = makeReq('POST', '/api/send'); @@ -1259,7 +1259,7 @@ describe('tower-routes', () => { agent: 'architect', }); mockGetTerminalManager.mockReturnValue({ - getSession: () => ({ write: vi.fn(), pid: 1234, isUserIdle: () => true, composing: false }), + getSession: () => ({ write: vi.fn(), pid: 1234, writable: true, isUserIdle: () => true, composing: false }), listSessions: () => [], }); const req = makeReq('POST', '/api/send'); @@ -1278,7 +1278,7 @@ describe('tower-routes', () => { }); const mockWrite = vi.fn(); mockGetTerminalManager.mockReturnValue({ - getSession: () => ({ write: mockWrite, pid: 1234, isUserIdle: () => true, composing: false }), + getSession: () => ({ write: mockWrite, pid: 1234, writable: true, isUserIdle: () => true, composing: false }), listSessions: () => [], }); const req = makeReq('POST', '/api/send'); @@ -1295,6 +1295,28 @@ describe('tower-routes', () => { expect(mockWrite).toHaveBeenCalled(); }); + it('returns 503 TERMINAL_NOT_WRITABLE instead of a false success when the shellper connection is down (#1198)', async () => { + mockParseJsonBody.mockResolvedValue({ to: 'architect', message: 'hello', workspace: '/tmp/ws' }); + mockResolveTarget.mockReturnValue({ + terminalId: 'term-zombie', + workspacePath: '/tmp/ws', + agent: 'architect', + }); + const mockWrite = vi.fn(); + mockGetTerminalManager.mockReturnValue({ + getSession: () => ({ write: mockWrite, pid: 1234, writable: false, isUserIdle: () => true, composing: false }), + listSessions: () => [], + }); + const req = makeReq('POST', '/api/send'); + const { res, statusCode, body } = makeRes(); + + await handleRequest(req, res, makeCtx()); + expect(statusCode()).toBe(503); + const parsed = JSON.parse(body()); + expect(parsed.error).toBe('TERMINAL_NOT_WRITABLE'); + expect(mockWrite).not.toHaveBeenCalled(); + }); + it('returns deferred:true when user is actively typing (Spec 403)', async () => { mockParseJsonBody.mockResolvedValue({ to: 'architect', message: 'hello', workspace: '/tmp/ws' }); mockResolveTarget.mockReturnValue({ @@ -1304,7 +1326,7 @@ describe('tower-routes', () => { }); const mockWrite = vi.fn(); mockGetTerminalManager.mockReturnValue({ - getSession: () => ({ write: mockWrite, pid: 1234, isUserIdle: () => false, composing: false }), + getSession: () => ({ write: mockWrite, pid: 1234, writable: true, isUserIdle: () => false, composing: false }), listSessions: () => [], }); const req = makeReq('POST', '/api/send'); @@ -1332,7 +1354,7 @@ describe('tower-routes', () => { }); const mockWrite = vi.fn(); mockGetTerminalManager.mockReturnValue({ - getSession: () => ({ write: mockWrite, pid: 1234, isUserIdle: () => false, composing: true }), + getSession: () => ({ write: mockWrite, pid: 1234, writable: true, isUserIdle: () => false, composing: true }), listSessions: () => [], }); const req = makeReq('POST', '/api/send'); @@ -1356,7 +1378,7 @@ describe('tower-routes', () => { }); const mockWrite = vi.fn(); mockGetTerminalManager.mockReturnValue({ - getSession: () => ({ write: mockWrite, pid: 1234, isUserIdle: () => true, composing: false }), + getSession: () => ({ write: mockWrite, pid: 1234, writable: true, isUserIdle: () => true, composing: false }), listSessions: () => [], }); const req = makeReq('POST', '/api/send'); @@ -1383,7 +1405,7 @@ describe('tower-routes', () => { }); const mockWrite = vi.fn(); mockGetTerminalManager.mockReturnValue({ - getSession: () => ({ write: mockWrite, pid: 1234, isUserIdle: () => true, composing: false }), + getSession: () => ({ write: mockWrite, pid: 1234, writable: true, isUserIdle: () => true, composing: false }), listSessions: () => [], }); const req = makeReq('POST', '/api/send'); @@ -1408,7 +1430,7 @@ describe('tower-routes', () => { // Bugfix #492: composing gets stuck true after non-Enter keystrokes. // Idle threshold alone is sufficient — deliver immediately. mockGetTerminalManager.mockReturnValue({ - getSession: () => ({ write: mockWrite, pid: 1234, isUserIdle: () => true, composing: true }), + getSession: () => ({ write: mockWrite, pid: 1234, writable: true, isUserIdle: () => true, composing: true }), listSessions: () => [], }); const req = makeReq('POST', '/api/send'); diff --git a/packages/codev/src/agent-farm/__tests__/tower-terminals.test.ts b/packages/codev/src/agent-farm/__tests__/tower-terminals.test.ts index 6474df320..47a43597a 100644 --- a/packages/codev/src/agent-farm/__tests__/tower-terminals.test.ts +++ b/packages/codev/src/agent-farm/__tests__/tower-terminals.test.ts @@ -529,6 +529,7 @@ describe('tower-terminals', () => { const reconnectGate = new Promise((r) => { releaseReconnect = r; }); const makeClient = () => ({ getReplayData: () => Buffer.alloc(0), + waitForReplay: async () => Buffer.alloc(0), connected: true, connect: vi.fn(), disconnect: vi.fn(), write: vi.fn(), resize: vi.fn(), signal: vi.fn(), spawn: vi.fn(), ping: vi.fn(), setReplayData: vi.fn(), @@ -678,6 +679,7 @@ describe('tower-terminals', () => { // Mock a shellper client that returns successfully const makeClient = () => ({ getReplayData: () => Buffer.alloc(0), + waitForReplay: async () => Buffer.alloc(0), connected: true, connect: vi.fn(), disconnect: vi.fn(), diff --git a/packages/codev/src/agent-farm/commands/tower.ts b/packages/codev/src/agent-farm/commands/tower.ts index 9819fdd22..24e91ac0c 100644 --- a/packages/codev/src/agent-farm/commands/tower.ts +++ b/packages/codev/src/agent-farm/commands/tower.ts @@ -22,6 +22,11 @@ const LOG_FILE = resolve(AGENT_FARM_DIR, 'tower.log'); const STARTUP_TIMEOUT_MS = 30000; const STARTUP_CHECK_INTERVAL_MS = 200; +// Stop settings (#1198): how long towerStop waits for the SIGTERMed process +// to exit before escalating to SIGKILL, and how often it polls. +const STOP_EXIT_TIMEOUT_MS = 8000; +const STOP_CHECK_INTERVAL_MS = 200; + export interface TowerStartOptions { port?: number; wait?: boolean; // Defaults to true. Set false for fire-and-forget startup. @@ -353,6 +358,36 @@ export async function towerStop(options: TowerStopOptions = {}): Promise { } } + // #1198: wait for the processes to actually exit before returning. + // Returning right after SIGTERM let `afx tower stop && afx tower start` + // overlap the old Tower's shellper teardown with the new Tower's adoption + // pass — an unnecessary race window during every restart. + const isAlive = (pid: number): boolean => { + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } + }; + const deadline = Date.now() + STOP_EXIT_TIMEOUT_MS; + let survivors = pids.filter(isAlive); + while (survivors.length > 0 && Date.now() < deadline) { + await new Promise((resolve) => setTimeout(resolve, STOP_CHECK_INTERVAL_MS)); + survivors = survivors.filter(isAlive); + } + + if (survivors.length > 0) { + for (const pid of survivors) { + try { + process.kill(pid, 'SIGKILL'); + } catch { + // Exited between the check and the kill + } + } + logger.warn(`Tower did not exit within ${STOP_EXIT_TIMEOUT_MS / 1000}s; sent SIGKILL to PID${survivors.length > 1 ? 's' : ''} ${survivors.join(', ')}`); + } + if (stopped > 0) { logger.success(`Tower stopped (${stopped} process${stopped > 1 ? 'es' : ''}: PIDs ${pids.join(', ')})`); } diff --git a/packages/codev/src/agent-farm/servers/send-buffer.ts b/packages/codev/src/agent-farm/servers/send-buffer.ts index 90ba2ace0..cba3c3d61 100644 --- a/packages/codev/src/agent-farm/servers/send-buffer.ts +++ b/packages/codev/src/agent-farm/servers/send-buffer.ts @@ -96,6 +96,20 @@ export class SendBuffer { const maxAgeExceeded = messages.some(m => now - m.timestamp >= this.maxBufferAgeMs); const isIdle = session.isUserIdle(this.idleThresholdMs); + // #1198: writes to a session whose shellper connection is down are + // dropped silently. Hold the messages while the connection recovers + // (in-place reconnect takes a few seconds); if it never does, drop + // loudly instead of logging a successful delivery. + if (!session.writable) { + if (forceAll || maxAgeExceeded) { + if (this.log) { + this.log('ERROR', `Dropping ${messages.length} buffered message(s) for unwritable session ${sessionId.slice(0, 8)}... (shellper connection down)`); + } + this.buffers.delete(sessionId); + } + continue; + } + // Deliver when: forced, user idle, or max age exceeded. // Bugfix #492: removed composing check — it gets stuck true after non-Enter // keystrokes (Ctrl+C, arrows, Tab), causing messages to wait 60s max age. diff --git a/packages/codev/src/agent-farm/servers/tower-instances.ts b/packages/codev/src/agent-farm/servers/tower-instances.ts index e8e43d183..8a03ad0bc 100644 --- a/packages/codev/src/agent-farm/servers/tower-instances.ts +++ b/packages/codev/src/agent-farm/servers/tower-instances.ts @@ -577,9 +577,10 @@ export async function launchInstance(workspacePath: string): Promise<{ success: ...defaultSessionOptions({ restartOnExit: true, restartDelay: 2000, maxRestarts: 50 }), }); - // Get replay data and shellper info - const replayData = client.getReplayData() ?? Buffer.alloc(0); + // Read session info BEFORE awaiting replay: an instantly-exiting + // child's EXIT frame can remove the session during the await (#1198) const shellperInfo = _deps.shellperManager.getSessionInfo(sessionId)!; + const replayData = await client.waitForReplay(); // #1198: fresh shellpers always send REPLAY (possibly empty) // Create a PtySession backed by the shellper client const session = manager.createSessionRaw({ @@ -1056,8 +1057,10 @@ export async function addArchitect( ...defaultSessionOptions({ restartOnExit: true, restartDelay: 2000, maxRestarts: 50 }), }); - const replayData = client.getReplayData() ?? Buffer.alloc(0); + // Read session info BEFORE awaiting replay: an instantly-exiting + // child's EXIT frame can remove the session during the await (#1198) const shellperInfo = _deps.shellperManager.getSessionInfo(shellperSessionId)!; + const replayData = await client.waitForReplay(); // #1198: fresh shellpers always send REPLAY (possibly empty) const session = manager.createSessionRaw({ label: `Architect (${name})`, cwd: workspacePath }); const ptySession = manager.getSession(session.id); diff --git a/packages/codev/src/agent-farm/servers/tower-routes.ts b/packages/codev/src/agent-farm/servers/tower-routes.ts index 9f7ef32ae..4286c9f1b 100644 --- a/packages/codev/src/agent-farm/servers/tower-routes.ts +++ b/packages/codev/src/agent-farm/servers/tower-routes.ts @@ -606,8 +606,11 @@ async function handleTerminalCreate( cols: cols || DEFAULT_COLS, }); - const replayData = client.getReplayData() ?? Buffer.alloc(0); + // Read session info BEFORE awaiting replay: an instantly-exiting + // child's EXIT frame can remove the session from the manager during + // the await, and this lookup must not miss (#1198). const shellperInfo = shellperManager.getSessionInfo(sessionId)!; + const replayData = await client.waitForReplay(); // #1198: fresh shellpers always send REPLAY (possibly empty); awaiting avoids racing early child output const session = manager.createSessionRaw({ label: label || `terminal-${sessionId.slice(0, 8)}`, @@ -1316,6 +1319,19 @@ async function handleSend( return; } + // #1198: a session whose shellper connection died still reports status + // 'running', but every write to it is dropped. Fail the send loudly + // instead of logging "Message sent" for a message that went nowhere. + if (!session.writable) { + ctx.log('ERROR', `Message DROPPED: ${from ?? 'unknown'} → ${result.agent} (terminal ${result.terminalId.slice(0, 8)}...): terminal not writable (shellper connection down)`); + res.writeHead(503, { 'Content-Type': 'application/json' }); + res.end(JSON.stringify({ + error: 'TERMINAL_NOT_WRITABLE', + message: `Terminal for '${result.agent}' is not accepting input (its process connection is down). Retry shortly; if this persists, check Tower logs.`, + })); + return; + } + // Format the message based on sender/target const isArchitectTarget = result.agent === 'architect'; let formattedMessage: string; @@ -2034,8 +2050,11 @@ async function handleWorkspaceShellCreate( ...defaultSessionOptions(), }); - const replayData = client.getReplayData() ?? Buffer.alloc(0); + // Read session info BEFORE awaiting replay: an instantly-exiting + // child's EXIT frame can remove the session from the manager during + // the await, and this lookup must not miss (#1198). const shellperInfo = shellperManager.getSessionInfo(sessionId)!; + const replayData = await client.waitForReplay(); // #1198: fresh shellpers always send REPLAY (possibly empty); awaiting avoids racing early child output const session = manager.createSessionRaw({ label: `Shell ${shellId.replace('shell-', '')}`, diff --git a/packages/codev/src/agent-farm/servers/tower-server.ts b/packages/codev/src/agent-farm/servers/tower-server.ts index 8e6412cad..55e902e69 100644 --- a/packages/codev/src/agent-farm/servers/tower-server.ts +++ b/packages/codev/src/agent-farm/servers/tower-server.ts @@ -376,6 +376,33 @@ server.listen(port, bindHost, async () => { nodeExecutable: process.execPath, logger: (msg: string) => log('INFO', msg), }); + + // #1198: a shellper connection that dies must leave a trace. Before this + // subscription existed, 'session-error' had no consumer anywhere, so a + // failed reconnect was completely invisible in the Tower log. + shellperManager.on('session-error', (sessionId: string, err: Error) => { + log('ERROR', `Shellper session ${sessionId}: ${err.message}`); + }); + + // #1198: SessionManager re-established a session's connection in place + // after an unexpected socket close. Re-attach the replacement client to the + // PtySession so viewer I/O resumes (attachShellper is idempotent and + // cancels the pending close-grace teardown). Replay is deliberately empty: + // the ring buffer already holds the session history. + shellperManager.on('session-reconnected', (sessionId: string, client: import('../../terminal/shellper-client.js').IShellperClient) => { + const ptySession = getTerminalManager().findByShellperSessionId(sessionId); + if (!ptySession) { + log('WARN', `Shellper session ${sessionId} reconnected but no matching terminal session found`); + return; + } + const info = shellperManager!.getSessionInfo(sessionId); + let pid = -1; + if (info) { + pid = info.pid; + } + ptySession.attachShellper(client, Buffer.alloc(0), pid, ptySession.shellperSessionId ?? sessionId); + log('INFO', `Shellper session ${sessionId} re-attached to terminal ${ptySession.id}`); + }); const staleCleaned = await shellperManager.cleanupStaleSockets(); if (staleCleaned > 0) { log('INFO', `Cleaned up ${staleCleaned} stale shellper socket(s)`); diff --git a/packages/codev/src/agent-farm/servers/tower-terminals.ts b/packages/codev/src/agent-farm/servers/tower-terminals.ts index ad5fc330a..9417af6c1 100644 --- a/packages/codev/src/agent-farm/servers/tower-terminals.ts +++ b/packages/codev/src/agent-farm/servers/tower-terminals.ts @@ -26,6 +26,23 @@ import { TerminalManager, DEFAULT_DISK_LOG_MAX_BYTES } from '../../terminal/inde * Extract shellper session UUID from socket path (Spec 468). * Socket format: shellper-.sock */ +/** + * #1198: cap the replay seeded into an adopted PtySession's ring buffer. + * A long-lived full-screen TUI session's replay can be many MB of + * newline-free history; seeding it whole means every subsequent viewer + * attach ships the entire payload to xterm.js in one bracketed write + * (multi-second parse, webview stalls). Seed only the most recent bytes: + * the client's post-connect resize nudge repaints full-screen apps, and the + * shellper retains the full history. + */ +const RING_SEED_MAX_BYTES = 1024 * 1024; // 1MB + +function capRingSeed(replayData: Buffer, sessionId: string): Buffer { + if (replayData.length <= RING_SEED_MAX_BYTES) return replayData; + _deps?.log('INFO', `Session ${sessionId} replay is ${replayData.length} bytes; seeding the most recent ${RING_SEED_MAX_BYTES}`); + return replayData.subarray(replayData.length - RING_SEED_MAX_BYTES); +} + function extractShellperSessionId(socketPath: string | null): string | null { if (!socketPath) return null; const match = path.basename(socketPath).match(/^shellper-(.+)\.sock$/); @@ -747,7 +764,10 @@ async function _reconcileTerminalSessionsInner(): Promise { const workspacePath = dbSession.workspace_path; const sessionCwd = dbSession.cwd ?? workspacePath; - const replayData = client.getReplayData() ?? Buffer.alloc(0); + // #1198: the REPLAY frame often arrives in a separate socket read after + // WELCOME, so a synchronous getReplayData() here raced it and adopted + // terminals rendered blank until new output arrived. Wait briefly for it. + const replayData = capRingSeed(await client.waitForReplay(), dbSession.id); const label = dbSession.label || (dbSession.type === 'architect' ? 'Architect' : (dbSession.role_id || 'unknown')); // Create a PtySession backed by the reconnected shellper client. @@ -958,7 +978,9 @@ export async function getTerminalsForWorkspace( restartOptions, ); if (client) { - const replayData = client.getReplayData() ?? Buffer.alloc(0); + // #1198: wait for the REPLAY frame instead of racing it (see the + // matching change in the startup adoption pass above). + const replayData = capRingSeed(await client.waitForReplay(), dbSession.id); const label = dbSession.label || (dbSession.type === 'architect' ? 'Architect' : (dbSession.role_id || dbSession.id)); // Reuse the persisted terminal id (#991) so the session keeps its // identity across the reconnect — clients holding `/ws/terminal/` diff --git a/packages/codev/src/terminal/__tests__/pty-session-attach.test.ts b/packages/codev/src/terminal/__tests__/pty-session-attach.test.ts index ff6c6583d..88ada7cb4 100644 --- a/packages/codev/src/terminal/__tests__/pty-session-attach.test.ts +++ b/packages/codev/src/terminal/__tests__/pty-session-attach.test.ts @@ -1,5 +1,8 @@ -import { describe, it, expect } from 'vitest'; +import { describe, it, expect, vi } from 'vitest'; import { EventEmitter } from 'node:events'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; import { PtySession, type PtySessionConfig } from '../pty-session.js'; import type { IShellperClient } from '../shellper-client.js'; @@ -10,10 +13,16 @@ import type { IShellperClient } from '../shellper-client.js'; * re-run onPtyData for every PTY byte (the listener-leak the hardening targets). */ -function makeFakeClient(): IShellperClient { - const emitter = new EventEmitter() as unknown as IShellperClient & { _lastDataAt: number }; +function makeFakeClient(): IShellperClient & { connectedState: boolean } { + const emitter = new EventEmitter() as unknown as IShellperClient & { connectedState: boolean }; // attachShellper reads client.lastDataAt and subscribes data/exit/close. Object.defineProperty(emitter, 'lastDataAt', { get: () => Date.now() }); + // #1198: PtySession.writable and write() consult the client's connection + // state; the fake models it with a mutable flag. + emitter.connectedState = true; + Object.defineProperty(emitter, 'connected', { get: () => emitter.connectedState }); + emitter.write = () => emitter.connectedState; + emitter.resize = () => emitter.connectedState; return emitter; } @@ -77,3 +86,69 @@ describe('PtySession.attachShellper idempotency (#1047 Fix E)', () => { expect(clientA.listenerCount('data')).toBeGreaterThanOrEqual(1); }); }); + +describe('PtySession disk-log handle on re-attach (#1198)', () => { + it('does not reopen the disk log when a recovery re-attach arrives', () => { + const logDir = fs.mkdtempSync(path.join(os.tmpdir(), 'pty-attach-log-')); + const config: PtySessionConfig = { + id: 'sess-log', + command: '', + args: [], + cols: 80, + rows: 24, + cwd: '/tmp', + env: {}, + label: 'test', + logDir, + diskLogEnabled: true, + }; + const session = new PtySession(config); + const openSpy = vi.spyOn(fs, 'openSync'); + const logOpens = () => openSpy.mock.calls.filter((c) => String(c[0]).startsWith(logDir)).length; + + try { + session.attachShellper(makeFakeClient(), Buffer.alloc(0), 1234); + expect(logOpens()).toBe(1); + + // In-place recovery delivers a replacement client. Before the guard, + // this leaked one append handle per reconnect. + session.attachShellper(makeFakeClient(), Buffer.alloc(0), 1234); + expect(logOpens()).toBe(1); + + // After a real teardown the handle is closed, so a fresh attach must + // reopen it. + session.detachShellper(); + session.attachShellper(makeFakeClient(), Buffer.alloc(0), 1234); + expect(logOpens()).toBe(2); + } finally { + openSpy.mockRestore(); + session.detachShellper(); + fs.rmSync(logDir, { recursive: true, force: true }); + } + }); +}); + +describe('PtySession.writable (#1198)', () => { + it('reflects the shellper client connection state, not just session status', () => { + const session = makeSession(); + const client = makeFakeClient(); + session.attachShellper(client, Buffer.alloc(0), 1234); + + expect(session.writable).toBe(true); + expect(session.write('reaches the pty')).toBe(true); + + // The connection dies but the session still reports status 'running': + // this is exactly the zombie state the getter exists to expose. + client.connectedState = false; + expect(session.status).toBe('running'); + expect(session.writable).toBe(false); + expect(session.write('dropped')).toBe(false); + expect(session.resize(100, 50)).toBe(false); + }); + + it('is false with no backing client', () => { + const session = makeSession(); + expect(session.writable).toBe(false); + expect(session.write('nowhere')).toBe(false); + }); +}); diff --git a/packages/codev/src/terminal/__tests__/session-manager.test.ts b/packages/codev/src/terminal/__tests__/session-manager.test.ts index b7230376e..54089e33e 100644 --- a/packages/codev/src/terminal/__tests__/session-manager.test.ts +++ b/packages/codev/src/terminal/__tests__/session-manager.test.ts @@ -957,6 +957,144 @@ describe('SessionManager', () => { }); }); + describe('in-place reconnect on unexpected close (#1198)', () => { + async function startMockShellper(socketPath: string): Promise { + const shellper = new ShellperProcess( + () => new MockPty(), + socketPath, + 100, + ); + await shellper.start('/bin/bash', [], '/tmp', {}, 80, 24); + return shellper; + } + + function makeManager(): SessionManager { + return new SessionManager({ + socketDir, + shellperScript: '/nonexistent/shellper.js', + nodeExecutable: process.execPath, + }); + } + + /** Destroy the Tower-side socket with an error: the production failure mode. */ + function killClientConnection(client: ShellperClient): void { + const socket = (client as unknown as { socket: net.Socket }).socket; + socket.destroy(new Error('transient socket error')); + } + + it('re-establishes the connection while the shellper stays alive', async () => { + const socketPath = path.join(socketDir, 'shellper-recover.sock'); + const shellper = await startMockShellper(socketPath); + cleanupFns.push(() => shellper.shutdown()); + + const manager = makeManager(); + cleanupFns.push(() => manager.shutdown()); + + const startTime = (await getProcessStartTime(process.pid))!; + const client = await manager.reconnectSession('recover-test', socketPath, process.pid, startTime); + expect(client).not.toBeNull(); + + const deadErrors: Error[] = []; + manager.on('session-error', (_id: string, err: Error) => { + if (err.message.includes('Shellper disconnected unexpectedly')) deadErrors.push(err); + }); + const reconnectedPromise = new Promise<{ id: string; newClient: ShellperClient }>((resolve) => { + manager.on('session-reconnected', (id: string, newClient: ShellperClient) => resolve({ id, newClient })); + }); + + killClientConnection(client as ShellperClient); + + const evt = await Promise.race([ + reconnectedPromise, + new Promise((_, reject) => setTimeout(() => reject(new Error('timeout waiting for session-reconnected')), 8000)), + ]); + + expect(evt.id).toBe('recover-test'); + expect(evt.newClient.connected).toBe(true); + expect(manager.listSessions().has('recover-test')).toBe(true); + // The live shellper's socket file must not have been unlinked. + expect(fs.existsSync(socketPath)).toBe(true); + expect(deadErrors).toEqual([]); + }, 15_000); + + it('declares the session dead when the shellper is unreachable', async () => { + const socketPath = path.join(socketDir, 'shellper-recover-dead.sock'); + const shellper = await startMockShellper(socketPath); + cleanupFns.push(() => shellper.shutdown()); + + const manager = makeManager(); + cleanupFns.push(() => manager.shutdown()); + + const startTime = (await getProcessStartTime(process.pid))!; + const client = await manager.reconnectSession('recover-dead', socketPath, process.pid, startTime); + expect(client).not.toBeNull(); + + const deadPromise = new Promise((resolve) => { + manager.on('session-error', (_id: string, err: Error) => { + if (err.message.includes('Shellper disconnected unexpectedly')) resolve(err); + }); + }); + + // Remove the socket file so recovery cannot possibly reach the shellper. + fs.unlinkSync(socketPath); + killClientConnection(client as ShellperClient); + + const err = await Promise.race([ + deadPromise, + new Promise((_, reject) => setTimeout(() => reject(new Error('timeout waiting for session-error')), 8000)), + ]); + + expect(err.message).toContain('Shellper disconnected unexpectedly'); + expect(manager.listSessions().has('recover-dead')).toBe(false); + }, 15_000); + + it('gives up after exhausting reconnect attempts against a broken socket', async () => { + const socketPath = path.join(socketDir, 'shellper-recover-exhaust.sock'); + const shellper = await startMockShellper(socketPath); + cleanupFns.push(() => shellper.shutdown()); + + const manager = makeManager(); + cleanupFns.push(() => manager.shutdown()); + + const startTime = (await getProcessStartTime(process.pid))!; + const client = await manager.reconnectSession('recover-exhaust', socketPath, process.pid, startTime); + expect(client).not.toBeNull(); + + // Swap the healthy shellper for a rogue server that accepts and + // immediately destroys each connection, so every reconnect attempt + // fails its handshake. + await shellper.shutdown(); + try { fs.unlinkSync(socketPath); } catch { /* already gone */ } + const rogue = net.createServer((socket) => socket.destroy()); + rogue.listen(socketPath); + cleanupFns.push(() => new Promise((resolve) => rogue.close(() => resolve()))); + + const deadPromise = new Promise((resolve) => { + manager.on('session-error', (_id: string, err: Error) => { + if (err.message.includes('Shellper disconnected unexpectedly')) resolve(err); + }); + }); + + killClientConnection(client as ShellperClient); + + const err = await Promise.race([ + deadPromise, + new Promise((_, reject) => setTimeout(() => reject(new Error('timeout waiting for session-error')), 12_000)), + ]); + + expect(err.message).toContain('Shellper disconnected unexpectedly'); + expect(manager.listSessions().has('recover-exhaust')).toBe(false); + // #1198 incident hardening: the process behind this session (the test + // process stands in for a live shellper) is alive with a matching + // start time, so the dead declaration must NOT unlink the socket — an + // unlinked socket flags a live shellper for the orphan sweeper's kill + // path. The cleanup decision is async; give a delayed unlink time to + // (wrongly) fire before asserting. + await new Promise((r) => setTimeout(r, 300)); + expect(fs.existsSync(socketPath)).toBe(true); + }, 20_000); + }); + describe('natural exit cleanup (no auto-restart)', () => { it('removes session from map when process exits and restartOnExit is false', async () => { const socketPath = path.join(socketDir, 'shellper-natural-exit.sock'); diff --git a/packages/codev/src/terminal/__tests__/shellper-client.test.ts b/packages/codev/src/terminal/__tests__/shellper-client.test.ts index ec8b93b48..26d9629f7 100644 --- a/packages/codev/src/terminal/__tests__/shellper-client.test.ts +++ b/packages/codev/src/terminal/__tests__/shellper-client.test.ts @@ -6,6 +6,7 @@ import os from 'node:os'; import { FrameType, PROTOCOL_VERSION, + MAX_FRAME_SIZE, createFrameParser, encodeFrame, encodeWelcome, @@ -733,4 +734,115 @@ describe('ShellperClient', () => { expect(warnings).toEqual([]); // No warnings when versions match }); }); + + describe('unexpected close emission (#1198)', () => { + function createCapturingShellper() { + let serverSocket: net.Socket | null = null; + const server = net.createServer((socket) => { + serverSocket = socket; + const parser = createFrameParser(); + socket.pipe(parser); + parser.on('data', (frame: ParsedFrame) => { + if (frame.type === FrameType.HELLO) { + socket.write(encodeWelcome({ version: PROTOCOL_VERSION, pid: 1, cols: 80, rows: 24, startTime: Date.now() })); + } + }); + }); + server.listen(socketPath); + cleanup.push(() => { server.close(); }); + return { getServerSocket: () => serverSocket }; + } + + it('emits close when a post-handshake error path runs cleanup first', async () => { + createCapturingShellper(); + + const client = new ShellperClient(socketPath); + cleanup.push(() => client.disconnect()); + await client.connect(); + + const errors: Error[] = []; + client.on('error', (err: Error) => errors.push(err)); + const closed = new Promise((resolve) => client.once('close', resolve)); + + // Destroy the client-side socket with an error: the production error + // path (socket 'error' → safeEmitError → cleanup) whose subsequent + // 'close' was swallowed before the fix, leaving a silent zombie. + const socket = (client as unknown as { socket: net.Socket }).socket; + socket.destroy(new Error('transient socket error')); + + await closed; + expect(errors.length).toBeGreaterThan(0); + expect(client.connected).toBe(false); + }); + + it('survives an oversized REPLAY frame: skips it, stays connected, keeps parsing (#1198 incident)', async () => { + const { getServerSocket } = createCapturingShellper(); + + const client = new ShellperClient(socketPath); + cleanup.push(() => client.disconnect()); + await client.connect(); + + let closeEmitted = false; + client.on('close', () => { closeEmitted = true; }); + const errors: Error[] = []; + client.on('error', (err: Error) => errors.push(err)); + const skipped: Array<{ type: number; size: number }> = []; + client.on('frame-skipped', (info: { type: number; size: number }) => skipped.push(info)); + const dataPromise = new Promise((resolve) => client.once('data', resolve)); + + // The incident shape: a shellper whose replay outgrew MAX_FRAME_SIZE + // sends it anyway (old shellper binaries still do). Before the fix the + // parser threw, the connection died on every reconnect, and the + // session was eventually orphaned and killed. + const oversized = Buffer.alloc(MAX_FRAME_SIZE + 1, 0x41); + const header = Buffer.alloc(5); + header[0] = FrameType.REPLAY; + header.writeUInt32BE(oversized.length, 1); + const server = getServerSocket()!; + server.write(header); + server.write(oversized); + // A frame after the oversized one must still be parsed. + server.write(encodeData('still alive')); + + const data = await dataPromise; + expect(data.toString()).toBe('still alive'); + expect(skipped).toEqual([{ type: FrameType.REPLAY, size: MAX_FRAME_SIZE + 1 }]); + expect(errors).toEqual([]); + expect(closeEmitted).toBe(false); + expect(client.connected).toBe(true); + // Replay waiters resolve with an empty replay instead of hanging. + const replay = await client.waitForReplay(100); + expect(replay.length).toBe(0); + }, 20_000); + + it('does not emit close on intentional disconnect', async () => { + createCapturingShellper(); + + const client = new ShellperClient(socketPath); + await client.connect(); + + let closeEmitted = false; + client.on('close', () => { closeEmitted = true; }); + + client.disconnect(); + await new Promise((r) => setTimeout(r, 100)); + expect(closeEmitted).toBe(false); + expect(client.connected).toBe(false); + }); + + it('write and resize report delivery: true while connected, false after', async () => { + createCapturingShellper(); + + const client = new ShellperClient(socketPath); + await client.connect(); + + expect(client.write('hello')).toBe(true); + expect(client.resize(100, 50)).toBe(true); + + client.disconnect(); + + expect(client.write('dropped')).toBe(false); + expect(client.resize(100, 50)).toBe(false); + }); + }); }); diff --git a/packages/codev/src/terminal/__tests__/shellper-process.test.ts b/packages/codev/src/terminal/__tests__/shellper-process.test.ts index 7f8d2967d..024aeb4cd 100644 --- a/packages/codev/src/terminal/__tests__/shellper-process.test.ts +++ b/packages/codev/src/terminal/__tests__/shellper-process.test.ts @@ -6,6 +6,7 @@ import os from 'node:os'; import { FrameType, PROTOCOL_VERSION, + REPLAY_PAYLOAD_MAX, encodeHello, encodeData, encodeResize, @@ -244,6 +245,68 @@ describe('ShellperProcess', () => { socket.destroy(); }); + it('sends an empty REPLAY frame when the buffer is empty (#1198)', async () => { + shellper = new ShellperProcess(createMockPty, socketPath); + await shellper.start('/bin/bash', [], '/tmp', {}, 80, 24); + expect(shellper.getReplayData().length).toBe(0); + + const socket = net.createConnection(socketPath); + const parser = createFrameParser(); + socket.pipe(parser); + const frames: ParsedFrame[] = []; + parser.on('data', (f: ParsedFrame) => frames.push(f)); + await new Promise((resolve) => socket.on('connect', () => resolve())); + socket.write(encodeHello({ version: PROTOCOL_VERSION, clientType: 'tower' })); + + const deadline = Date.now() + 5000; + while (!frames.some((f) => f.type === FrameType.REPLAY) && Date.now() < deadline) { + await new Promise((r) => setTimeout(r, 25)); + } + // Empty REPLAY still arrives, so replay waiters resolve immediately + // instead of burning their timeout on every fresh-terminal creation. + const replayFrame = frames.find((f) => f.type === FrameType.REPLAY); + expect(replayFrame).toBeDefined(); + expect(replayFrame!.payload.length).toBe(0); + + socket.destroy(); + }, 10_000); + + it('caps an oversized replay to the most recent bytes on connect (#1198)', async () => { + shellper = new ShellperProcess(createMockPty, socketPath); + await shellper.start('/bin/bash', [], '/tmp', {}, 80, 24); + + // A newline-free stream (full-screen TUI shape) larger than the cap. + const chunk = 'x'.repeat(1024 * 1024); + for (let i = 0; i < 9; i++) { + mockPty.simulateData(chunk); + } + mockPty.simulateData('TAIL-MARKER'); + expect(shellper.getReplayData().length).toBeGreaterThan(REPLAY_PAYLOAD_MAX); + + // Raw socket with the parser attached BEFORE the handshake, so the + // REPLAY frame cannot be swallowed by a helper's internal reads. + const socket = net.createConnection(socketPath); + const parser = createFrameParser(); + socket.pipe(parser); + const frames: ParsedFrame[] = []; + parser.on('data', (f: ParsedFrame) => frames.push(f)); + await new Promise((resolve) => socket.on('connect', () => resolve())); + socket.write(encodeHello({ version: PROTOCOL_VERSION, clientType: 'tower' })); + + const deadline = Date.now() + 10_000; + while (!frames.some((f) => f.type === FrameType.REPLAY) && Date.now() < deadline) { + await new Promise((r) => setTimeout(r, 50)); + } + const replayFrame = frames.find((f) => f.type === FrameType.REPLAY); + expect(replayFrame).toBeDefined(); + // Capped to the most recent bytes: fits the frame budget and ends with + // the newest output. + expect(replayFrame!.payload.length).toBe(REPLAY_PAYLOAD_MAX); + expect(replayFrame!.payload.subarray(-11).toString()).toBe('TAIL-MARKER'); + + socket.destroy(); + }, 20_000); + it('new tower connection replaces old tower connection', async () => { shellper = new ShellperProcess(createMockPty, socketPath); await shellper.start('/bin/bash', [], '/tmp', {}, 80, 24); diff --git a/packages/codev/src/terminal/__tests__/shellper-protocol.test.ts b/packages/codev/src/terminal/__tests__/shellper-protocol.test.ts index 06e6d8545..f3c384241 100644 --- a/packages/codev/src/terminal/__tests__/shellper-protocol.test.ts +++ b/packages/codev/src/terminal/__tests__/shellper-protocol.test.ts @@ -268,20 +268,41 @@ describe('shellper-protocol', () => { expect(frames[0].payload.toString()).toBe('test'); }); - it('rejects oversized frames (>16MB)', async () => { - // Craft a header claiming a huge payload + it('skips oversized frames (>16MB) and keeps parsing (#1198)', async () => { + // #1198: an oversized frame is a per-frame condition, not a fatal + // stream error — treating it as fatal deterministically killed every + // reconnect to a shellper whose replay buffer outgrew the cap. + const oversizedSize = MAX_FRAME_SIZE + 1; const header = Buffer.allocUnsafe(HEADER_SIZE); - header[0] = FrameType.DATA; - header.writeUInt32BE(MAX_FRAME_SIZE + 1, 1); + header[0] = FrameType.REPLAY; + header.writeUInt32BE(oversizedSize, 1); + const oversizedPayload = Buffer.alloc(oversizedSize, 0x42); + const followUp = encodeFrame(FrameType.DATA, Buffer.from('after')); const parser = createFrameParser(); + const frames: ParsedFrame[] = []; + const skipped: Array<{ type: number; size: number }> = []; + const errors: Error[] = []; + parser.on('data', (f: ParsedFrame) => frames.push(f)); + parser.on('frame-skipped', (info: { type: number; size: number }) => skipped.push(info)); + parser.on('error', (err: Error) => errors.push(err)); - await expect(new Promise((resolve, reject) => { - parser.on('data', resolve); - parser.on('error', reject); + await new Promise((resolve) => { parser.write(header); - })).rejects.toThrow(/exceeds maximum/); - }); + // Deliver the oversized payload in fragments, interleaved with the + // next frame in the final chunk (the realistic socket shape). + const half = Math.floor(oversizedSize / 2); + parser.write(oversizedPayload.subarray(0, half)); + parser.write(Buffer.concat([oversizedPayload.subarray(half), followUp])); + parser.end(); + setTimeout(resolve, 10); + }); + + expect(errors).toEqual([]); + expect(skipped).toEqual([{ type: FrameType.REPLAY, size: oversizedSize }]); + expect(frames).toHaveLength(1); + expect(frames[0].payload.toString()).toBe('after'); + }, 20_000); it('passes through unknown frame types', async () => { const unknownType = 0xff; diff --git a/packages/codev/src/terminal/__tests__/tower-shellper-integration.test.ts b/packages/codev/src/terminal/__tests__/tower-shellper-integration.test.ts index b1ec6f38e..da4f36364 100644 --- a/packages/codev/src/terminal/__tests__/tower-shellper-integration.test.ts +++ b/packages/codev/src/terminal/__tests__/tower-shellper-integration.test.ts @@ -10,7 +10,7 @@ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; import os from 'node:os'; import path from 'node:path'; import fs from 'node:fs'; -import { PtySession } from '../pty-session.js'; +import { PtySession, SHELLPER_CLOSE_GRACE_MS } from '../pty-session.js'; import type { PtySessionConfig } from '../pty-session.js'; import { EventEmitter } from 'node:events'; import type { IShellperClient } from '../shellper-client.js'; @@ -35,12 +35,16 @@ class MockShellperClient extends EventEmitter implements IShellperClient { disconnect(): void { this._connected = false; } - write(data: string | Buffer): void { + write(data: string | Buffer): boolean { + if (!this._connected) return false; this.writeData.push(typeof data === 'string' ? data : data.toString('utf-8')); + return true; } - resize(cols: number, rows: number): void { + resize(cols: number, rows: number): boolean { + if (!this._connected) return false; this.resizeCalls.push({ cols, rows }); + return true; } signal(sig: number): void { @@ -213,15 +217,48 @@ describe('PtySession + ShellperClient integration', () => { expect(session.info.exitCode).toBe(0); }); - it('emits exit with code -1 on unexpected disconnect', () => { + it('emits exit with code -1 when an unexpected disconnect outlives the grace window (#1198)', () => { + vi.useFakeTimers(); session.attachShellper(mockClient, Buffer.alloc(0), 9999); const exitSpy = vi.fn(); session.on('exit', exitSpy); mockClient.simulateClose(); + // Not torn down immediately: SessionManager gets a grace window to + // reconnect in place first. + expect(exitSpy).not.toHaveBeenCalled(); + expect(session.status).toBe('running'); + + vi.advanceTimersByTime(SHELLPER_CLOSE_GRACE_MS + 1); + expect(exitSpy).toHaveBeenCalledWith(-1); expect(session.status).toBe('exited'); + vi.useRealTimers(); + }); + + it('a re-attach during the grace window cancels the teardown (#1198)', () => { + vi.useFakeTimers(); + session.attachShellper(mockClient, Buffer.alloc(0), 9999); + const exitSpy = vi.fn(); + session.on('exit', exitSpy); + + mockClient.simulateClose(); + expect(exitSpy).not.toHaveBeenCalled(); + + // SessionManager reconnected in place and Tower re-attached the + // replacement client before the grace window expired. + const replacement = new MockShellperClient(); + session.attachShellper(replacement, Buffer.alloc(0), 9999); + + vi.advanceTimersByTime(SHELLPER_CLOSE_GRACE_MS * 2); + expect(exitSpy).not.toHaveBeenCalled(); + expect(session.status).toBe('running'); + + // I/O flows through the replacement client. + session.write('after recovery'); + expect(replacement.writeData).toContain('after recovery'); + vi.useRealTimers(); }); it('does not double-emit exit on close after exit', () => { diff --git a/packages/codev/src/terminal/pty-manager.ts b/packages/codev/src/terminal/pty-manager.ts index dcfeeadb3..ff91b9331 100644 --- a/packages/codev/src/terminal/pty-manager.ts +++ b/packages/codev/src/terminal/pty-manager.ts @@ -198,6 +198,21 @@ export class TerminalManager { return this.sessions.get(id); } + /** + * Find the session backed by the given SessionManager session id (#1198). + * Adopted sessions are keyed by terminal id in both maps, so the direct + * lookup hits; freshly created sessions use a separate shellper UUID, which + * PtySession records at attach time. + */ + findByShellperSessionId(shellperSessionId: string): PtySession | undefined { + const byId = this.sessions.get(shellperSessionId); + if (byId) return byId; + for (const session of this.sessions.values()) { + if (session.shellperSessionId === shellperSessionId) return session; + } + return undefined; + } + /** Kill and remove a session. */ killSession(id: string): boolean { const session = this.sessions.get(id); diff --git a/packages/codev/src/terminal/pty-session.ts b/packages/codev/src/terminal/pty-session.ts index f282ee8d9..9884e54d7 100644 --- a/packages/codev/src/terminal/pty-session.ts +++ b/packages/codev/src/terminal/pty-session.ts @@ -38,6 +38,14 @@ export interface PtySessionInfo { persistent?: boolean; } +/** + * #1198: how long an unexpected shellper disconnect may remain unresolved + * before the session is torn down. Sized above SessionManager's full + * in-place-reconnect budget (3 rounds of backoff plus connect time) so a + * successful recovery always lands before the teardown fires. + */ +export const SHELLPER_CLOSE_GRACE_MS = 15_000; + export class PtySession extends EventEmitter { readonly id: string; label: string; @@ -51,6 +59,10 @@ export class PtySession extends EventEmitter { private _restartOnExit = false; private _restartCleanupTimeout: ReturnType | null = null; private _restartCancelFn: (() => void) | null = null; + // #1198: pending teardown after an unexpected shellper disconnect. Started + // by the client 'close' handler, cancelled by attachShellper() when + // SessionManager's in-place reconnect delivers a replacement client. + private _closeGraceTimer: ReturnType | null = null; private shellperPid = -1; private cols: number; private rows: number; @@ -126,6 +138,12 @@ export class PtySession extends EventEmitter { this.shellperClient.removeAllListeners('exit'); this.shellperClient.removeAllListeners('close'); } + // #1198: a replacement client arriving means the connection recovered. + // Cancel any pending unexpected-close teardown. + if (this._closeGraceTimer) { + clearTimeout(this._closeGraceTimer); + this._closeGraceTimer = null; + } this._shellperBacked = true; this.shellperClient = client; this.shellperPid = shellperPid; @@ -136,8 +154,11 @@ export class PtySession extends EventEmitter { // keeps it bumped going forward via onPtyData. this._lastDataAt = client.lastDataAt; - // Ensure log directory exists - if (this.diskLogEnabled) { + // Ensure log directory exists. Guarded on logFd: with #1198 re-attach is + // a routine recovery step, and reopening unconditionally would leak one + // append handle per reconnect. cleanupShellper() closes and nulls the fd, + // so a post-teardown attach still reopens. + if (this.diskLogEnabled && this.logFd === null) { fs.mkdirSync(path.dirname(this.logPath), { recursive: true }); this.logFd = fs.openSync(this.logPath, 'a'); } @@ -196,12 +217,20 @@ export class PtySession extends EventEmitter { // Handle shellper disconnect (socket closed without EXIT) client.on('close', () => { - if (this.exitCode === undefined) { - // Unexpected disconnect — shellper may have crashed + if (this.exitCode !== undefined) return; + if (this.shellperClient !== client) return; + if (this._closeGraceTimer) return; + // #1198: SessionManager attempts an in-place reconnect first, so give + // it a grace window instead of tearing down immediately. A successful + // re-attach cancels the timer; expiry means the connection is truly + // gone and the historical teardown proceeds. + this._closeGraceTimer = setTimeout(() => { + this._closeGraceTimer = null; + if (this.exitCode !== undefined) return; this.exitCode = -1; this.emit('exit', -1); this.cleanupShellper(); - } + }, SHELLPER_CLOSE_GRACE_MS); }); } @@ -247,6 +276,10 @@ export class PtySession extends EventEmitter { clearTimeout(this.disconnectTimer); this.disconnectTimer = null; } + if (this._closeGraceTimer) { + clearTimeout(this._closeGraceTimer); + this._closeGraceTimer = null; + } this.clients.clear(); // Close disk log handle if (this.logFd !== null) { @@ -302,32 +335,56 @@ export class PtySession extends EventEmitter { this.logBytes = 0; } - /** Write user input to the PTY or shellper. */ - write(data: string): void { + /** + * Whether input can actually reach the underlying process right now. + * For shellper-backed sessions this checks the live socket connection, not + * just the session status: a session whose shellper connection died reports + * status 'running' until teardown, and writes to it are dropped (#1198). + */ + get writable(): boolean { + if (this.status !== 'running') return false; + if (this._shellperBacked) { + return this.shellperClient !== null && this.shellperClient.connected; + } + return this.pty !== null; + } + + /** + * Write user input to the PTY or shellper. + * Returns false when the input was dropped (#1198). + */ + write(data: string): boolean { if (this._shellperBacked) { if (this.shellperClient && this.status === 'running') { - this.shellperClient.write(data); + return this.shellperClient.write(data); } - return; + return false; } if (this.pty && this.status === 'running') { this.pty.write(data); + return true; } + return false; } - /** Resize the PTY or shellper. */ - resize(cols: number, rows: number): void { + /** + * Resize the PTY or shellper. + * Returns false when the resize was dropped (#1198). + */ + resize(cols: number, rows: number): boolean { this.cols = cols; this.rows = rows; if (this._shellperBacked) { if (this.shellperClient && this.status === 'running') { - this.shellperClient.resize(cols, rows); + return this.shellperClient.resize(cols, rows); } - return; + return false; } if (this.pty && this.status === 'running') { this.pty.resize(cols, rows); + return true; } + return false; } /** Kill the PTY process or send signal to shellper. */ diff --git a/packages/codev/src/terminal/session-manager.ts b/packages/codev/src/terminal/session-manager.ts index 0a2dc01a3..08c4c8a17 100644 --- a/packages/codev/src/terminal/session-manager.ts +++ b/packages/codev/src/terminal/session-manager.ts @@ -73,6 +73,17 @@ export const CRASH_LOOP_WINDOW_MS = 30_000; /** Failing exits within the window needed to declare a crash loop. */ export const CRASH_LOOP_THRESHOLD = 3; +/** + * #1198 in-place reconnect tuning. When a session's socket closes without an + * EXIT frame but the shellper process is still alive, the connection is + * re-established instead of tearing the session down: backoff per attempt + * within one recovery round, a stability window that resets the round + * counter, and a cap on consecutive unstable rounds. + */ +export const RECONNECT_BACKOFF_MS = [500, 1000, 2000]; +export const RECONNECT_STABILITY_MS = 30_000; +export const MAX_RECOVERY_ROUNDS = 3; + /** * Issue #1149: true when the recorded failing-exit timestamps amount to a * crash loop (>= CRASH_LOOP_THRESHOLD failures inside the trailing @@ -159,6 +170,14 @@ interface ManagedSession { stderrBuffer: StderrBuffer | null; stderrStream: Readable | null; stderrTailLogged: boolean; + /** Consecutive close-triggered recovery rounds without a stable connection (#1198). */ + recoveryRounds: number; + /** Epoch of the last successful in-place reconnect, for the stability reset (#1198). */ + lastRecoveryAt: number; +} + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); } export class SessionManager extends EventEmitter { @@ -260,51 +279,163 @@ export class SessionManager extends EventEmitter { stderrBuffer: null, // stderr goes to file, not pipe (Bugfix #324) stderrStream: null, stderrTailLogged: false, + recoveryRounds: 0, + lastRecoveryAt: 0, }; this.sessions.set(opts.sessionId, session); this.log(`Session ${opts.sessionId} created: pid=${info.pid}`); - // Set up auto-restart if configured + this.wireClientEvents(opts.sessionId, session); + + // Start restart reset timer if configured if (opts.restartOnExit) { - this.setupAutoRestart(session, opts.sessionId); + this.startRestartResetTimer(session); } + return client; + } + + /** + * Subscribe to a session client's lifecycle events (exit, error, close) and + * set up auto-restart when configured. Called on session creation, on + * reconnection after a Tower restart, and again for the replacement client + * after an in-place reconnect (#1198), so all three paths behave identically. + */ + private wireClientEvents(sessionId: string, session: ManagedSession): void { + const client = session.client; + // Forward exit events and clean up dead sessions - client.on('exit', (exitInfo) => { - this.log(`Session ${opts.sessionId} exited (code=${exitInfo.code})`); - this.logStderrTail(opts.sessionId, session, exitInfo.code); - this.emit('session-exit', opts.sessionId, exitInfo); + client.on('exit', (exitInfo: ExitMessage) => { + this.log(`Session ${sessionId} exited (code=${exitInfo.code})`); + this.logStderrTail(sessionId, session, exitInfo.code ?? -1); + this.emit('session-exit', sessionId, exitInfo); // If not auto-restarting, remove the dead session from the map // so listSessions() doesn't report it and cleanupStaleSockets() // can clean its socket file. - if (!opts.restartOnExit) { - this.removeDeadSession(opts.sessionId); + if (!session.options.restartOnExit) { + this.removeDeadSession(sessionId); } }); - client.on('error', (err) => { - this.emit('session-error', opts.sessionId, err); + client.on('error', (err: Error) => { + this.emit('session-error', sessionId, err); }); - // Handle shellper crash (socket disconnects without EXIT frame) + client.on('frame-skipped', (info: { type: number; size: number }) => { + this.log(`Session ${sessionId} skipped oversized frame (type=${info.type}, ${info.size} bytes)`); + }); + + // Socket closed without an EXIT frame. Historically treated as "shellper + // crashed", but #1198 showed it also fires for a transient socket error + // against a perfectly healthy shellper, so try to reconnect in place + // before declaring the session dead. client.on('close', () => { - // If the session is still in the map (wasn't already cleaned up by exit/kill), - // the shellper died without sending EXIT. Remove the dead session. - if (this.sessions.has(opts.sessionId)) { - this.log(`Session ${opts.sessionId} shellper disconnected unexpectedly`); - this.logStderrTail(opts.sessionId, session, -1); - this.removeDeadSession(opts.sessionId); - this.emit('session-error', opts.sessionId, new Error('Shellper disconnected unexpectedly')); - } + if (!this.sessions.has(sessionId)) return; + // A replaced client's stale wiring must not trigger recovery again. + if (session.client !== client) return; + this.log(`Session ${sessionId} shellper connection lost unexpectedly`); + this.recoverSession(sessionId, session).catch((err) => { + this.log(`Session ${sessionId} recovery failed unexpectedly: ${(err as Error).message}`); + this.declareSessionDead(sessionId, session); + }); }); - // Start restart reset timer if configured - if (opts.restartOnExit) { - this.startRestartResetTimer(session); + if (session.options.restartOnExit) { + this.setupAutoRestart(session, sessionId); } + } - return client; + /** + * #1198: attempt to re-establish a session's shellper connection after an + * unexpected socket close. The shellper protocol is built for reconnection + * (that is how sessions survive Tower restarts), so a transient socket + * error must not tear down a healthy shellper: doing so unlinks the live + * socket and orphans the process. Falls through to the historical + * dead-session path when the process is gone or every attempt fails. + */ + private async recoverSession(sessionId: string, session: ManagedSession): Promise { + if (Date.now() - session.lastRecoveryAt > RECONNECT_STABILITY_MS) { + session.recoveryRounds = 0; + } + session.recoveryRounds++; + + if (session.recoveryRounds > MAX_RECOVERY_ROUNDS) { + this.log(`Session ${sessionId} gave up reconnecting after ${MAX_RECOVERY_ROUNDS} unstable recovery rounds`); + this.declareSessionDead(sessionId, session); + return; + } + + for (let attempt = 0; attempt < RECONNECT_BACKOFF_MS.length; attempt++) { + await sleep(RECONNECT_BACKOFF_MS[attempt]); + // The session may have been killed or shut down while we waited. + if (this.sessions.get(sessionId) !== session) return; + if (!(await this.canReachShellper(session))) { + break; + } + + const client = new ShellperClient(session.socketPath); + try { + await client.connect(); + } catch (err) { + this.log(`Session ${sessionId} reconnect attempt ${attempt + 1}/${RECONNECT_BACKOFF_MS.length} failed: ${(err as Error).message}`); + continue; + } + + if (this.sessions.get(sessionId) !== session) { + // Killed while we were connecting + client.disconnect(); + return; + } + + session.client = client; + session.lastRecoveryAt = Date.now(); + this.wireClientEvents(sessionId, session); + this.log(`Session ${sessionId} shellper connection re-established (round ${session.recoveryRounds}, attempt ${attempt + 1})`); + this.emit('session-reconnected', sessionId, client); + return; + } + + this.declareSessionDead(sessionId, session); + } + + /** + * Whether the session's recorded shellper process is still the one running + * under its PID: alive, and with a matching start time (guards PID reuse). + */ + private async isShellperProcessCurrent(session: ManagedSession): Promise { + if (!this.isProcessAlive(session.pid)) return false; + const actualStartTime = await getProcessStartTime(session.pid); + if (actualStartTime === null || Math.abs(actualStartTime - session.startTime) > 2000) { + return false; + } + return true; + } + + /** + * Whether an in-place reconnect can possibly succeed: the shellper process + * is current (alive, same start time) and its socket file still exists. + */ + private async canReachShellper(session: ManagedSession): Promise { + if (!(await this.isShellperProcessCurrent(session))) return false; + try { + const stat = fs.lstatSync(session.socketPath); + return stat.isSocket(); + } catch { + return false; + } + } + + /** + * The historical unexpected-death path: remove the session, clean up its + * socket, and emit 'session-error' so the Tower layer can log the loss. + */ + private declareSessionDead(sessionId: string, session: ManagedSession): void { + if (this.sessions.get(sessionId) !== session) return; + this.log(`Session ${sessionId} shellper disconnected unexpectedly`); + this.logStderrTail(sessionId, session, -1); + this.removeUnreachableSession(sessionId); + this.emit('session-error', sessionId, new Error('Shellper disconnected unexpectedly')); } /** @@ -382,35 +513,14 @@ export class SessionManager extends EventEmitter { stderrBuffer: null, stderrStream: null, stderrTailLogged: false, + recoveryRounds: 0, + lastRecoveryAt: 0, }; this.sessions.set(sessionId, session); this.log(`Session ${sessionId} reconnected: pid=${pid}`); - // Set up auto-restart if configured (e.g. architect sessions after Tower restart) - if (hasRestart) { - this.setupAutoRestart(session, sessionId); - } - - client.on('exit', (exitInfo) => { - this.emit('session-exit', sessionId, exitInfo); - if (!hasRestart) { - this.removeDeadSession(sessionId); - } - }); - - client.on('error', (err) => { - this.emit('session-error', sessionId, err); - }); - - // Handle shellper crash (socket disconnects without EXIT frame) - client.on('close', () => { - if (this.sessions.has(sessionId)) { - this.log(`Session ${sessionId} shellper disconnected unexpectedly`); - this.removeDeadSession(sessionId); - this.emit('session-error', sessionId, new Error('Shellper disconnected unexpectedly')); - } - }); + this.wireClientEvents(sessionId, session); // Start restart reset timer if auto-restart is enabled if (hasRestart) { @@ -701,14 +811,68 @@ export class SessionManager extends EventEmitter { * Remove a dead session from the map, clear its timers, and clean up socket. * Called when a session exits naturally (no restart) or exhausts maxRestarts. */ + /** + * Removal for sessions whose CHILD process ended (clean exit, or restart + * exhaustion). The shellper deliberately lingers after its child exits to + * serve late connections (#905), so it is a spent husk here: unlink the + * socket unconditionally — that is what marks the lingering shellper + * unresponsive so killOrphanedShellpers reaps it. Preserving the socket on + * this path leaks an immortal empty shellper per exited session (#1198 + * e2e regression). + */ private removeDeadSession(sessionId: string): void { - const session = this.sessions.get(sessionId); + const session = this.detachSessionFromMap(sessionId); + if (!session) return; + this.unlinkSocketIfExists(session.socketPath); + } + + /** + * #1198: removal for sessions whose CONNECTION died (recovery exhausted or + * shellper unreachable). Unlike the child-exit path, the shellper may + * still host a live conversation, and unlinking a live shellper's socket + * makes it unreachable AND flags it for the orphan sweeper's kill path — + * so this path preserves the socket while the recorded process is current, + * degrading to a recoverable orphan (re-adoptable, attachable), never a + * killed live process. + */ + private removeUnreachableSession(sessionId: string): void { + const session = this.detachSessionFromMap(sessionId); if (!session) return; + this.cleanupDeadSessionSocket(sessionId, session).catch((err) => { + this.log(`Session ${sessionId} socket cleanup failed: ${(err as Error).message}`); + }); + } + + private detachSessionFromMap(sessionId: string): ManagedSession | null { + const session = this.sessions.get(sessionId); + if (!session) return null; if (session.restartResetTimer) { clearTimeout(session.restartResetTimer); session.restartResetTimer = null; } this.sessions.delete(sessionId); + return session; + } + + /** + * Socket cleanup for the connection-failure path: unlink only when the + * recorded shellper process is gone. The check uses the same start-time + * guard as reconnection, so a reused PID counts as "gone" and a dead + * shellper's socket/log files do not leak. Async and best-effort by + * design; the map mutation stays sync in the caller. + */ + private async cleanupDeadSessionSocket(sessionId: string, session: ManagedSession): Promise { + if (await this.isShellperProcessCurrent(session)) { + this.log(`Session ${sessionId} removed but shellper pid=${session.pid} is alive; preserving socket`); + return; + } + // Socket paths are keyed by session id, so if a new session claimed this + // id while we awaited (an in-process re-create/re-adopt), the path now + // belongs to the new occupant — leave it alone. The caller already + // deleted OUR map entry, so any present entry is a successor's. + if (this.sessions.has(sessionId)) { + return; + } this.unlinkSocketIfExists(session.socketPath); } diff --git a/packages/codev/src/terminal/shellper-client.ts b/packages/codev/src/terminal/shellper-client.ts index 76678092e..f8af01dbe 100644 --- a/packages/codev/src/terminal/shellper-client.ts +++ b/packages/codev/src/terminal/shellper-client.ts @@ -37,8 +37,10 @@ import { export interface IShellperClient extends EventEmitter { connect(): Promise; disconnect(): void; - write(data: string | Buffer): void; - resize(cols: number, rows: number): void; + /** Returns false when the frame was dropped because the client is not connected (#1198). */ + write(data: string | Buffer): boolean; + /** Returns false when the frame was dropped because the client is not connected (#1198). */ + resize(cols: number, rows: number): boolean; signal(sig: number): void; spawn(msg: SpawnMessage): void; ping(): void; @@ -57,6 +59,13 @@ export interface IShellperClient extends EventEmitter { export class ShellperClient extends EventEmitter implements IShellperClient { private socket: net.Socket | null = null; private _connected = false; + // #1198: a 'close' emission is owed to consumers. Recorded by cleanup() + // when it tears down a live connection that did not ask to die (anything + // other than disconnect()); consumed by the socket's 'close' event. The + // decision must be captured at teardown time because error paths run + // cleanup() before that event fires — reading _connected inside the close + // handler is what swallowed the emission. + private _closePending = false; private replayData: Buffer | null = null; // Wall-clock epoch (ms) of the last PTY byte the shellper has seen. // @@ -121,6 +130,7 @@ export class ShellperClient extends EventEmitter implements IShellperClient { reject(new Error('Already connected')); return; } + this._closePending = false; const socket = net.createConnection(this.socketPath); this.socket = socket; @@ -143,6 +153,18 @@ export class ShellperClient extends EventEmitter implements IShellperClient { this.safeEmitError(err); this.cleanup(); }); + // #1198: the parser drops oversized frames instead of erroring (a + // long-lived shellper's replay can exceed the frame cap). The + // connection stays healthy; surface what was lost. For a dropped + // REPLAY, unblock replay waiters with an empty replay so adoption + // proceeds (viewers repaint via the post-connect resize nudge). + parser.on('frame-skipped', (info: { type: number; size: number }) => { + if (info.type === FrameType.REPLAY && this.replayData === null) { + this.replayData = Buffer.alloc(0); + this.emit('replay', this.replayData); + } + this.emit('frame-skipped', info); + }); socket.on('connect', () => { socket.pipe(parser); @@ -151,9 +173,13 @@ export class ShellperClient extends EventEmitter implements IShellperClient { }); socket.on('close', () => { - const wasConnected = this._connected; this.cleanup(); - if (wasConnected) { + // Emit exactly once per unexpectedly lost live connection. + // _closePending was recorded by whichever cleanup() ran first — an + // error path's or the one just above — so error-path closes are no + // longer swallowed (#1198). + if (this._closePending) { + this._closePending = false; this.emit('close'); } if (!handshakeResolved) { @@ -260,10 +286,13 @@ export class ShellperClient extends EventEmitter implements IShellperClient { } disconnect(): void { - this.cleanup(); + this.cleanup(true); } - private cleanup(): void { + private cleanup(intentional = false): void { + if (this._connected && !intentional) { + this._closePending = true; + } this._connected = false; if (this.socket && !this.socket.destroyed) { this.socket.destroy(); @@ -271,14 +300,16 @@ export class ShellperClient extends EventEmitter implements IShellperClient { this.socket = null; } - write(data: string | Buffer): void { - if (!this._connected || !this.socket) return; + write(data: string | Buffer): boolean { + if (!this._connected || !this.socket) return false; this.socket.write(encodeData(data)); + return true; } - resize(cols: number, rows: number): void { - if (!this._connected || !this.socket) return; + resize(cols: number, rows: number): boolean { + if (!this._connected || !this.socket) return false; this.socket.write(encodeResize({ cols, rows })); + return true; } signal(sig: number): void { diff --git a/packages/codev/src/terminal/shellper-process.ts b/packages/codev/src/terminal/shellper-process.ts index 089c39696..5b4535094 100644 --- a/packages/codev/src/terminal/shellper-process.ts +++ b/packages/codev/src/terminal/shellper-process.ts @@ -21,6 +21,7 @@ import { FrameType, PROTOCOL_VERSION, ALLOWED_SIGNALS, + REPLAY_PAYLOAD_MAX, createFrameParser, encodeData, encodeWelcome, @@ -377,11 +378,23 @@ export class ShellperProcess extends EventEmitter { socket.write(welcome); this.log(`WELCOME sent: pid=${pid}, version=${PROTOCOL_VERSION}`); - // Send replay buffer - const replayData = this.replayBuffer.getReplayData(); - if (replayData.length > 0) { - socket.write(encodeReplay(replayData)); + // Send replay buffer. #1198: never emit a frame the peer's parser must + // drop — a long-lived TUI session's replay (newline-free, unbounded + // partial) can exceed MAX_FRAME_SIZE, which is exactly what zombified + // long-lived terminals on every reconnect. Send the most recent bytes + // that fit; a tail-trimmed replay can render imperfectly for alt-screen + // TUIs (#1047), but the client's post-connect resize nudge repaints, + // and a truncated replay beats a dead connection. + let replayData = this.replayBuffer.getReplayData(); + if (replayData.length > REPLAY_PAYLOAD_MAX) { + this.log(`Replay ${replayData.length} bytes exceeds cap; sending last ${REPLAY_PAYLOAD_MAX}`); + replayData = replayData.subarray(replayData.length - REPLAY_PAYLOAD_MAX); } + // #1198: send REPLAY even when empty, so a client awaiting the frame + // resolves immediately instead of burning its timeout. Creation-time + // attach awaits the frame to avoid racing early child output into a + // dropped replay; old clients treat an empty REPLAY as no replay data. + socket.write(encodeReplay(replayData)); // If the PTY already exited before this client connected, the original // EXIT broadcast missed it. Replay the retained EXIT frame so the client diff --git a/packages/codev/src/terminal/shellper-protocol.ts b/packages/codev/src/terminal/shellper-protocol.ts index 770c7ae6d..8773f3692 100644 --- a/packages/codev/src/terminal/shellper-protocol.ts +++ b/packages/codev/src/terminal/shellper-protocol.ts @@ -33,6 +33,11 @@ export type FrameTypeValue = (typeof FrameType)[keyof typeof FrameType]; export const PROTOCOL_VERSION = 1; export const MAX_FRAME_SIZE = 16 * 1024 * 1024; // 16MB export const HEADER_SIZE = 5; // 1 byte type + 4 bytes length +// #1198: cap on the replay payload a shellper sends on (re)connection. Kept +// well under MAX_FRAME_SIZE so a REPLAY frame can never be one the peer's +// parser must drop. When a session's replay outgrows this, the most recent +// bytes are sent (see ShellperProcess.handleHello). +export const REPLAY_PAYLOAD_MAX = 8 * 1024 * 1024; // 8MB // --- Allowed Signals --- @@ -171,12 +176,18 @@ export interface ParsedFrame { * Input: raw socket data (may be fragmented across chunks) * Output: ParsedFrame objects (in objectMode) * - * Emits 'frame-error' event on protocol violations (oversized frames, - * malformed headers). The stream is destroyed on error. + * An oversized frame (declared payload > MAX_FRAME_SIZE) is NOT a stream + * error: it is discarded incrementally and surfaced via a 'frame-skipped' + * event ({ type, size }), and parsing continues with the next frame. #1198: + * a long-lived shellper's replay buffer can legitimately outgrow the cap; + * treating that as fatal killed the connection deterministically on every + * reconnect and cascaded into orphaned/killed sessions. */ export class FrameParser extends Transform { private chunks: Buffer[] = []; private bufferedLength = 0; + // Bytes of an oversized frame's payload still to be dropped (#1198). + private discardRemaining = 0; constructor() { super({ readableObjectMode: true }); @@ -205,21 +216,41 @@ export class FrameParser extends Transform { } private drainFrames(): void { - while (this.bufferedLength >= HEADER_SIZE) { + while (true) { + // Finish dropping an oversized frame's payload before parsing resumes. + if (this.discardRemaining > 0) { + const drop = Math.min(this.discardRemaining, this.bufferedLength); + if (drop > 0) { + this.discard(drop); + this.discardRemaining -= drop; + } + if (this.discardRemaining > 0) { + return; // wait for more data to drop + } + } + + if (this.bufferedLength < HEADER_SIZE) { + return; + } + // Peek at the header without consuming const header = this.peek(HEADER_SIZE); const payloadLength = header.readUInt32BE(1); if (payloadLength > MAX_FRAME_SIZE) { - throw new Error( - `Frame payload size ${payloadLength} exceeds maximum ${MAX_FRAME_SIZE}`, - ); + // #1198: drop the frame, keep the stream. The consumer learns what + // was lost via 'frame-skipped' and decides how to degrade (for a + // REPLAY, the terminal repaints via the post-connect resize nudge). + this.emit('frame-skipped', { type: header[0], size: payloadLength }); + this.discard(HEADER_SIZE); + this.discardRemaining = payloadLength; + continue; } const totalLength = HEADER_SIZE + payloadLength; if (this.bufferedLength < totalLength) { // Need more data - break; + return; } // Consume the full frame @@ -231,6 +262,22 @@ export class FrameParser extends Transform { } } + /** Drop n buffered bytes without materializing them (n <= bufferedLength). */ + private discard(n: number): void { + let remaining = n; + while (remaining > 0 && this.chunks.length > 0) { + const head = this.chunks[0]; + if (head.length <= remaining) { + remaining -= head.length; + this.chunks.shift(); + } else { + this.chunks[0] = head.subarray(remaining); + remaining = 0; + } + } + this.bufferedLength -= n; + } + private peek(n: number): Buffer { if (this.chunks.length === 1 && this.chunks[0].length >= n) { return this.chunks[0].subarray(0, n);