Tower upgrade stalls terminal adoption: waitForReplay burns 500ms per idle old-binary shellper (version skew from #1198)

[amrmelsayed]

Symptom

After installing the latest Tower build (PR #1204, merged 2026-07-21), terminal sessions are unresponsive for several seconds when first opened after a Tower restart, sometimes needing a close/reopen. Reported 2026-07-22 — the day after merge.

Root cause (confirmed in source)

A version-skew gap in #1198's replay-await fix:

1. Shellpers are detached and outlive Tower (shellper-main.ts:4, session-manager.ts:230 detached: true) — by design, so sessions survive Tower restarts. Consequence: upgrading Tower does NOT upgrade running shellpers; every pre-upgrade shellper keeps executing the old binary indefinitely.
2. Shellper reconnect error is swallowed: terminal becomes a silent zombie (no input/output, 'Message sent' logged for dropped frames) until next Tower restart #1198 changed the reconnect/adoption path (tower-terminals.ts:770, 983) from synchronous getReplayData() to await client.waitForReplay() — fixing a real race where adopted terminals rendered blank.
3. New shellpers always send a REPLAY frame, even empty (commit 7a2f8053, from codex round-2 consult) — precisely so the await resolves in milliseconds. Old-binary shellpers only send REPLAY when their buffer is non-empty: an idle old shellper sends nothing.
4. So for every idle pre-upgrade shellper, waitForReplay() burns its full 500ms default timeout (shellper-client.ts:341) before resolving empty.
5. The adoption loop processes probe results sequentially (tower-terminals.ts:753: "Process probe results sequentially (shared state mutations)") — so ~20 adopted sessions, mostly idle, ≈ up to ~10s of serialized stall in the post-restart adoption pass, during which terminals appear dead/blank.

The #1198 review notes said "creation always talks to a freshly spawned new-binary shellper" — true for creation, but adoption/reconnect talks to old-binary shellpers by definition. The skew slipped past two consult rounds and human review.

Why it matters beyond this week

Self-limiting for THIS upgrade (old shellpers die off and respawn on the new binary), but the class recurs on every future Tower upgrade that changes a protocol expectation while detached shellpers persist. Needs a structural answer, not a wait-it-out.

Fix directions (plan-phase decision)

1. Protocol-version-aware wait: the WELCOME frame already carries a protocol version — thread it to the client so waitForReplay() knows whether the peer guarantees a REPLAY frame; skip/short-circuit the wait for old-protocol peers. Surgical, no behavior change for new shellpers.
2. Overlap the waits: move waitForReplay() into the parallel probe phase (bounded concurrency 5 already exists) so per-session timeouts overlap instead of serializing; keep only the shared-state mutations sequential. Cheap, halves-to-quarters the stall, composes with option 1.
3. Bounce old shellpers on upgrade: have local-install/upgrade flow (or Tower startup detecting a version mismatch) proactively restart old-binary shellpers. Heavyweight — kills live Claude conversations unless combined with resume; probably wrong as primary, note why if rejected.
4. Consider lowering the 500ms default and/or logging when the timeout is burned (observability so the next skew is visible in afx tower log instead of feeling like mystery lag).

Also reported alongside (NOT explained by this): VS Code "window terminated unexpectedly (reason: 'killed', code: '9')"

Investigated same day: no causal link found to codev. Extension has no terminal webview (native Pseudoterminal), no unbounded buffers, no relevant changes in a month; Tower's kill paths (towerStop SIGKILL escalation, killOrphanedShellpers) cannot reach VS Code's process tree. Electron code-9 is typically OS OOM. Track separately if it recurs; collect VS Code crash logs (Help → Open Logs Folder) at time of incident. Mentioned here only so the planner knows it was ruled out, not in scope.

Origin

Follow-up to #1198 / PR #1204 (branch builder/pir-1198). Root-caused by main architect 2026-07-22 from a user report. PIR: touches the shared shellper protocol/adoption machinery days after #1198's own live-deploy incident in the same code; the version-awareness design decision deserves a plan gate.

[amrmelsayed]

On it! Working on this with the PIR protocol (plan + dev-approval gates before PR).