[rush-daemon-transport] Reap a dead daemon's orphaned operation processes on reclaim

[Sean Larkin (TheLarkInn)]

Summary

When a rushd daemon dies uncleanly (SIGKILL / OOM killer), the phased operation child processes it spawned keep running. They share the daemon's session and process group (sid = pgid = daemon pid, because the daemon is spawned detached and ShellOperationRunner children are not). The auto-started successor then reclaims the socket and re-runs the same operation concurrently on the same outputs as the orphan. This PR makes reclaim terminate those orphans before the successor does any work.

Root cause

reclaimStaleDaemonAsync proved the owner dead (dead lockfile PID plus a failed connect probe) and removed the lockfile and socket. It did nothing about the processes the dead daemon left behind in its process group.

Fix

• New DaemonOrphanReaper.reapDeadDaemonProcessGroupAsync(deadPid, options?), called from reclaimUnderLockAsync. It runs under the reclaim mutex, after the two-factor dead-owner proof, and before the artifacts are removed and the successor binds. It:
  • sends SIGTERM to process group deadPid and polls for up to a bounded grace period (2 s);
  • then sends SIGKILL and polls again for up to the same grace period;
  • aborts the reclaim by throwing if the group still exists after that, so no successor binds while orphans may still be writing;
  • logs what was reclaimed via process.emitWarning(..., { code: 'RUSH_DAEMON_ORPHANS_REAPED' }), e.g. Reclaimed dead daemon 1234: its orphaned operation process group was terminated|killed.
• PID-reuse guard (documented in the TSDoc): only group id deadPid is ever signaled. That is the proven-dead daemon's pid recorded in the lockfile, and it is signaled only when deadPid is not alive and group deadPid still exists. POSIX never assigns a pid that is still in use as a process group id, so no unrelated process can have taken deadPid while that group exists, and every remaining member belongs to the dead daemon.
• Fail-closed guards: never signals pids < 2 (kill(-0)/kill(-1)), non-integers, the caller's own pid, or the caller's own process group (read from /proc/self/stat). Does nothing when the caller's group cannot be determined (no /proc) or on Windows. Only ESRCH counts as "group gone"; EPERM and other probe or signal errors propagate and fail the reclaim.
• Probing, signaling, the clock and logging sit behind an injectable IDaemonProcessGroupOps (DaemonProcessGroup.ts). There is no public API change: the new modules are not exported from the package index.

This is only the orphan-reaping part of the A05 prototype.

Tests

• DaemonOrphanReaper.test.ts (fake ops) covers:
  • the SIGTERM-only path;
  • SIGKILL escalation;
  • abort when the group survives SIGKILL;
  • no signal for a live daemon or when the group is gone;
  • no signal for unsafe ids (1, own pid, NaN), for the caller's own or unknown group, or on Windows;
  • that the log line is emitted.
• DaemonProcessGroup.test.ts: ESRCH means gone, EPERM propagates, and the signal is sent to the negated group id.
• OrphanReapOnReclaim.test.ts (POSIX, real processes): a detached fake daemon with one non-detached child (like a phased operation) is SIGKILLed, and reclaimStaleDaemonAsync reaps the orphaned child and logs it.

Linux validation (WSL Ubuntu-24.04)

• rush build --to @rushstack/rush-daemon-transport and rush test --only @rushstack/rush-daemon-transport pass, including the package's ultra-strict lint contract.
• End-to-end repro: kill -9 the daemon during a 30 s build --only p01, then run the next build. Details are in the PR comment.
  • Before: 2 orphans (sh -c node build.js, node build.js) were still running after the next build.
  • After: 0 orphans remained; they were terminated by SIGTERM and the RUSH_DAEMON_ORPHANS_REAPED warning was logged.
  • I re-ran this after the review changes and got the same result.

Out of scope / follow-ups

• Children spawned into their own detached group are not covered: global commands, rushx and IPCOperationRunner, all via SubprocessTerminator.RECOMMENDED_OPTIONS. Covering them needs @rushstack/rush-daemon to persist those group ids, plus a PID-reuse guard for groups whose leader may have exited (for example, comparing start times).
• macOS: the caller's own group is not determinable without /proc, so reaping is currently disabled there (fail closed).
• Surface the reclaim warning in rush-client output explicitly.

Fixes #6055

This fix came out of the automated rushd Linux analysis ("Rushd Hive").

[Sean Larkin (TheLarkInn)]

Linux e2e validation (WSL Ubuntu-24.04, synthetic 12-project workspace)

Repro: start rush-client build --only p01 (op sleeps 30 s), kill -9 the daemon mid-build, then run the next build (which auto-starts a successor daemon and reclaims).

	orphans in dead daemon's pgid before next build	next build	orphans after next build
before (unfixed rush-client)	2 (sh -c node build.js, node build.js)	exit 0, 3.52 s	2 (still running)
after (this PR)	2	exit 0, 3.38 s	0

After-run log line:

[RUSH_DAEMON_ORPHANS_REAPED] Warning: Reclaimed dead daemon 3553277: its orphaned operation process group was terminated.

(SIGTERM sufficed; no SIGKILL escalation needed.) Script: rushd-lab/fixes/orphans-e2e.sh.

[Copilot]

Copilot review overview

🟡 Changes recommended

Current rushd operations use separate detached process groups, so targeting only the dead daemon’s group does not reap the actual orphaned operations.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 · 2 · 1

Open (5)

• Reaper targets daemon group instead of detached operation groups · New
• Treat only ESRCH as proof that a process group is absent · New
• Fail closed when the caller's process group is unknown · New
• Wait for process-group quiescence after SIGKILL · New
• Test uses incorrect process-group topology · New

What changed in this PR

Adds stale-daemon process cleanup during socket reclamation.

Changes:

• Adds guarded POSIX process-group termination with SIGTERM/SIGKILL escalation.
• Integrates orphan reaping into daemon reclaim.
• Adds unit and process-level tests plus a patch change record.

File	Description
DaemonReclaim.ts	Invokes orphan cleanup before removing artifacts.
DaemonProcessGroup.ts	Implements injectable process-group operations.
DaemonOrphanReaper.ts	Adds guarded termination and escalation logic.
DaemonOrphanReaper.test.ts	Tests reaper outcomes and safety checks.
OrphanReapOnReclaim.test.ts	Adds process-level reclaim coverage.
rushd-reap-orphans_2026-09-24-03-00.json	Records the patch change.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.