[rush-daemon] Use directory-level async watchers on Linux

[Sean Larkin (TheLarkInn)]

Summary

With RUSH_DAEMON_WATCH=1 on Linux, rushd used Node's recursive fs.watch. Node emulates that on Linux with one inotify watch per file, including node_modules entries and build outputs, and walks the tree synchronously on the event loop. This PR replaces it on Linux with non-recursive per-directory watches created by an async walk. macOS and Windows are unchanged.

Root cause

WorkspaceSessionFileWatcher defaulted to fs.watch(folder, { recursive: true }). On Linux (internal/fs/recursive_watch), that:

• calls statSync/readdirSync for the whole project tree inside fs.watch(). rushd makes that call from watchProjects() in the warm-set configureIteration hook, so every daemon client (status, queue feedback, other builds' output) freezes while newly requested projects are walked (Removed unused @types/chalk references #232: daemon status spikes to 5.7 s during a lib/ cache restore);
• adds an inotify watch for every file and every rename in the tree, including thousands of output files, so it can hit fs.inotify.max_user_watches;
• emits ENOENT when an entry disappears between readdir and stat (Make quiet the default, add --verbose #111). That error marks the watcher unhealthy for the rest of the session.

Fix

• New internal LinuxTreeWatcher (src/LinuxTreeWatcher.ts). It is fs.FSWatcher-compatible (emits error/close, supports ref/unref):
  • Creates one non-recursive fs.watch per directory (an inotify watch on a directory already reports its files) using an async opendir walk. The root is registered synchronously, so a missing root still throws like fs.watch.
  • Prunes node_modules and .git at any depth. For project watchers, it also prunes the project's .rush/temp and every declared operationSettings[].outputFolderNames from rush-project.json, loaded through the public RushProjectConfiguration API. It only prunes folders strictly inside the project, and a load failure just means nothing extra is pruned. Events for pruned paths are suppressed.
  • Follows directories created later: the parent's rename event triggers lstat, then a watch and walk of the new directory, then a coverage event for it. Each watch is created before its directory is listed, so files written during the walk are not lost. Removed directories drop their watch and every watch below it.
  • Treats ENOENT/ENOTDIR/EACCES/EPERM while registering or observing a child directory as the directory having gone away. The parent's rename event already reports the change, so the watcher is not marked unhealthy.
  • Reports ENOSPC once, as an explicit error naming fs.inotify.max_user_watches and the remedy, and stops registering. The existing onError path then marks the watcher unhealthy, so the daemon falls back to full invalidation.
• WorkspaceSessionFileWatcher: when no watchFactory is injected, Linux recursive paths use LinuxTreeWatcher. Project watchers report their root once the async walk finishes, to cover the registration window. startAsync waits for the (small) permanent common/config walks before initialization is acknowledged. Injected watchFactory behavior and the public API are unchanged, so no API report change is needed.

Tests

• LinuxTreeWatcher.test.ts (9 tests, uses a fake directory-watch function over a real temp tree, so it is deterministic on every OS):
  • watches directories only, with the pruning set
  • suppresses events for pruned paths
  • follows directories created or removed later
  • tolerates a transient ENOENT without an error
  • reports ENOSPC once, with an explicit message
  • throws on a missing root
  • reports coverage after the walk and emits close
  • adds a Linux-only real-inotify test
• WorkspaceSessionFileWatcher.test.ts: covers the project exclusion set.
• Linux (WSL Ubuntu-24.04, Node 22.23.2): rush build --to @rushstack/rush-daemon, including lint, succeeds with no warnings. heft test --test-path-pattern "LinuxTreeWatcher|WorkspaceSessionFileWatcher|WorkspaceWatchPolicy|WorkspaceWarmSet" passes 41/41. WorkspaceWatchPolicy and WorkspaceWarmSet exercise the new default watcher against real files.

Linux validation: watch counts

Standalone measurement in one process that watches all 196 rushstack projects of a lab clone, which was built only up to @rushstack/rush-daemon, so outputs exist only for part of the graph. The script loads the built lib-commonjs from this PR and uses real rush-project.json exclusions: node linux-watcher-count.js <clone> <proto|native>. It counts inotify lines in /proc/self/fdinfo.

mode	projects	inotify watches	max event-loop block	RSS
Node recursive fs.watch (before)	196	21,955	798 ms (synchronous)	166 MB
LinuxTreeWatcher (after)	196	2,101 (10.4x fewer)	47 ms	146 MB

This matches the A07 prototype measurement on a fully built rushstack clone under heavy host load: 182 projects, 32,822 → 2,921 watches, with the event-loop block going from 100.9 s → 22 ms.

Linux validation: in-situ daemon repro

linux-watcher-repro.sh runs both clients in one lab invocation on a fresh mkws-synth --projects 12 workspace with RUSH_DAEMON_WATCH=1. "Before" is the unfixed rush-client; "after" is apps/rush-cli-client/bin/rush-client from this branch. The steps are:

1. Run a full build.
2. Write 400 files into p03/lib, an output folder.
3. Create p05/src/newdir/deep/added.ts, a new nested source directory.
4. Rebuild.

client	daemon inotify watches after build	after 400 output writes + new src dir	build 2 re-executed
before	271	674	p05–p12
after	62	64	p05–p12 (identical)

The change inside the newly created nested directory is detected and triggers exactly the same rebuild set. Output writes no longer add watches.

The full rush test --only @rushstack/rush-daemon suite also passes on Linux.

Follow-ups (not in this PR)

• Surface the ENOSPC state in daemon status (for example watcherUnhealthy: 'inotify-limit'), not only as the error message.
• Optionally bound concurrency in the async walk, and re-read outputFolderNames when rush-project.json changes without a generation reload.

Fixes #6078

---

This came out of the automated rushd Linux performance and behavior analysis ("Rushd Hive", bug #200; related #111, #211, #232).

[Copilot]

Copilot review overview

🟡 Changes recommended

Directory replacement races, incomplete error handling, and exclusion-loading gaps can leave Linux watches incomplete or incorrectly active.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 5

Open (5)

• Preserve null filename for unknown invalidation · New
• Prevent watches for paths excluded after async load · New
• Reconcile watches after directory replacement · New
• Handle non-transient opendir failures · New
• Preserve base exclusion when configuration loading fails · New

What changed in this PR

Replaces Linux recursive fs.watch usage with asynchronous directory-level watchers to reduce event-loop blocking and inotify consumption.

Changes:

• Adds Linux-specific tree watching with pruning and dynamic directory tracking.
• Integrates project output exclusions and expands watcher tests.
• Adds a patch change entry.

File	Description
libraries/​rush-daemon/​src/​LinuxTreeWatcher.ts	Implements the Linux directory watcher.
libraries/​rush-daemon/​src/​WorkspaceSessionFileWatcher.ts	Selects the Linux watcher and loads exclusions.
libraries/​rush-daemon/​src/​test/​LinuxTreeWatcher.test.ts	Tests traversal, pruning, errors, and events.
libraries/​rush-daemon/​src/​test/​WorkspaceSessionFileWatcher.test.ts	Tests project exclusion discovery.
common/​changes/​@rushstack/​rush-daemon/​linux-tree-watcher_2026-09-24.json	Records the patch change.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.