diff --git a/codev/projects/1238-agent-farm-pty-session-logs-ha/status.yaml b/codev/projects/1238-agent-farm-pty-session-logs-ha/status.yaml new file mode 100644 index 000000000..bb066161c --- /dev/null +++ b/codev/projects/1238-agent-farm-pty-session-logs-ha/status.yaml @@ -0,0 +1,17 @@ +id: '1238' +title: agent-farm-pty-session-logs-ha +protocol: air +phase: pr +plan_phases: [] +current_plan_phase: null +gates: + pr: + status: approved + requested_at: '2026-07-25T09:57:08.830Z' + approved_at: '2026-07-25T12:37:59.070Z' +iteration: 1 +build_complete: false +history: [] +started_at: '2026-07-25T09:40:38.172Z' +updated_at: '2026-07-25T12:37:59.071Z' +pr_ready_for_human: false diff --git a/codev/resources/arch.md b/codev/resources/arch.md index 6cbf4e90c..95f2566b8 100644 --- a/codev/resources/arch.md +++ b/codev/resources/arch.md @@ -860,6 +860,8 @@ async cleanupStaleSockets(): Promise { **Husk sweep (Issue #1227).** The "orphaned shellpers are never killed" policy above is deliberately permissive for the reconnectable case, but it also permanently protects a shellper whose PTY child has already exited (a "husk") — a husk's socket still responds, so `killOrphanedShellpers` skips it forever, even though it is not reconnectable. A separate, stricter sweep (`shellper-husk-sweep.ts`) closes that gap: it reaps a shellper only when it is simultaneously **unregistered** (no live PID in `terminal_sessions`), **childless** (no OS child process), and **aged** past a grace period (`SHELLPER_HUSK_GRACE_MS`, default 1h) — three conditions that together are true only for a genuine husk, never a live or reconnectable session. It runs on three triggers: Tower startup, an hourly in-process timer, and on-demand via `afx tower sweep-husks` (dry-run preview by default, `--apply` to reap, mirroring `afx workspace recover`'s UX). Fleet-wide RSS and an unregistered-shellper count are surfaced in `/health` and `afx status` for observability, computed from a shared `process-census.ts` `ps` scan. +**Session log retention (Issue #1238).** PTY session logs live in `~/.agent-farm/logs` as one `.log` per terminal Tower has ever opened (plus at most one `.log.1` rotation — `PtySession` caps a single log at 50 MB). That per-file cap was the only bound: nothing ever deleted the log of a session that ended, so the directory accreted forever (an audited machine reached 29,728 files / 19 GB, 97% of it untouched for over a month). `session-log-sweep.ts` adds the missing cross-file policy: a log is unlinked once its **mtime** is older than the retention window (`AGENT_FARM_LOG_RETENTION_DAYS`, default 30; `0` disables). mtime is the retention key because the last byte written to a session log is effectively when that session stopped producing output — the dead session's DB row is long gone, so there is nothing to join against. Live sessions are excluded **by id** regardless of mtime, because their log fd stays open for the session's life and unlinking under a live writer would silently redirect every subsequent write into an unlinked inode. Two triggers: Tower startup (ordered *after* `reconcileTerminalSessions` so surviving sessions are in the terminal manager's map and therefore excluded) and a daily unref'd in-process timer (`AGENT_FARM_LOG_SWEEP_INTERVAL_MS`). `codev doctor` reports the footprint as a health line, warning past 2 GB. + #### Dead Process Cleanup Tower cleans up stale entries on state load: diff --git a/codev/state/air-1238_thread.md b/codev/state/air-1238_thread.md new file mode 100644 index 000000000..f567169fe --- /dev/null +++ b/codev/state/air-1238_thread.md @@ -0,0 +1,154 @@ +# air-1238 — PTY session log retention + +Protocol: AIR (strict). Issue #1238: `~/.agent-farm/logs` grew to 29,728 files / 19 GB +with no rotation or retention. + +## Findings (implement phase, start) + +- Log dir for real sessions is **user-global**: `tower-terminals.ts:201` sets + `logDir: path.join(AGENT_FARM_DIR, 'logs')` → `~/.agent-farm/logs`. + (`TerminalManager`'s own default is `/.agent-farm/logs`, only used + by tests.) +- Per-file rotation **already exists**: `PtySession` caps at + `DEFAULT_DISK_LOG_MAX_BYTES` (50 MB) and rotates to `.log.1`. So the gap is + purely **cross-file retention** — nothing ever deletes a dead session's log. +- Filename shape: `.log` plus optional `.log.1`. +- Exemplar for the sweep: `shellper-husk-sweep.ts` + its wiring in `tower-server.ts` + (startup run + `setInterval`, env-configurable, dependency seams for tests). + Following that pattern rather than inventing a new one. + +## Design + +1. New `agent-farm/servers/session-log-sweep.ts`: + - `resolveLogRetentionDays(env)` — `AGENT_FARM_LOG_RETENTION_DAYS`, default 30, + `0` disables. + - `resolveLogSweepIntervalMs(env)` — `AGENT_FARM_LOG_SWEEP_INTERVAL_MS`, default 24h. + - `measureSessionLogs(logDir)` — footprint, for `codev doctor`. + - `sweepSessionLogs({...})` — delete logs whose mtime is older than retention, + skipping any id that is currently a live session. +2. Wire into `tower-server.ts`: startup (after terminal reconciliation, so live + sessions are in the map) + unref'd periodic timer, cleared on shutdown. +3. `codev doctor`: "Session Logs" health line with file count + footprint. +4. Unit tests for the sweep module. + +mtime is the retention key: the last write to a session log is effectively when that +session stopped producing output, which is the "session ended" signal the issue asks +for, without needing to join against DB session state. + +## Implement phase — done + +Shipped: +- `agent-farm/servers/session-log-sweep.ts` (new) — resolvers, `measureSessionLogs`, + `sweepSessionLogs`, `formatBytes`. +- `tower-server.ts` — startup sweep + footprint log line + unref'd daily timer, + cleared in shutdown. Placed after `reconcileTerminalSessions()` deliberately. +- `doctor.ts` — "Session Logs" section; warn past 2 GB. +- `arch.md` — paragraph next to the husk-sweep one. +- 18 unit tests. + +### Verification (beyond "it compiled") + +The worktree had **no `node_modules`** on spawn, so my first `npx tsc --noEmit` exited 0 +without actually resolving anything — a silent false pass. Ran `pnpm install +--frozen-lockfile` and re-ran everything. Worth knowing for other builders in this +cohort: don't trust a green typecheck before you've confirmed deps exist. + +- typecheck clean; `pnpm build` clean. +- 2205 unit tests pass (0 fail) across `src/__tests__/doctor.test.ts` + all of + `src/agent-farm/__tests__/`. +- **Real-data sweep**: ran the built module against an mtime-preserving *copy* of the + live `~/.agent-farm/logs` (737 files / 2.1 GB). 30d retention deleted 12 logs / 29 MB; + 7d retention deleted 418 logs / 1.4 GB. Never touched the real directory. +- **Live-session exclusion on real data**: passed a 23-day-old log's id as active + against a 7-day retention — it survived while 429 others were deleted. +- **`codev doctor` render**: ran the built CLI. The new section prints + `⚠ 741 file(s), 2.1 GB (retention 30d)` and the warning reaches the summary block. + +Did NOT restart Tower to observe the startup sweep live — that would kill every running +builder session in this workspace, and it needs the human's go-ahead. The wiring is +typechecked and the swept function is verified against real data; flagging the gap +rather than papering over it. + +## CMAP at PR (PR #1243) + +Ran codex + claude. Skipped the agy/gemini lane deliberately — that lane is known-broken +for `--type` reviews (returns no VERDICT). + +- **claude: APPROVE** (HIGH). One real finding: `MS_PER_DAY` declared but unused. + **Accepted** — exported it and used it in `tower-server.ts` for the `retentionMs` + computation, replacing an inline `24 * 60 * 60 * 1000`. +- **codex: REQUEST_CHANGES** (MEDIUM). Two findings: + 1. *"`doctor.ts` gained user-visible logic with no focused tests; it reads the real + user-global `~/.agent-farm/logs`, so it's insufficiently isolated."* — **Accepted the + substance.** Extracted `checkSessionLogs(logDir, env)` (exported) so thresholds, the + retention-disabled wording, and the remediation hint are unit-testable against a temp + dir with an injected env. Added `src/__tests__/doctor-session-logs.test.ts` (8 tests, + sparse files via `ftruncate` so the 2 GB threshold case costs no real disk). Verified + the rendered output is byte-identical after the refactor. Note `doctor()` itself still + reads the real log dir — that is the *feature*, not a defect. + 2. *"Remove `status.yaml` and `air-1238_thread.md` — workflow/runtime artifacts, not + feature code."* — **Rejected; the reviewer doesn't know Codev's conventions.** + `status.yaml` is porch-managed and is committed to main at the pr-ready gate by + design (it's where project completion stats live for non-bug projects). The thread + file's documented default disposition is COMMIT — it ships to `main` in + `codev/state/` alongside `codev/reviews/` and becomes part of the historical review + record. Stripping either would break the protocol, not clean up the PR. Flagging for + the architect rather than silently complying. + +### Unrelated observation (not fixed — out of scope) + +`codev doctor` reports "Architect state files (codev/state/.md) are not +gitignored" even though `.gitignore:15-16` carries the rule. + +**I first attributed this to `auditStateFileIgnore`. That was wrong** — the architect +pushed back with `git check-ignore -q codev/state/__doctor-probe__.md` exiting 0 in this +worktree (it does), and investigating properly found the real cause. + +**It's `findWorkspaceRoot()` (`doctor.ts:444-456`), and it reproduces on `main`.** The +walk tests `existsSync(resolve(current, 'codev'))`. Run from `packages/codev`, it walks up +to `packages/`, where `resolve('packages', 'codev')` matches **`packages/codev` — the +package directory, not a codev instance** — and returns `/packages` as the +workspace root. `auditStateFileIgnore` is then handed that wrong root and probes +`packages/codev/state/...`, which the root-anchored `codev/state/*.md` rule correctly does +not match. The audit logic is fine; its input isn't. + +Repro is purely about cwd: + +``` +cd /packages/codev && codev doctor → warns +cd && codev doctor → ✓ Project structure OK +``` + +I hit it because CLAUDE.md's Directory Map says to work from `packages/codev/`. Calling +the built audit directly (with node_modules present, so not a no-deps artifact): +`` → `[]`, `/packages` → warns, main root → `[]`, +`main/packages` → **warns**. + +Blast radius is wider than this one line: every workspaceRoot-derived doctor check +(framework-ref, PR-gate, drift, forge config) silently audits the wrong tree when doctor +is run from `packages/`. Those checks mostly fail *quietly* — a check with nothing to +audit reports ✓ — so doctor can print a clean bill of health for a tree it never looked +at. That's worse than the false warning, because it's invisible. + +**Filed as issue #1246** (`area/scaffold`) at the architect's direction, with the repro, +mechanism, evidence table, blast radius, and a fix direction. Two traps I encoded there +after checking the actual directories: `templates/` is useless as an instance signature +(`packages/codev/templates/` exists too, so it wouldn't fix the bug), and `protocols/` +must not be *required* (the four-tier resolver means a project that customizes nothing +legitimately has no local `codev/protocols/`). Also flagged that `.git` is a **file** in a +worktree gitlink, so any refactor to `statSync().isDirectory()` would break every builder +worktree. Deliberately **not** fixed in PR #1243 — keeping that PR scoped. + +### Why this worktree had no node_modules (architect asked me to keep the details) + +There is **no `.codev/config.json`** in the main checkout at all — only +`.codev/config.local.json`, holding just a `shell` block. So no `worktree.symlinks` and +no `worktree.postSpawn` exist, and a fresh worktree therefore gets no `pnpm install`. + +The dangerous part is the failure *mode*: with no node_modules, +`npx tsc --noEmit -p tsconfig.json` **exits 0 and prints nothing** — it resolves no types +and checks nothing, indistinguishable from a genuine clean typecheck. Only vitest failed +loudly (`Cannot find package 'vitest'`). A builder that ran just tsc would have reported +a green typecheck on entirely unverified code. Suggested guards: assert `node_modules` +exists before trusting a typecheck, and/or add +`worktree.postSpawn: ["pnpm install --frozen-lockfile"]`. diff --git a/packages/codev/src/__tests__/doctor-session-logs.test.ts b/packages/codev/src/__tests__/doctor-session-logs.test.ts new file mode 100644 index 000000000..538b8b51d --- /dev/null +++ b/packages/codev/src/__tests__/doctor-session-logs.test.ts @@ -0,0 +1,111 @@ +/** + * Issue #1238: tests for `codev doctor`'s "Session Logs" health line. + * + * Lives in its own file rather than doctor.test.ts because that file mocks + * node:child_process at module scope for the dependency-probe tests; this check + * is pure filesystem accounting and wants a real temp directory instead. + * + * The env is passed explicitly rather than mutated on `process.env`, so these + * cases stay hermetic regardless of what the developer running them has set. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { checkSessionLogs, SESSION_LOG_WARN_BYTES } from '../commands/doctor.js'; + +let logDir: string; + +/** A session log of an exact size, without actually writing GBs. */ +function writeSparseLog(name: string, size: number): void { + const fd = fs.openSync(path.join(logDir, name), 'w'); + if (size > 0) { + fs.ftruncateSync(fd, size); // sparse — no real bytes hit the disk + } + fs.closeSync(fd); +} + +beforeEach(() => { + logDir = fs.mkdtempSync(path.join(os.tmpdir(), 'doctor-session-logs-')); +}); + +afterEach(() => { + fs.rmSync(logDir, { recursive: true, force: true }); +}); + +describe('checkSessionLogs', () => { + it('is ok with a distinct message when there are no logs', () => { + const result = checkSessionLogs(logDir, {}); + + expect(result.status).toBe('ok'); + expect(result.files).toBe(0); + expect(result.bytes).toBe(0); + expect(result.summary).toBe('No PTY session logs on disk'); + expect(result.recommendation).toBeUndefined(); + }); + + it('is ok, with a footprint summary, below the warn threshold', () => { + writeSparseLog('a.log', 1024 * 1024); + writeSparseLog('b.log', 1024 * 1024); + + const result = checkSessionLogs(logDir, {}); + + expect(result.status).toBe('ok'); + expect(result.files).toBe(2); + expect(result.summary).toBe('2 file(s), 2.0 MB'); + expect(result.retentionNote).toBe('retention 30d'); + expect(result.recommendation).toBeUndefined(); + }); + + it('warns past the threshold and recommends a Tower restart', () => { + writeSparseLog('big.log', SESSION_LOG_WARN_BYTES); + + const result = checkSessionLogs(logDir, {}); + + expect(result.status).toBe('warn'); + expect(result.summary).toBe('1 file(s), 2.0 GB'); + expect(result.recommendation).toContain('restart Tower'); + }); + + it('stays ok exactly one byte below the threshold (boundary is inclusive)', () => { + writeSparseLog('big.log', SESSION_LOG_WARN_BYTES - 1); + + expect(checkSessionLogs(logDir, {}).status).toBe('ok'); + }); + + it('reports the configured retention window rather than assuming the default', () => { + writeSparseLog('a.log', 10); + + const result = checkSessionLogs(logDir, { AGENT_FARM_LOG_RETENTION_DAYS: '7' }); + + expect(result.retentionDays).toBe(7); + expect(result.retentionNote).toBe('retention 7d'); + }); + + it('names retention as the culprit when the sweep is disabled and the dir is large', () => { + writeSparseLog('big.log', SESSION_LOG_WARN_BYTES); + + const result = checkSessionLogs(logDir, { AGENT_FARM_LOG_RETENTION_DAYS: '0' }); + + expect(result.status).toBe('warn'); + expect(result.retentionNote).toBe('retention disabled'); + expect(result.recommendation).toContain('AGENT_FARM_LOG_RETENTION_DAYS=0'); + }); + + it('treats a missing log directory as healthy rather than throwing', () => { + const missing = path.join(logDir, 'never-created'); + + expect(() => checkSessionLogs(missing, {})).not.toThrow(); + expect(checkSessionLogs(missing, {}).status).toBe('ok'); + expect(checkSessionLogs(missing, {}).files).toBe(0); + }); + + it('ignores files that are not session logs', () => { + writeSparseLog('sess.log', 100); + fs.writeFileSync(path.join(logDir, 'tower.log.gz'), 'archive'); + fs.writeFileSync(path.join(logDir, 'notes.txt'), 'hello'); + + expect(checkSessionLogs(logDir, {}).files).toBe(1); + }); +}); diff --git a/packages/codev/src/agent-farm/__tests__/session-log-sweep.test.ts b/packages/codev/src/agent-farm/__tests__/session-log-sweep.test.ts new file mode 100644 index 000000000..21c51d668 --- /dev/null +++ b/packages/codev/src/agent-farm/__tests__/session-log-sweep.test.ts @@ -0,0 +1,213 @@ +/** + * Issue #1238: tests for PTY session log retention. + * + * Uses a real temp directory rather than mocking `fs`: the whole point of the + * module is filesystem behavior (mtime-based eligibility, unlink, resilience to + * unreadable entries), and a real tmpdir exercises that faithfully while staying + * fast. `now` is injected so age is deterministic without sleeping. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { + sweepSessionLogs, + measureSessionLogs, + resolveLogRetentionDays, + resolveLogSweepIntervalMs, + formatBytes, + DEFAULT_LOG_RETENTION_DAYS, + DEFAULT_LOG_SWEEP_INTERVAL_MS, +} from '../servers/session-log-sweep.js'; + +const DAY_MS = 24 * 60 * 60 * 1000; +const NOW = 1_800_000_000_000; // fixed clock so ages are exact +const RETENTION_MS = 30 * DAY_MS; + +let logDir: string; + +/** Write a session log with a specific age in days. */ +function writeLog(name: string, ageDays: number, contents = 'x'): string { + const full = path.join(logDir, name); + fs.writeFileSync(full, contents); + const mtime = new Date(NOW - ageDays * DAY_MS); + fs.utimesSync(full, mtime, mtime); + return full; +} + +beforeEach(() => { + logDir = fs.mkdtempSync(path.join(os.tmpdir(), 'session-log-sweep-')); +}); + +afterEach(() => { + fs.rmSync(logDir, { recursive: true, force: true }); +}); + +describe('sweepSessionLogs', () => { + it('deletes logs older than the retention window and keeps fresh ones', () => { + writeLog('old-a.log', 45); + writeLog('old-b.log', 31); + writeLog('fresh.log', 2); + writeLog('boundary.log', 29); + + const result = sweepSessionLogs({ logDir, retentionMs: RETENTION_MS, now: NOW }); + + expect(result.deleted).toBe(2); + expect(result.scanned).toBe(4); + expect(fs.existsSync(path.join(logDir, 'old-a.log'))).toBe(false); + expect(fs.existsSync(path.join(logDir, 'old-b.log'))).toBe(false); + expect(fs.existsSync(path.join(logDir, 'fresh.log'))).toBe(true); + expect(fs.existsSync(path.join(logDir, 'boundary.log'))).toBe(true); + }); + + it('reports bytes freed', () => { + writeLog('old.log', 40, 'a'.repeat(2048)); + writeLog('fresh.log', 1, 'b'.repeat(4096)); + + const result = sweepSessionLogs({ logDir, retentionMs: RETENTION_MS, now: NOW }); + + expect(result.deleted).toBe(1); + expect(result.bytesFreed).toBe(2048); + }); + + it('sweeps rotated .log.1 files too', () => { + writeLog('sess.log', 40); + writeLog('sess.log.1', 40); + + const result = sweepSessionLogs({ logDir, retentionMs: RETENTION_MS, now: NOW }); + + expect(result.deleted).toBe(2); + expect(fs.readdirSync(logDir)).toEqual([]); + }); + + it('never deletes the log of a live session, however old the mtime', () => { + writeLog('live-session.log', 400); + writeLog('live-session.log.1', 400); + writeLog('dead-session.log', 400); + + const result = sweepSessionLogs({ + logDir, + retentionMs: RETENTION_MS, + now: NOW, + activeSessionIds: ['live-session'], + }); + + expect(result.deleted).toBe(1); + expect(result.skippedActive).toBe(2); + expect(fs.existsSync(path.join(logDir, 'live-session.log'))).toBe(true); + expect(fs.existsSync(path.join(logDir, 'live-session.log.1'))).toBe(true); + expect(fs.existsSync(path.join(logDir, 'dead-session.log'))).toBe(false); + }); + + it('is a no-op when retention is disabled (0)', () => { + writeLog('ancient.log', 9999); + + const result = sweepSessionLogs({ logDir, retentionMs: 0, now: NOW }); + + expect(result).toEqual({ scanned: 0, deleted: 0, bytesFreed: 0, skippedActive: 0, failed: 0 }); + expect(fs.existsSync(path.join(logDir, 'ancient.log'))).toBe(true); + }); + + it('ignores non-log files and subdirectories', () => { + writeLog('sess.log', 40); + const keep = path.join(logDir, 'tower.log.gz'); + fs.writeFileSync(keep, 'archive'); + fs.utimesSync(keep, new Date(NOW - 400 * DAY_MS), new Date(NOW - 400 * DAY_MS)); + const subdir = path.join(logDir, 'nested.log'); + fs.mkdirSync(subdir); + + const result = sweepSessionLogs({ logDir, retentionMs: RETENTION_MS, now: NOW }); + + expect(result.scanned).toBe(1); + expect(result.deleted).toBe(1); + expect(fs.existsSync(keep)).toBe(true); + expect(fs.existsSync(subdir)).toBe(true); + }); + + it('returns zeros for a missing log directory instead of throwing', () => { + const missing = path.join(logDir, 'does-not-exist'); + + expect(() => sweepSessionLogs({ logDir: missing, retentionMs: RETENTION_MS, now: NOW })).not.toThrow(); + expect(sweepSessionLogs({ logDir: missing, retentionMs: RETENTION_MS, now: NOW }).deleted).toBe(0); + }); + + it('logs a summary only when it actually deleted something', () => { + writeLog('fresh.log', 1); + const lines: string[] = []; + sweepSessionLogs({ logDir, retentionMs: RETENTION_MS, now: NOW, log: (m) => lines.push(m) }); + expect(lines).toEqual([]); + + writeLog('old.log', 40); + sweepSessionLogs({ logDir, retentionMs: RETENTION_MS, now: NOW, log: (m) => lines.push(m) }); + expect(lines).toHaveLength(1); + expect(lines[0]).toContain('deleted 1 log(s)'); + }); +}); + +describe('measureSessionLogs', () => { + it('sums files and bytes and reports the oldest mtime', () => { + writeLog('a.log', 10, 'a'.repeat(100)); + writeLog('b.log', 50, 'b'.repeat(200)); + writeLog('b.log.1', 20, 'c'.repeat(300)); + + const footprint = measureSessionLogs(logDir); + + expect(footprint.files).toBe(3); + expect(footprint.bytes).toBe(600); + expect(footprint.oldestMtimeMs).toBeCloseTo(NOW - 50 * DAY_MS, -3); + }); + + it('is a healthy zero for a missing directory', () => { + expect(measureSessionLogs(path.join(logDir, 'nope'))).toEqual({ + files: 0, + bytes: 0, + oldestMtimeMs: null, + }); + }); +}); + +describe('resolveLogRetentionDays', () => { + it('defaults to 30 days', () => { + expect(resolveLogRetentionDays({})).toBe(DEFAULT_LOG_RETENTION_DAYS); + expect(DEFAULT_LOG_RETENTION_DAYS).toBe(30); + }); + + it('honours an explicit override', () => { + expect(resolveLogRetentionDays({ AGENT_FARM_LOG_RETENTION_DAYS: '7' })).toBe(7); + }); + + it('honours an explicit 0 as "disabled" rather than falling back to the default', () => { + expect(resolveLogRetentionDays({ AGENT_FARM_LOG_RETENTION_DAYS: '0' })).toBe(0); + }); + + it('falls back to the default for garbage and clamps negatives to 0', () => { + expect(resolveLogRetentionDays({ AGENT_FARM_LOG_RETENTION_DAYS: 'banana' })).toBe(30); + expect(resolveLogRetentionDays({ AGENT_FARM_LOG_RETENTION_DAYS: '-5' })).toBe(0); + }); +}); + +describe('resolveLogSweepIntervalMs', () => { + it('defaults to 24 hours', () => { + expect(resolveLogSweepIntervalMs({})).toBe(DEFAULT_LOG_SWEEP_INTERVAL_MS); + }); + + it('floors at 1s so a bad value cannot spin the event loop', () => { + expect(resolveLogSweepIntervalMs({ AGENT_FARM_LOG_SWEEP_INTERVAL_MS: '10' })).toBe(1000); + expect(resolveLogSweepIntervalMs({ AGENT_FARM_LOG_SWEEP_INTERVAL_MS: '0' })).toBe(1000); + }); + + it('honours a valid override', () => { + expect(resolveLogSweepIntervalMs({ AGENT_FARM_LOG_SWEEP_INTERVAL_MS: '5000' })).toBe(5000); + }); +}); + +describe('formatBytes', () => { + it('formats across units', () => { + expect(formatBytes(0)).toBe('0 B'); + expect(formatBytes(512)).toBe('512 B'); + expect(formatBytes(2048)).toBe('2.0 KB'); + expect(formatBytes(5 * 1024 * 1024)).toBe('5.0 MB'); + expect(formatBytes(19 * 1024 * 1024 * 1024)).toBe('19 GB'); + }); +}); diff --git a/packages/codev/src/agent-farm/servers/session-log-sweep.ts b/packages/codev/src/agent-farm/servers/session-log-sweep.ts new file mode 100644 index 000000000..6cfc9fb98 --- /dev/null +++ b/packages/codev/src/agent-farm/servers/session-log-sweep.ts @@ -0,0 +1,208 @@ +/** + * Issue #1238: retention for PTY session logs. + * + * `PtySession` already bounds a *single* session's log (50 MB, one `.1` + * rotation — see `../../terminal/pty-session.ts`), but nothing has ever + * deleted the log of a session that ended. Every terminal Tower has ever + * opened leaves a file behind forever: an audited machine reached 29,728 files + * / 19 GB in `~/.agent-farm/logs`, 97% of it untouched for over a month. + * + * This module adds the missing cross-file policy: a log is deleted once its + * last write is older than the retention window. mtime is the retention key + * because the last byte written to a session log is, in effect, the moment + * that session stopped producing output — the "session ended" signal, without + * needing to join against DB session state (which is itself pruned, so dead + * sessions leave no row to join against). + * + * Live sessions are excluded by id regardless of mtime. A session log's fd is + * held open for the session's whole life, so unlinking it under a live writer + * would silently redirect every subsequent write into an unlinked inode — + * losing the log of the one session someone might actually want to read. A + * 30-day-idle live session is vanishingly unlikely, but the guard is cheap. + * + * Wired in `./tower-server.ts` on two triggers (Tower startup, and a daily + * in-process timer), mirroring the husk sweep in `./shellper-husk-sweep.ts`. + */ + +import fs from 'node:fs'; +import path from 'node:path'; + +/** `.log` and its single rotation `.log.1`. */ +const SESSION_LOG_RE = /^(.+)\.log(\.1)?$/; + +export const DEFAULT_LOG_RETENTION_DAYS = 30; +export const DEFAULT_LOG_SWEEP_INTERVAL_MS = 24 * 60 * 60 * 1000; + +export const MS_PER_DAY = 24 * 60 * 60 * 1000; + +/** + * Retention window in days (env `AGENT_FARM_LOG_RETENTION_DAYS`, default 30). + * `0` disables the sweep entirely — an explicit "keep everything" opt-out for + * anyone who treats these logs as an audit trail. + * + * NaN-checked rather than `parsed || default` because `0` is a meaningful + * override and is falsy in JS (same reasoning as `resolveHuskGraceMs`). + */ +export function resolveLogRetentionDays(env: NodeJS.ProcessEnv = process.env): number { + const parsed = parseInt(env.AGENT_FARM_LOG_RETENTION_DAYS ?? '', 10); + return Math.max(Number.isNaN(parsed) ? DEFAULT_LOG_RETENTION_DAYS : parsed, 0); +} + +/** + * Sweep cadence (env `AGENT_FARM_LOG_SWEEP_INTERVAL_MS`, default 24h). Floored + * at 1s so a bad value can't spin the event loop; tests shorten it deliberately. + */ +export function resolveLogSweepIntervalMs(env: NodeJS.ProcessEnv = process.env): number { + const parsed = parseInt(env.AGENT_FARM_LOG_SWEEP_INTERVAL_MS ?? '', 10); + return Math.max(Number.isNaN(parsed) ? DEFAULT_LOG_SWEEP_INTERVAL_MS : parsed, 1000); +} + +export interface SessionLogFootprint { + /** Number of session log files (including `.log.1` rotations). */ + files: number; + /** Total bytes on disk. */ + bytes: number; + /** mtime of the oldest log, or null when there are none. */ + oldestMtimeMs: number | null; +} + +/** + * Total footprint of the session log directory. Read-only — used by + * `codev doctor` to surface the number the issue was filed about, and by the + * sweep's own logging. A missing directory is a healthy zero, not an error. + */ +export function measureSessionLogs(logDir: string): SessionLogFootprint { + const footprint: SessionLogFootprint = { files: 0, bytes: 0, oldestMtimeMs: null }; + let entries: string[]; + try { + entries = fs.readdirSync(logDir); + } catch { + return footprint; + } + + for (const name of entries) { + if (!SESSION_LOG_RE.test(name)) continue; + let stat: fs.Stats; + try { + stat = fs.statSync(path.join(logDir, name)); + } catch { + continue; // vanished mid-scan — nothing to account for + } + if (!stat.isFile()) continue; + footprint.files += 1; + footprint.bytes += stat.size; + if (footprint.oldestMtimeMs === null || stat.mtimeMs < footprint.oldestMtimeMs) { + footprint.oldestMtimeMs = stat.mtimeMs; + } + } + return footprint; +} + +export interface SweepSessionLogsOptions { + /** The session log directory, i.e. `~/.agent-farm/logs`. */ + logDir: string; + /** Age at which a log becomes eligible for deletion. `0` disables the sweep. */ + retentionMs: number; + /** + * Ids of sessions that are alive right now. Their logs are never deleted, + * however old the mtime looks, because their fd is still open. + */ + activeSessionIds?: Iterable; + /** Seam, defaults to `Date.now()`. */ + now?: number; + log?: (msg: string) => void; +} + +export interface SweepSessionLogsResult { + /** Session log files examined. */ + scanned: number; + /** Files unlinked. */ + deleted: number; + /** Bytes reclaimed (sum of the deleted files' sizes). */ + bytesFreed: number; + /** Aged-out files kept because their session is still live. */ + skippedActive: number; + /** Files that were eligible but could not be unlinked. */ + failed: number; +} + +/** + * Delete session logs whose last write is older than `retentionMs`. + * + * Never throws: a log directory that can't be read, or an individual file that + * can't be stat'd or unlinked, is counted and skipped. Log retention is + * janitorial — it must not be able to take Tower's startup path down with it. + */ +export function sweepSessionLogs(opts: SweepSessionLogsOptions): SweepSessionLogsResult { + const result: SweepSessionLogsResult = { + scanned: 0, + deleted: 0, + bytesFreed: 0, + skippedActive: 0, + failed: 0, + }; + if (opts.retentionMs <= 0) return result; + + const now = opts.now ?? Date.now(); + const active = new Set(opts.activeSessionIds ?? []); + + let entries: string[]; + try { + entries = fs.readdirSync(opts.logDir); + } catch { + return result; // no log dir yet (fresh install) — nothing to sweep + } + + for (const name of entries) { + const match = SESSION_LOG_RE.exec(name); + if (!match) continue; + const fullPath = path.join(opts.logDir, name); + + let stat: fs.Stats; + try { + stat = fs.statSync(fullPath); + } catch { + continue; + } + if (!stat.isFile()) continue; + result.scanned += 1; + + if (now - stat.mtimeMs < opts.retentionMs) continue; + if (active.has(match[1])) { + result.skippedActive += 1; + continue; + } + + try { + fs.unlinkSync(fullPath); + result.deleted += 1; + result.bytesFreed += stat.size; + } catch { + result.failed += 1; + } + } + + if (result.deleted > 0 || result.failed > 0) { + opts.log?.( + `Session log sweep: deleted ${result.deleted} log(s), freed ${formatBytes(result.bytesFreed)}` + + ` (${result.scanned} scanned` + + (result.skippedActive > 0 ? `, ${result.skippedActive} kept as live` : '') + + (result.failed > 0 ? `, ${result.failed} failed` : '') + + ')', + ); + } + return result; +} + +/** Human-readable byte count for log lines and `codev doctor` output. */ +export function formatBytes(bytes: number): string { + if (bytes < 1024) return `${bytes} B`; + const units = ['KB', 'MB', 'GB', 'TB']; + let value = bytes / 1024; + let unit = 0; + while (value >= 1024 && unit < units.length - 1) { + value /= 1024; + unit += 1; + } + return `${value < 10 ? value.toFixed(1) : Math.round(value)} ${units[unit]}`; +} diff --git a/packages/codev/src/agent-farm/servers/tower-server.ts b/packages/codev/src/agent-farm/servers/tower-server.ts index d53515ab3..e4c6680c3 100644 --- a/packages/codev/src/agent-farm/servers/tower-server.ts +++ b/packages/codev/src/agent-farm/servers/tower-server.ts @@ -19,6 +19,14 @@ import { SessionManager } from '../../terminal/session-manager.js'; import type { SSEClient } from './tower-types.js'; import { startRateLimitCleanup } from './tower-utils.js'; import { sweepShellperHusks, resolveHuskGraceMs } from './shellper-husk-sweep.js'; +import { + sweepSessionLogs, + measureSessionLogs, + resolveLogRetentionDays, + resolveLogSweepIntervalMs, + formatBytes, + MS_PER_DAY, +} from './session-log-sweep.js'; import { initTunnel, shutdownTunnel, @@ -53,7 +61,7 @@ import type { RouteContext } from './tower-routes.js'; import { setCodevConfigNotifier, stopAllCodevConfigWatchers } from './codev-config-watcher.js'; import { getGlobalDb } from '../db/index.js'; import { runBootConsolidation } from '../db/consolidate.js'; -import { DEFAULT_TOWER_PORT } from '../lib/tower-client.js'; +import { DEFAULT_TOWER_PORT, AGENT_FARM_DIR } from '../lib/tower-client.js'; import { validateHost } from '../utils/server-utils.js'; import { version } from '../../version.js'; @@ -68,6 +76,7 @@ let shellperManager: SessionManager | null = null; let shellperCleanupInterval: NodeJS.Timeout | null = null; let shellperHuskSweepInterval: NodeJS.Timeout | null = null; let terminalPartialMonitorInterval: NodeJS.Timeout | null = null; +let sessionLogSweepInterval: NodeJS.Timeout | null = null; // Observability for Issue #1047: the ring-buffer partial is kept whole (no // byte cap) so reconnection replay stays faithful, which means a no-newline @@ -169,6 +178,7 @@ async function gracefulShutdown(signal: string): Promise { if (shellperCleanupInterval) clearInterval(shellperCleanupInterval); if (shellperHuskSweepInterval) clearInterval(shellperHuskSweepInterval); if (terminalPartialMonitorInterval) clearInterval(terminalPartialMonitorInterval); + if (sessionLogSweepInterval) clearInterval(sessionLogSweepInterval); clearInterval(sseHeartbeatInterval); // 4b. Flush and stop send buffer (Spec 403) — delivers any deferred messages @@ -525,6 +535,40 @@ server.listen(port, bindHost, async () => { // reconciliation so a reconnected session's shellper is registered). await runHuskSweep(); + // Issue #1238: PTY session log retention. Session logs are per-session files + // in ~/.agent-farm/logs that nothing ever deleted, so they accreted forever + // (19 GB / 29,728 files on an audited machine). Sweep aged-out logs at + // startup and then daily. Deliberately placed AFTER reconcileTerminalSessions + // so surviving sessions are in the terminal manager's map and are excluded by + // id — the sweep must never unlink a log whose fd is still open. + const logRetentionDays = resolveLogRetentionDays(); + const sessionLogDir = path.join(AGENT_FARM_DIR, 'logs'); + const runSessionLogSweep = (): void => { + try { + sweepSessionLogs({ + logDir: sessionLogDir, + retentionMs: logRetentionDays * MS_PER_DAY, + activeSessionIds: getTerminalManager().listSessions().map((s) => s.id), + log: (msg: string) => log('INFO', msg), + }); + } catch (err) { + log('ERROR', `Session log sweep failed: ${(err as Error).message}`); + } + }; + if (logRetentionDays === 0) { + log('INFO', 'Session log retention disabled (AGENT_FARM_LOG_RETENTION_DAYS=0)'); + } else { + runSessionLogSweep(); + const footprint = measureSessionLogs(sessionLogDir); + log( + 'INFO', + `Session logs: ${footprint.files} file(s), ${formatBytes(footprint.bytes)} ` + + `(retention ${logRetentionDays}d)`, + ); + sessionLogSweepInterval = setInterval(runSessionLogSweep, resolveLogSweepIntervalMs()); + sessionLogSweepInterval.unref(); + } + // Spec 0105 Phase 3: Initialize instance lifecycle module. // Placed after reconciliation so getInstances() returns [] during startup // (since _deps is null), preventing race conditions with reconciliation. diff --git a/packages/codev/src/commands/doctor.ts b/packages/codev/src/commands/doctor.ts index 5bdb83182..c67734e6a 100644 --- a/packages/codev/src/commands/doctor.ts +++ b/packages/codev/src/commands/doctor.ts @@ -22,6 +22,12 @@ import { formatStaleness, } from '../lib/protocol-drift-audit.js'; import { resolveAgyBin, AGY_OAUTH_MARKERS } from './consult/index.js'; +import { AGENT_FARM_DIR } from '@cluesmith/codev-core/constants'; +import { + measureSessionLogs, + resolveLogRetentionDays, + formatBytes, +} from '../agent-farm/servers/session-log-sweep.js'; const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename); @@ -47,6 +53,72 @@ interface CheckResult { const isMacOS = process.platform === 'darwin'; +/** + * Footprint at which the PTY session log directory becomes a reportable warning + * rather than an informational line (#1238). Well past what a healthy retention + * sweep leaves behind, and well under the 19 GB that prompted the issue. + */ +export const SESSION_LOG_WARN_BYTES = 2 * 1024 * 1024 * 1024; + +export interface SessionLogCheck { + status: 'ok' | 'warn'; + files: number; + bytes: number; + retentionDays: number; + /** Human-readable footprint, e.g. `741 file(s), 2.1 GB`. No colour codes. */ + summary: string; + /** e.g. `retention 30d` / `retention disabled`. */ + retentionNote: string; + /** Present only when `status === 'warn'`. */ + recommendation?: string; +} + +/** + * Decide what the "Session Logs" doctor section should say (#1238). + * + * Split out from `doctor()` so the decision — thresholds, the retention-disabled + * wording, the remediation hint — is unit-testable against a temp directory. + * `doctor()` itself passes the real user-global `~/.agent-farm/logs`, which is + * the whole point of the check, so the *reporting* path is deliberately not + * hermetic; this function is where the logic lives and is pinned by tests. + */ +export function checkSessionLogs( + logDir: string, + env: NodeJS.ProcessEnv = process.env, +): SessionLogCheck { + const { files, bytes } = measureSessionLogs(logDir); + const retentionDays = resolveLogRetentionDays(env); + const retentionNote = retentionDays === 0 ? 'retention disabled' : `retention ${retentionDays}d`; + + if (files === 0) { + return { + status: 'ok', + files, + bytes, + retentionDays, + summary: 'No PTY session logs on disk', + retentionNote, + }; + } + + const summary = `${files} file(s), ${formatBytes(bytes)}`; + if (bytes < SESSION_LOG_WARN_BYTES) { + return { status: 'ok', files, bytes, retentionDays, summary, retentionNote }; + } + + return { + status: 'warn', + files, + bytes, + retentionDays, + summary, + retentionNote, + recommendation: retentionDays === 0 + ? 'retention is disabled (AGENT_FARM_LOG_RETENTION_DAYS=0) — unset it and restart Tower to let the sweep run' + : 'restart Tower to run the retention sweep (afx tower stop && afx tower start)', + }; +} + /** * Compare semantic versions: returns true if v1 >= v2 */ @@ -931,6 +1003,28 @@ export async function doctor(): Promise { console.log(''); } + // Session log footprint (#1238). User-global (~/.agent-farm/logs), so this is + // reported outside the project-scoped section. Tower sweeps aged-out logs, but + // a machine that has not run a recent Tower — or has retention disabled — can + // still be sitting on many GB, and that was invisible until this line existed. + const sessionLogs = checkSessionLogs(resolve(AGENT_FARM_DIR, 'logs')); + console.log(chalk.bold('Session Logs') + ' (~/.agent-farm/logs)'); + console.log(''); + if (sessionLogs.status === 'warn') { + console.log(` ${chalk.yellow('⚠')} ${sessionLogs.summary} ${chalk.blue(`(${sessionLogs.retentionNote})`)}`); + warnings++; + warningDetails.push({ + name: 'Session logs', + issue: `${sessionLogs.summary} in ~/.agent-farm/logs`, + recommendation: sessionLogs.recommendation, + }); + } else if (sessionLogs.files === 0) { + console.log(` ${chalk.green('✓')} ${sessionLogs.summary}`); + } else { + console.log(` ${chalk.green('✓')} ${sessionLogs.summary} ${chalk.blue(`(${sessionLogs.retentionNote})`)}`); + } + console.log(''); + // Summary console.log('============================================'); if (errors > 0) {