codegen: retained-on-failure LLVM scratch is unbounded and never reaped — one failed multi-module compile can retain GB, and it compounds into the next attempt

[proggeramlug]

Measured on two machines while working #8228's workaround. Filing with a correction to the obvious framing: this is not the compiler forgetting to clean up. crates/perry-codegen/src/linker.rs:712-720 states the policy deliberately, and the reasoning is good:

• success: the object is read into memory, and the per-call directory goes with it — a compile leaves nothing behind (codegen: content-addressed temp .ll files are never deleted (~1 GB on a dev box) #7144).
• failure (clang non-zero, or the object cannot be read): everything is left on disk. The error message names the .ll, and a failed compile is exactly when someone wants to look at the IR that produced it.

Retention on failure is intentional and useful. The defect is that it is unbounded in three directions at once, and the combination is what bites.

1. Unbounded per compile: one dir per failing module, not one per failure

A whole-project compile that fails retains a scratch dir for every failing module. My failed 104-module fixture build left 124 directories totalling 4.9 GB in $TMPDIR from a single attempt. All 105 module failures had the identical cause, so 104 of those IR copies serve no debugging purpose whatever — the policy's rationale ("someone wants to look at the IR") is fully satisfied by the first one.

This is much worse on the PERRY_LLVM_INPROCESS=off text path (#8228's interim workaround), which materialises 100–200 MB of IR for the oversized modules — 171.3 MB for app-page.runtime.prod.js on my box, 222.6 MB for one module on another.

2. Unbounded over time: nothing ever reaps it

Two machines, neither recently provisioned:

host	stale perry_llvm_scratch_* dirs	older than 2 h	size	oldest
dev box	1,180	1,177	567 MB	2026-08-15
bench mini	4,155	4,155	~627 MB	—

None had a live owning process. The size is modest; the inode pressure and the readdir cost on a temp dir with thousands of entries are not, and neither is the confusion of picking through them.

3. It compounds: attempt N's retention causes attempt N+1's failure

This is the part that turns an annoyance into a trap. Attempt N fails → retains ~5 GB → attempt N+1 hits ENOSPC because of attempt N → retains another ~5 GB. And per #8228 (comment 5309169441), ENOSPC does not present as a disk error: clang -c exits non-zero with empty stderr, so every module reports clang -c failed (stderr empty) and the printed diagnostic suggests a triple/ABI mismatch and points at PERRY_LLVM_CLANG. The user chases a toolchain theory while the real cause is a disk their own previous attempt filled.

Suggested fixes, cheapest first

1. Cap retention per compile. Keep the first failing module's scratch and drop the rest, or keep at most K. The error already names the retained path, so one example is what the policy is actually for. This alone removes the GB-scale case.
2. Reap on startup. Remove perry_llvm_scratch_* older than some threshold with no live owner. Note the owner check matters: the pid is in the directory name, and on a shared box other sessions' compiles are live in the same $TMPDIR — there were six live perry compiles on mine while I was cleaning up my own, matched by the pid-derived prefix in my error message.
3. Distinguish environmental failure from compile failure. An empty clang stderr with a non-zero exit is not IR anyone wants to inspect. Checking free space there and saying so would fix both this and the misdirection in [codegen] In-process LLVM backend fails the 5 biggest Next App Route modules on current main (silent per-module failure; regression in 3c95020f8..07c8040bf) #8228.

crates/perry-codegen/src/linker_temp_lifecycle_tests.rs:483 already records a prior incident of this family ("58 compiles left 58 perry_llvm_scratch_<pid>_<counter>"), so the lifecycle has a test suite to extend rather than one to invent.