fix(timeline): write timeline.json atomically - #1200
will-lamerton merged 2 commits into
Conversation
will-lamerton
left a comment
There was a problem hiding this comment.
Thanks for this - the diagnosis is right and the fix is correctly scoped. I verified locally: the spec passes on your branch (20/20), and reverting only timeline-manager.ts to the parent commit fails exactly one test (saves the index atomically over a read-only live file) with EACCES ... open timeline.json, so the discriminating test really does discriminate. biome and tsc are clean.
One change needed before merge:
Use the shared atomic-write util instead of a fourth copy. source/utils/atomic-write.ts already exists (atomicWriteFileSync / atomicWriteJson), and there are two private async duplicates in source/session/session-manager.ts:64 and source/memory/semantic-memory-manager.ts:111. The helper added at source/services/timeline-manager.ts:41 is a byte-identical third async copy. Please add the async variant to source/utils/atomic-write.ts and import it here:
export async function atomicWriteFile(
filePath: string,
data: string,
options?: {mode?: number},
): Promise<void>Migrating the two existing copies can be a follow-up; a new call site just shouldn't fork the pattern again.
Two optional notes:
saveIndexassignsthis.index = indexbefore the write, andloadIndexreturns the cached value, so a failed save leaves the process holding an index that never reached disk. Your "keeps the previous index intact" test only asserts the on-disk bytes, so it passes while the in-memory guarantee it names does not hold. Restoring the previousthis.indexin a catch would make the claim true end to end. Pre-existing behaviour, so your call.- Both new permission tests rely on POSIX mode bits: as root (Docker dev containers) the
t.throwsAsynccase will not throw, and on Windowschmod(dir, 0o555)is a no-op. CI is non-root ubuntu so this is green today, but aprocess.platform === 'win32' || process.getuid?.() === 0skip guard would save contributors some confusion.
Nit: the changeset says a crash or kill mid-save "can no longer leave a truncated index behind". True for process death, which is what #1130 is about, but rename without fsync does not cover power loss. Not worth adding fsync on every capture, just worth softening the wording.
saveIndex() overwrote the live index in place, so a crash or kill mid-write could truncate timeline.json and silently discard every checkpoint of the session on next load. Write to a temp file and rename into place instead, mirroring the atomic writes already used by session-manager and the stats ledger. Closes Nano-Collective#1130
…ailed save Address review feedback on Nano-Collective#1200: - Add async atomicWriteFile(filePath, data, {mode}) to the shared source/utils/atomic-write.ts instead of a third private copy, and use it from TimelineManager.saveIndex. Existing session-manager and semantic-memory copies are left alone as follow-up material. - On a failed save, drop the unpersisted cache entry and reload the index from disk, so reads after a failure report what was actually persisted. - Skip the chmod-based tests on Windows and as root, where mode bits do not fail writes. - Soften the changeset: rename covers process death, not power loss. Closes Nano-Collective#1130
40cbed1 to
2764a5f
Compare
|
Thanks for verifying the discriminating test, and for pointing at the shared util I missed. |
|
Hi, @will-lamerton I have made the required changes as request could you verify once again please. |
|
Great work @puri-adityakumar :) |
Description
Closes #1130.
TimelineManager.saveIndex()overwrote the livetimeline.jsonin place, so a crash mid-save could truncate the index.loadIndex()then discards an unparseable index and starts empty, silently losing every checkpoint of the session.What changed:
atomicWriteFile(filePath, data, options?: {mode?: number})to the sharedsource/utils/atomic-write.ts(with spec coverage) and switchedsaveIndex()to it: write to a temp file, rename into place. Readers only ever see the complete old or complete new index.tmplitterThis covers process death mid-save; rename without fsync does not cover power loss (noted in a code comment).
Not changed on purpose: before-image files in
capture()stay non-atomic. They land in a fresh per-entry UUID dir before the index is saved, so a torn write there only leaves an orphan thatpruneStaleSessionscleans up later. The index is the single point that needed atomicity. Migrating the two remaining private async copies insession-manager.tsandsemantic-memory-manager.tsto the shared util is left as follow-up material.Type of Change
Changeset
pnpm changeset) describing this change for the changelogTesting
Automated Tests
.spec.ts/tsxfilespnpm test:allcompletes successfully)Note: 4 git-tool tests (
git-diff,getGitStatusSummarySync) fail on a cleanmaincheckout too. They spawn temp repos and the repo's own commit-msg hook rejects their fixture commits in this environment. Unrelated to this PR.New tests:
timeline-manager.spec.ts: saves atomically over a read-only live file (the discriminating test, fails on the old implementation), keeps the previous index intact on disk and in memory when a save fails, recovers from a corrupted indexatomic-write.spec.ts(new file): rename-based write, replace-existing,modehonored, no.tmpleft on failureManual Testing
Verified failure modes via read-only file/directory injection (EACCES) and truncated-JSON corruption. Biome, tsc and knip all clean.
Checklist