feat: disk-backed primitives — skipLoad, getManifest, streamSnapshot, getWalOps#11
Merged
Conversation
… getWalOps - skipLoad option: open store for writes without loading snapshot/WAL into memory - getManifest(): public accessor for current manifest (snapshot/WAL paths) - streamSnapshot(): async generator yielding [id, record] from snapshot - getWalOps(since?): async generator yielding WAL ops for incremental replay - skipLoad disables checkpointOnClose (empty Map would overwrite real data) - 12 new tests in disk-primitives.test.ts
…ation, test coverage - Return read-only ManifestInfo from getManifest() instead of raw Manifest - Single-writer getWalOps yields directly without intermediate allOps array - Add JSDoc noting streamSnapshot memory characteristics - Add 6 tests: error-before-open, snapshot-after-delete, read-only manifest, skipLoad+archive, covering all reviewer gaps
Snapshots now written as JSONL: header line with version/timestamp,
followed by one {id, data} line per record. Enables streamSnapshot()
to use readline for true streaming — one record in memory at a time.
Legacy monolithic JSON snapshots auto-detected and read correctly.
Snapshot file extension changed from .json to .jsonl.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
skipLoadoption —store.open(dir, { skipLoad: true })opens the store for writes without loading snapshot or replaying WAL into memory. Guardscompact()to prevent writing empty snapshots.store.getManifest()— returns read-onlyManifestInfowith snapshot/WAL paths and statsstore.streamSnapshot()— async generator yielding[id, record]pairs with true streaming viareadline(JSONL format)store.getWalOps(sinceTimestamp?)— async generator yielding WAL operations for incremental replay. Multi-writer ops merge-sorted by Lamport clock.ManifestInfotype — read-only facade for manifest dataThese primitives enable AgentDB v1.2.0 to build a Parquet-backed disk storage mode on top of opslog.
Breaking
.jsonto.jsonl. Existing.jsonsnapshots still readable. Custom StorageBackend implementations may need updating (see opslog-s3 task).Test plan