Add graph serialization - #1009
Open
paracycle wants to merge 3 commits into
Open
Conversation
Indexing and resolving a large workspace costs tens of seconds and gigabytes of memory, and the graph is thrown away when the process exits. Add a snapshot format so a later process can read the graph back instead of building it again. The format is an rkyv archive: the bytes on disk are the in-memory layout of the graph. `Snapshot::open` maps the file and casts a pointer, so it allocates nothing and copies nothing. The operating system pages in only the parts a query touches. `Snapshot::to_graph` is the opposite trade, and returns an owned, mutable graph. Every id is already a deterministic xxh3 content hash, so ids stay valid in another process and nothing needs relocating on load. Two types cannot derive the rkyv traits: - `Id<T>` pairs a `NonZeroU64` with a zero-sized marker. It archives as a bare little-endian u64, so an archived map is keyed by the eight bytes the id already is. - `DefinitionFlags` comes from `bitflags!`, which hides the backing field, so it archives through its u8 bits. `Config` keeps a `Box<Path>`, which is unsized and foreign, so it travels as a string through an `ArchiveWith` wrapper. `Document` no longer stores its line index directly. `LineIndex` has no constructor but `new(text)`, and its fields are private, so an archived one could never be rebuilt. The field becomes a `OnceLock` that a restored document fills from disk on first use. This adds the library API only. No tool reads or writes a snapshot yet.
A snapshot could not tell that the workspace had moved on, so a caller had to trust it blindly or throw it away. Record a Merkle tree of the files the snapshot was built from, and re-index only what changed. A root hash answers "did anything change" in one comparison, but a root is only as cheap as the leaves under it. Measured on a workspace of 110,447 files: recursive walk of the tree 1.415 s stat every known file, in parallel 74 ms stat every known directory, in parallel 22 ms re-read one changed directory 89 us fold every leaf into a root 3.7 ms Folding is free, so the tree does not pay for itself at the leaves. It pays at the interior nodes, because editing a file bumps that file's mtime and not its directory's, while adding, removing or renaming an entry bumps the containing directory's. Recording directories as interior nodes therefore removes the walk: stat the known files to catch edits and deletions, stat the known directories to catch insertions, and re-read only the directories whose mtime moved. `Verification::Metadata` trusts size and mtime, the same bet make and cargo make. `Verification::Content` re-hashes every file instead, and costs about twenty times as much. `Snapshot::catch_up` applies the result: it drops documents whose file is gone, re-indexes the modified and the new ones, and resolves the invalidated subset. A test asserts that the outcome equals a full rebuild, declaration for declaration. The manifest also records the exclusion patterns it was built with. Discovery must apply the same rule the original listing did, or a reload would resurrect files the workspace deliberately skips. The manifest and the graph are two archives in one file, so a staleness check reads megabytes and never faults in the graph.
The snapshot machinery existed but every caller had to drive it. Move the decision into graph construction, so `rdx query`, `rdx lint` and `rdx mcp` all get it through the graph they already build, with no per-command plumbing. `snapshot::load_or_index` is the single entry point. It reuses a snapshot when one applies, re-indexes only the files that changed and resolves the invalidated subset. Otherwise it indexes everything, resolves, and writes a snapshot for the next run. Either way the graph comes back resolved, so `Graph#index_workspace` no longer needs a separate `resolve` call. A snapshot is rejected, and the run falls back to a full index, when the file is missing, unreadable or a different format version, when it was built from different roots, or when it was built with different exclusion patterns. A different exclusion set changes the file list in a way no per-file check can detect. The snapshot lives under the user cache directory, honouring XDG_CACHE_HOME, named by a hash of the indexed roots. It must not sit inside the workspace: a file written into an indexed directory bumps that directory's mtime, and the tree would then report a change on every run. Failing to write a snapshot is not an error. The graph is correct either way and the next run pays the full cost again. On a workspace of 110,698 files, `rdx query` drops from 32.07 s to 4.42 s, and reports 1 file re-indexed after a single edit. Pass `--no-cache` to restore the previous behaviour, or `--verify-content` to compare file contents instead of mtimes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This is a prototype of what a graph serialization for Rubydex might look like.
After some investigation with Claude for potential opportunities for graph serialization, we settled in on using the
rkyv(get it? pronouncesarchivewhen read 😂) for zero-copy serialization and cheap mmap'ed deserialization.The integration with
rkyvis mainly via deriving the crate's traits, so the intrusiveness is minimal, except for a few types that need special handling.A report against Core is as following: