feat(mem-wal): derive index catch-up from the version a commit read

[hamersaw]

What

An index absent from index_catchup reads as fully caught up, so the WAL may retire an SSTable whose rows no index holds. Nothing ever wrote the field, so every table reads that way today.

This records it at commit time, derived from state the commit already has.

Alternative design for the problem #8263 addresses, opened as a draft for discussion rather than to compete with it. Same goal and the same safety property; the difference is that nothing is transmitted, so transaction.proto, the feature flags, and the cross-language bindings are untouched.

How

An index covering every fragment that was live at the transaction's read version holds every row compaction had copied into the base table by then, so it is caught up to the compacted_sstables progress at that version.

That is the only proof available: nothing maps a compaction generation to the fragments its rows landed in, so covering the whole table as the transaction read it is how an index shows it covered those rows. Fragments appended since are a later catch-up gap.

Both halves come from state the commit already holds:

Needed	Source
fragments the index was built against	read version's manifest
generations it may claim	read version's __lance_mem_wal details
which indices, and what they cover	the final index list being published

read_version is already on every transaction and commit_transaction already materializes that version, so no claim is passed and no message changes.

build_manifest gains one parameter, Option<ReadVersionState> — the read version's manifest and index list, symmetric with the existing current_manifest / current_indices. It is None only where there is no version to read: dataset creation and detached commits. Deriving MemWAL's view from it happens inside build_manifest, so the commit path stays generic — commit.rs needs no knowledge of the system index's name or layout, and anything else that later needs the read version has it.

Per logical index:

• covers the read version → credit compacted_sstables@V, capped by what the commit itself records as compacted (so a read version since rolled back cannot retire SSTables no live commit copied in)
• otherwise physically unchanged → carry the prior position forward
• otherwise → generation 0

Every index is named explicitly, including at 0, because a missing entry reads as "caught up". A position is never lowered, so a commit reading an older version does not withdraw coverage an index already demonstrated.

Why not derive it purely at commit time

Two simpler variants were tried and are unsound or ineffective:

• Compare the index against the fragments live at commit time. The index was built at V but is judged at W. Any commit landing in between — a compaction flushing SSTables in, an ordinary append — makes the check fail, so on a table taking writes it never fires.
• Store a scalar max_fragment_id watermark per generation. Live fragments are not a prefix of the id space. Background compaction rewrites F1, F2 → F3, so {live : id ≤ watermark} shrinks or empties, and an empty required set is a subset of every index — including one holding none of those rows. It fails in the unsafe direction, and with the frag-reuse index deferring remapping (conflict_resolver.rs), it fails on the ordinary path rather than in a rare race.

Anchoring to the read version avoids both: the index and the fragment set are captured in the same id-space, and a rewrite moves rows between fragments without destroying them, so the conclusion stays true afterward.

Scope

protos/transaction.proto	untouched
protos/table.proto	comments only
feature_flags.rs	untouched
OptimizeOptions	untouched
Python / Java bindings	untouched

Consumers need no new API — read index_catchup at trim time.

Tests

12 unit tests in dataset::transaction::tests::mem_wal_index_coverage: coverage recorded; capped by the read version; capped by the commit; explicit 0 when unproven; carry-forward for an unchanged index; dropped for a changed one; never lowering a recorded position; union across delta segments; segment without a fragment bitmap; no-op leaves the system index UUID untouched; never-compacted table records nothing; indices judged independently.

• cargo test -p lance --lib — 2854 passed, 0 failed
• cargo clippy -p lance --tests -- -D warnings — clean
• cargo fmt --all -- --check — clean

Those three were run against the first commit. The follow-up refactor moves types and parameters without touching the coverage rule; it is cargo check --tests and cargo fmt clean, and relies on CI for the rest.

Notes for review

• No mixed-version writers. An older writer can change an index without withdrawing its recorded position. This assumes the compactor and trim logic move in tandem with this change.
• Expect SSTable retention to increase initially. is_index_caught_up returns true unconditionally today because nothing writes the field. Indices that genuinely lag will now report 0. That is the point of the change, but it will look like a storage regression the first time it happens.
• No partial credit. An index covering 90% of the read version records nothing. Addressing that needs a generation → fragment mapping recorded at compaction time.
• Stable row ids would collapse this area. Every rejected variant above failed on fragment ids being an unstable coordinate system; stable row ids do not move under rewrite. See the existing // TODO: this will change with stable row ids. on the Rewrite arm of the index conflict rules.
• Related: fix(mem-wal): carry fragments through UpdateMemWalState #8438 (UpdateMemWalState publishes no fragments). This design commits no standalone UpdateMemWalState, so it does not make that latent bug reachable.

🤖 Generated with Claude Code

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.