perf: serialize updated_fragment_offsets as RoaringBitmap bytes in proto by jerryjch · Pull Request #7432 · lance-format/lance

jerryjch · 2026-06-24T01:22:17Z

Summary

Follow-up to #6650. The updated_fragment_offsets field (proto field 9) stores per-fragment
matched row offsets as map<uint64, UInt32List> -- one uint32 per matched row. For dense
rewrites this produces multi-GB manifests (e.g. 86k matched rows x 4 bytes x many fragments).

This PR adds proto field 10 (map<uint64, bytes>) using portable RoaringBitmap serialization,
which typically compresses the same data to tens of bytes per fragment. Writers emit field 10
only; readers prefer field 10, falling back to field 9 for manifests written before this change.

Background

PR #6650 added updated_fragment_offsets to the Update transaction message so that
build_manifest can partially refresh _row_last_updated_at_version for matched rows only.
The encoding choice -- one uint32 per offset in a UInt32List -- was flagged post-merge as a
size regression for dense updates. The offsets are already stored internally as RoaringBitmap;
this PR aligns the proto encoding with that representation.

Changes

protos/transaction.proto

Deprecate field 9 (map<uint64, UInt32List> updated_fragment_offsets) with a comment
pointing to field 10.
Add field 10: map<uint64, bytes> updated_fragment_offset_bitmaps with documentation of
the dual-read strategy.

rust/lance/src/dataset/transaction.rs

Serialization (From<&Transaction> for pb::Transaction):

Write field 10 only: RoaringBitmap::serialize_into produces portable bytes for each
fragment's bitmap.
Set field 9 to an empty HashMap (forward compat; old readers ignore unknown fields).

Deserialization (TryFrom<pb::Transaction> for Transaction):

If field 10 is non-empty: deserialize each entry with RoaringBitmap::deserialize_from.
Else if field 9 is non-empty: convert each UInt32List to RoaringBitmap::from_iter
(legacy fallback).
Same if !new_field.is_empty() { ... } else { ... } pattern used by the existing
Rewrite.groups / Rewrite.old_fragments migration.

Invalid field 10 bytes fail deserialize with Error::invalid_input.

In-memory type unchanged: UpdatedFragmentOffsets(HashMap<u64, RoaringBitmap>).

Test plan

test_proto_round_trip_field_10 -- write a transaction with field 10, read back, verify
offsets match for two fragments.
test_proto_legacy_field_9_read -- construct a proto with only field 9 populated
(simulating an old writer), deserialize, verify offsets are correctly recovered.
test_proto_field_10_takes_precedence_over_field_9 -- when both fields are present,
field 10 values are used and field 9 is ignored.

Proto wire format change; team vote may be needed.

Backward compatibility

Proto field numbers: field 9 is kept (deprecated, not removed). Field 10 is new. No field
number reuse.
Old readers: ignore unknown field 10; they only read field 9, which is now empty on new
commits. Old Lance versions deserializing commits written by this PR will not recover
offsets from the txn; that only affects audit/readTransaction() on historical
commits, not table data or OCC.
New readers: prefer field 10; fall back to field 9 for manifests written by older Lance
versions that predate this change.
No JNI or Java changes. The in-memory type (UpdatedFragmentOffsets) is unchanged; only
the proto wire encoding changes.

Independent of #6748 and lance-spark #528 (JNI wiring). No mutual merge dependencies.

github-actions · 2026-06-24T01:22:26Z

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

Start a vote following the Lance community voting process.
Format specification modifications need 3 binding +1 votes (excluding the
proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
Once the vote passes, link the completed vote in this PR. It should not be
merged until the vote is linked.

codecov · 2026-06-24T02:02:51Z

Codecov Report

❌ Patch coverage is 93.84615% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/transaction.rs	93.84%	6 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

jerryjch · 2026-06-24T04:58:49Z

Hi, @wjones127 @pengw0048 I created a PR for #7080. Can you check if this is a good approach for the issue?

pengw0048 · 2026-06-29T13:44:45Z

Thanks, @jerryjch . I'm not very familiar with the Lance internals and it would be best if a maintainer can review this PR.

wjones127

I have one question. Once that's addressed them I'm happy to open a PMC vote on this change.

wjones127 · 2026-07-02T17:18:24Z

    // Per-fragment physical row offsets that matched an update_columns hash join (RewriteColumns).
+    // Deprecated: use updated_fragment_offset_bitmaps (field 10) instead.
    map<uint64, UInt32List> updated_fragment_offsets = 9;
+    // Per-fragment matched offsets as portable RoaringBitmap bytes (replaces field 9).
+    // Writers emit field 10 only. Readers prefer field 10; fall back to field 9 for
+    // manifests written before this change.
+    map<uint64, bytes> updated_fragment_offset_bitmaps = 10;


question(blocking): if an older reader sees just field 10, it will ignore it. What are the consequences of that?

@wjones127 Thanks for the review.
If an old reader deserializes a commit that only has field 10, it will not populate updated_fragment_offsets (field 9 is empty and field 10 is unknown to the old generated proto). That does not affect table data or OCC. Each writer—old or new—applies in-memory offsets in build_manifest at commit time and updates fragment.last_updated_at_version_meta in the manifest, which is what drives _row_last_updated_at_version at scan time. Both old and new readers read that from fragment metadata. The conflict resolver does not use updated_fragment_offsets. Field 9/10 is only the on-disk protobuf txn encoding.

The only consequence is on old Lance reading the stored txn (e.g. readTransaction()): offsets won’t round-trip for audit/history. Committed dataset versions remain correct. We chose single-write to field 10 to avoid persisting the large UInt32List on new commits. New readers can fall back to field 9 for txns written by older Lance. Old binaries need an upgrade to decode field 10 from txn bytes. This is pretty normal for the kind of proto migration.

dshepelev15 · 2026-07-07T15:28:35Z

Hello @jerryjch, do you have any progress in the pr?

jerryjch · 2026-07-08T17:55:53Z

@dshepelev15, The PR is pending review and approval. cc @wjones127

wjones127 · 2026-07-09T18:06:52Z

Thanks, I've opened a PMC vote thread here: #7705

That will close in one week, at which point we can merge this PR.

pengw0048 · 2026-07-17T14:03:03Z

Hi @wjones127 , are we good with the vote?

wjones127 · 2026-07-20T20:33:54Z

Hi @wjones127 , are we good with the vote?

Hi @pengw0048 I'm still trying to get votes. We need 3 +1 votes. We currently have zero. I'm asking PMC members to weigh in.

jiaoew1991 · 2026-08-06T16:51:44Z

Hi @wjones127 , #7705 has enough votes 😄

Xuanwo

Let's move!

vote has been passed.

`main` does not compile its tests. `cargo test -p lance --lib` stops at three uses of a field that no longer exists: error[E0559]: variant `dataset::transaction::Operation::Update` has no field named `merged_generations` --> rust/lance/src/dataset/transaction.rs:5453:17 | 5453 | merged_generations: vec![], | ^^^^^^^^^^^^^^^^^^ `dataset::transaction::Operation::Update` does not have this field | = note: available fields are: `compacted_sstables` #7957 renamed that field to `compacted_sstables` in `Operation::Update` and in `pb::transaction::Update`. #7432 was branched before it and its tests still name the old field, so the merge produced code that no compiler had seen: neither branch was wrong on its own base, and git had no textual conflict to report. All three sites pass an empty list, so they take the new name unchanged. With it, `cargo test -p lance --lib dataset::transaction::tests` is back to 64 passed. Co-authored-by: Vova Kolmakov <wombatukun@apache.org>

… binding (#8447) ## Summary `Operation::Update.updated_fragment_offsets` (#6650, #7432) is not reachable from Python: the Rust-to-Python export drops the field, and the Python-to-Rust conversion hardcodes `None`. This PR wires the field through the Python bindings as `dict[int, bytes]`. The bytes are the portable RoaringBitmap serialization, the same encoding as proto field 10 (#7432). This keeps dense offset sets compact and avoids materializing one Python int per matched row. A Python RoaringBitmap wrapper type (#7695) can replace the raw bytes later. ## Changes ### python/python/lance/dataset.py - Add `updated_fragment_offsets: Optional[Dict[int, bytes]] = None` to `LanceOperation.Update`, with attribute documentation. ### python/src/transaction.rs - Python to Rust: extract the dict and deserialize each value with `RoaringBitmap::deserialize_from`. Invalid bytes raise `ValueError`. A missing attribute (objects predating the field) extracts as `None`. - Rust to Python: serialize each bitmap with `RoaringBitmap::serialize_into` and pass the resulting `dict[int, bytes]` to the dataclass. ## Test plan - `test_update_with_commit_updated_fragment_offsets`: commit an `Update` carrying offsets for two fragments, read it back with `read_transaction`, and verify a byte-identical round trip through proto field 10. - `test_update_with_commit_rejects_invalid_offset_bytes`: a commit with invalid bitmap bytes fails with `ValueError`. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

perf: serialize updated_fragment_offsets as RoaringBitmap bytes in proto

3ab1c20

github-actions Bot added A-format On-disk format: protos and format spec docs performance labels Jun 24, 2026

wjones127 self-requested a review July 1, 2026 22:46

wjones127 previously requested changes Jul 2, 2026

View reviewed changes

Merge branch 'main' into fix-for-issue-7080

dc2316b

jerryjch requested a review from wjones127 July 8, 2026 17:56

Xuanwo approved these changes Aug 10, 2026

View reviewed changes

Xuanwo merged commit 2a10333 into lance-format:main Aug 10, 2026
31 checks passed

wombatu-kun mentioned this pull request Aug 10, 2026

fix: compile the lance tests after the compacted_sstables rename #8440

Merged

pengw0048 mentioned this pull request Aug 10, 2026

feat(python): expose updated_fragment_offsets in the Update operation binding #8447

Merged

Uh oh!

Conversation

jerryjch commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

Changes

protos/transaction.proto

rust/lance/src/dataset/transaction.rs

Test plan

Backward compatibility

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

codecov Bot commented Jun 24, 2026

Codecov Report

Uh oh!

jerryjch commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pengw0048 commented Jun 29, 2026

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

wjones127 Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

jerryjch Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dshepelev15 commented Jul 7, 2026

Uh oh!

jerryjch commented Jul 8, 2026

Uh oh!

wjones127 commented Jul 9, 2026

Uh oh!

pengw0048 commented Jul 17, 2026

Uh oh!

wjones127 commented Jul 20, 2026

Uh oh!

jiaoew1991 commented Aug 6, 2026

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jerryjch commented Jun 24, 2026 •

edited

Loading

jerryjch commented Jun 24, 2026 •

edited

Loading

jerryjch Jul 2, 2026 •

edited

Loading