Skip to content

feat(python): expose updated_fragment_offsets in the Update operation binding - #8447

Merged
Xuanwo merged 1 commit into
lance-format:mainfrom
pengw0048:python-updated-fragment-offsets
Aug 10, 2026
Merged

feat(python): expose updated_fragment_offsets in the Update operation binding#8447
Xuanwo merged 1 commit into
lance-format:mainfrom
pengw0048:python-updated-fragment-offsets

Conversation

@pengw0048

Copy link
Copy Markdown
Contributor

Summary

Operation::Update.updated_fragment_offsets (#6650, #7432) is not reachable from Python: the Rust-to-Python export drops the field, and the Python-to-Rust conversion hardcodes None. This PR wires the field through the Python bindings as dict[int, bytes].

The bytes are the portable RoaringBitmap serialization, the same encoding as proto field 10 (#7432). This keeps dense offset sets compact and avoids materializing one Python int per matched row. A Python RoaringBitmap wrapper type (#7695) can replace the raw bytes later.

Changes

python/python/lance/dataset.py

  • Add updated_fragment_offsets: Optional[Dict[int, bytes]] = None to LanceOperation.Update, with attribute documentation.

python/src/transaction.rs

  • Python to Rust: extract the dict and deserialize each value with RoaringBitmap::deserialize_from. Invalid bytes raise ValueError. A missing attribute (objects predating the field) extracts as None.
  • Rust to Python: serialize each bitmap with RoaringBitmap::serialize_into and pass the resulting dict[int, bytes] to the dataclass.

Test plan

  • test_update_with_commit_updated_fragment_offsets: commit an Update carrying offsets for two fragments, read it back with read_transaction, and verify a byte-identical round trip through proto field 10.
  • test_update_with_commit_rejects_invalid_offset_bytes: a commit with invalid bitmap bytes fails with ValueError.

… binding

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added A-python Python bindings enhancement New feature or request labels Aug 10, 2026
@pengw0048
pengw0048 marked this pull request as ready for review August 10, 2026 13:57

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

This closes the Python transaction-binding gap at the existing wire boundary: populated offset bitmaps round-trip as compact portable bytes, absent legacy fields remain None, and malformed bytes fail with fragment context. Keeping raw bytes is preferable to expanding dense bitmaps into Python integers while the dedicated bitmap wrapper remains separate.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 10, 2026

@Xuanwo Xuanwo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@Xuanwo
Xuanwo merged commit 890c8bc into lance-format:main Aug 10, 2026
18 checks passed
@jerryjch

Copy link
Copy Markdown
Contributor

@Xuanwo @pengw0048 I have a PR on the Java JNI side, can you please help review #6748?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants