fix(java): expose updatedFragmentOffsets on Update operation for RewriteColumns - #6748
fix(java): expose updatedFragmentOffsets on Update operation for RewriteColumns#6748jerryjch wants to merge 17 commits into
Conversation
c116d01 to
456e6dc
Compare
ed07fea to
cbf1475
Compare
…iteColumns commits
cbf1475 to
1884a7d
Compare
| let mut iter = jmap.iter(env)?; | ||
| let mut offsets: HashMap<u64, RoaringBitmap> = HashMap::new(); | ||
| env.with_local_frame(32, |env| { | ||
| while let Some((key, value)) = iter.next(env)? { |
There was a problem hiding this comment.
The Java-to-Rust offset map import keeps all iterator-created JNI local references alive for the whole map. Large multi-fragment updates can exhaust the local reference table and fail the commit before Rust validation runs.
| let frag_id = | ||
| env.call_method(&key, "longValue", "()J", &[])?.j()? as u64; | ||
| let buf: Vec<u8> = env.convert_byte_array(JByteArray::from(value))?; | ||
| let bitmap = RoaringBitmap::deserialize_from(buf.as_slice())?; |
There was a problem hiding this comment.
Malformed bitmap bytes are reported as I/O failures even though they come from Java API input. Callers can misclassify bad arguments as retryable storage errors.
…UpdateResult Serialize matched_offsets once at the executor JNI boundary instead of expanding to long[]. Add getUpdatedRowOffsetBytes(); deprecate getUpdatedRowOffsets() with expandRowOffsetsFromBytes for lance-format#6650 compat. Follow-up to lance-format#6650. Pairs with Update.updatedFragmentOffsets() for lance-spark#418. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
@Xuanwo Thanks for the review. I addressed your comments in commit |
| * with callers compiled against the #6650 API. | ||
| */ | ||
| @Deprecated | ||
| public FragmentUpdateResult( |
There was a problem hiding this comment.
The retained long[] compatibility path now calls native helpers from a class that never loads the JNI library. Existing callers can hit UnsatisfiedLinkError unless another Lance class happened to initialize first.
| env.call_method(&key, "longValue", "()J", &[])?.j()? as u64; | ||
| let buf: Vec<u8> = | ||
| env.convert_byte_array(JByteArray::from(value))?; | ||
| let bitmap = RoaringBitmap::deserialize_from(buf.as_slice()) |
There was a problem hiding this comment.
The commit path accepts arbitrary offset bitmaps and later expands them into a full offset vector before checking fragment bounds. A compact valid Roaring bitmap can force huge allocations during a Java RewriteColumns commit.
| this(updatedFragment, updatedFieldIds, new byte[0]); | ||
| } | ||
|
|
||
| public FragmentUpdateResult( |
There was a problem hiding this comment.
Adding a public byte[] constructor with the same arity as the retained long[] constructor makes existing source calls that pass null for offsets fail overload resolution.
| "updatedRowOffsets must be non-negative, got {offset}" | ||
| ))); | ||
| } | ||
| bitmap.insert(offset as u32); |
There was a problem hiding this comment.
The deprecated long[] encoder casts every non-negative value to u32, so offsets above u32::MAX silently become different rows and can corrupt last_updated metadata.
| Update.builder() | ||
| .removedFragmentIds(Collections.singletonList(fragmentId)) | ||
| .newFragments(Collections.singletonList(newFragment)) | ||
| .updateMode(Optional.of(UpdateMode.RewriteRows)) |
There was a problem hiding this comment.
The round-trip test uses RewriteRows, but updatedFragmentOffsets is consumed only by the RewriteColumns stable-row-id branch. The Java/Spark path can regress while this test still passes.
The perf(java) commit picked up an incidental Cargo.lock change adding "rayon" when cargo ran during the pre-commit hook after the main branch merge. Revert to the pre-commit state. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add static { JniLoader.ensureLoaded(); } so deprecated native methods
do not throw UnsatisfiedLinkError when no other Lance class has been
touched first.
- Add public static create() factory as the primary bytes API; make the
byte[] constructor private to eliminate null-argument overload ambiguity
with the deprecated long[] constructor.
- Add upper bound check (offset > u32::MAX) in the deprecated
encodeRowOffsetsToBytes JNI helper to prevent silent u32 truncation.
- Update FragmentUpdateResultTest to use FragmentUpdateResult.create().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
updatedFragmentOffsets is consumed only by the RewriteColumns path in build_manifest. The test was using RewriteRows, which never exercises the offset round-trip. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A compact RLE RoaringBitmap (as few as 22 bytes) can represent all u32 values. The .collect() expansion at build_manifest would allocate ~32GB. Validate bitmap len() and max() against the fragment's physical_rows from existing_fragments (manifest truth) before expanding. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Add tests for cardinality exceeding physical_rows, max offset exceeding physical_rows, and exact-boundary success case. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Hi, @Xuanwo Could you review the PR again? I have addressed your comments. Thanks. |
|
Hi, @Xuanwo could you help review the PR again? |
…oto (#7432) Fixes: #7080 ## Summary Follow-up to #6650. The `updated_fragment_offsets` field (proto field 9) stores per-fragment matched row offsets as `map<uint64, UInt32List>` -- one uint32 per matched row. For dense rewrites this produces multi-GB manifests (e.g. 86k matched rows x 4 bytes x many fragments). This PR adds proto field 10 (`map<uint64, bytes>`) using portable RoaringBitmap serialization, which typically compresses the same data to tens of bytes per fragment. Writers emit field 10 only; readers prefer field 10, falling back to field 9 for manifests written before this change. ## Background PR #6650 added `updated_fragment_offsets` to the `Update` transaction message so that `build_manifest` can partially refresh `_row_last_updated_at_version` for matched rows only. The encoding choice -- one uint32 per offset in a `UInt32List` -- was flagged post-merge as a size regression for dense updates. The offsets are already stored internally as `RoaringBitmap`; this PR aligns the proto encoding with that representation. ## Changes ### protos/transaction.proto - Deprecate field 9 (`map<uint64, UInt32List> updated_fragment_offsets`) with a comment pointing to field 10. - Add field 10: `map<uint64, bytes> updated_fragment_offset_bitmaps` with documentation of the dual-read strategy. ### rust/lance/src/dataset/transaction.rs Serialization (`From<&Transaction> for pb::Transaction`): - Write field 10 only: `RoaringBitmap::serialize_into` produces portable bytes for each fragment's bitmap. - Set field 9 to an empty `HashMap` (forward compat; old readers ignore unknown fields). Deserialization (`TryFrom<pb::Transaction> for Transaction`): - If field 10 is non-empty: deserialize each entry with `RoaringBitmap::deserialize_from`. - Else if field 9 is non-empty: convert each `UInt32List` to `RoaringBitmap::from_iter` (legacy fallback). - Same `if !new_field.is_empty() { ... } else { ... }` pattern used by the existing `Rewrite.groups` / `Rewrite.old_fragments` migration. Invalid field 10 bytes fail deserialize with `Error::invalid_input`. In-memory type unchanged: `UpdatedFragmentOffsets(HashMap<u64, RoaringBitmap>)`. ## Test plan - `test_proto_round_trip_field_10` -- write a transaction with field 10, read back, verify offsets match for two fragments. - `test_proto_legacy_field_9_read` -- construct a proto with only field 9 populated (simulating an old writer), deserialize, verify offsets are correctly recovered. - `test_proto_field_10_takes_precedence_over_field_9` -- when both fields are present, field 10 values are used and field 9 is ignored. Proto wire format change; team vote may be needed. ## Backward compatibility - Proto field numbers: field 9 is kept (deprecated, not removed). Field 10 is new. No field number reuse. - Old readers: ignore unknown field 10; they only read field 9, which is now empty on new commits. Old Lance versions deserializing commits written by this PR will not recover offsets from the txn; that only affects audit/`readTransaction()` on historical commits, not table data or OCC. - New readers: prefer field 10; fall back to field 9 for manifests written by older Lance versions that predate this change. - No JNI or Java changes. The in-memory type (`UpdatedFragmentOffsets`) is unchanged; only the proto wire encoding changes. Independent of #6748 and lance-spark #528 (JNI wiring). No mutual merge dependencies. Co-authored-by: Jing chen He <jingh@adobe.com>
DataFile::new no longer takes separate major/minor args — use ConcreteFileVersion::from(LanceFileVersion::Stable). RowIdMeta::Inline now wraps InlineRowIds, requiring .into() on write_row_ids output. Fragment gained an overlays field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
The offset map must be constrained to fragments actually rewritten by the operation. Otherwise a mismatched fragment ID can refresh version metadata on an unchanged fragment while leaving the rewritten fragment stale, producing incorrect change-data results.
Require every non-empty updatedFragmentOffsets key to match an updatedFragments entry, and cover the invariant with a stable-row-ID, multi-fragment commit test.
| match entry { | ||
| None => break, | ||
| Some((frag_id, bitmap)) => { | ||
| offsets.insert(frag_id, bitmap); |
There was a problem hiding this comment.
Any non-empty updatedFragmentOffsets entry is accepted here without checking that frag_id belongs to updated_fragments. In a stable-row-ID RewriteColumns transaction, build_manifest later applies every supplied key to whichever existing fragment has that ID. A caller that rewrites fragment A but keys the bitmap to existing fragment B leaves A metadata stale and stamps unchanged B, so change-data reads gain false positives and miss real changes.
Reject keys not present in updated_fragments before constructing the operation, and enforce the same invariant in core transaction validation.
Reproducer run at this head: JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/home/agent/tmp/lance-gate-6748 ./mvnw -Dtest=org.lance.operation.UpdateTest#testUpdatedFragmentOffsetsRoundTrip test passes. That test constructs RewriteColumns with no updatedFragments, removes fragment A, and still supplies offsets keyed to A; the commit succeeds, whereas this transaction should be rejected. Add a stable-row-ID, two-fragment variant that verifies a wrong key cannot update unrelated version metadata.
There was a problem hiding this comment.
Fixed in fbd32c2. Core validation now rejects offset keys absent from updated_fragments, build_manifest has a defense-in-depth guard, and the two-fragment regression plus corrected Java round-trip pass locally. This closes the stale or misattributed metadata path.
A key in updatedFragmentOffsets that does not match any entry in updated_fragments could stamp version metadata on an unrelated fragment, causing false positives in change-data reads and missing real changes. Fixes: - validate_operation: reject any updatedFragmentOffsets key absent from updated_fragments before the commit proceeds - build_manifest: defense-in-depth guard skips stamping fragments not in updated_by_id even if a stray key reaches the loop - testUpdatedFragmentOffsetsRoundTrip: fix the test to supply offsets keyed to a fragment that is in updatedFragments - Add test_updated_fragment_offsets_key_not_in_updated_fragments_is_rejected: two-fragment scenario verifying the error is returned Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The prior fragment-key mismatch is fixed: core validation now rejects offset keys outside the rewritten fragment set, and manifest construction retains a defense-in-depth guard.
The portable Roaring representation is preferable to expanding offsets across JNI because it preserves exact stable-row-ID metadata semantics while keeping transport proportional to the compressed bitmap.
The key-presence check added in the previous commit was applied to all Operation::Update variants. rewrite_rows mode legitimately supplies offsets for fragments outside updated_fragments, causing test_dataset.py to fail. Gate the check on update_mode == RewriteColumns, matching the build_manifest guard that actually uses the offsets for version metadata stamping. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tion tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The fragment-key invariant is enforced at the correct boundary: RewriteColumns offsets can update metadata only for fragments in the rewrite, while other update modes can round-trip the serialized map without affecting manifest metadata.
Keeping the portable Roaring bitmap on the transaction is preferable to expanding offsets across JNI because it preserves exact stable-row-ID semantics and keeps transport proportional to compressed data.
Summary
RewriteColumns).
Update.java: addsMap<Long, byte[]> updatedFragmentOffsetsfield, 7-arg constructor,accessor
updatedFragmentOffsets(), andBuilder.updatedFragmentOffsets(...)setter.Defaults to
Collections.emptyMap(). Values are portable RoaringBitmap bytes.java/lance-jni/src/transaction.rs— two JNI directions updated: FromJava deserializeseach
byte[]value into aRoaringBitmapand setsupdated_fragment_offsetson the Rustoperation; IntoJava serializes each bitmap to
byte[]and populates aHashMap<Long, byte[]>passed to the 7-arg
Updateconstructor (previously the field was ignored and the 6-argform was used).
Update.javaequals/hashCode: deep-comparesbyte[]values by content;hashCodeadded per the Java contract.
Background
PR #6650 added
updated_fragment_offsetson the RustOperation::Update(proto field 9),build_manifestpartial refresh logic, andFragmentUpdateResult.getUpdatedRowOffsets().Two gaps remained:
The Java
Updateclass had no field for these offsets andconvert_to_rust_operationalways set
updated_fragment_offsets: None, so the lance-spark commit path(UpdateColumnsBackfillBatchWrite) had no way to pass offsets to Rust and the partial
refresh in
build_manifestcould never activate from a JVM caller.convert_to_java_operation_innerstill used the old 6-arg constructor signature fornew_object. With the 6-arg constructor removed fromUpdate.java(replaced by the7-arg form), any Rust→Java materialization of
Operation::Update(e.g. reading back atransaction) would fail at runtime with
NoSuchMethodError.Implementation notes
stays O(bitmap size) rather than O(n matched rows).
with_local_frame(4, ..)per bitmap entry in IntoJava bounds local-ref growth on largeoffset maps.
JMapwas avoided inside the frame because it holds aJObjectwith theouter frame's lifetime, causing borrow-checker conflicts;
call_methodon the outerjava_mapreference is used instead.Vec<u8>buffer for each bitmap is allocated in Rust before entering the frame, soits lifetime is independent of JNI frame scope.
with_local_frame(8, ..)per iteration in FromJava bounds local-ref growth for largemulti-fragment maps.
build_manifestvalidates bitmap cardinality and max offset against the fragment'sphysical_rowsfromexisting_fragmentsbefore.collect(), preventing a compact RLEbitmap from expanding into an unbounded allocation.
UpdatedFragmentOffsetsadded to thelance::dataset::transactionimport.Why the protobuf field alone is not enough
lance-spark commits by calling
CommitBuilder.execute(transaction), which passes the JavaTransactionobject tonativeCommitToDatasetvia JNI. The JNI handler callsconvert_to_rust_transaction→convert_to_rust_operation, which reflects on the JavaUpdateobject to build the RustOperation::Updatestruct. The protobuf field (field 9)is only used when a Transaction is serialized as a proto blob; it has no effect on the
reflection-based JNI path unless the Java
Updateclass exposes the field and the JNIdeserialization reads it.
Additional change
FragmentUpdateResult(from #6650) returned matched row offsets as an expandedlong[]atthe executor JNI boundary. This PR also passes those offsets as portable RoaringBitmap bytes
so lance-spark can wire them through to
Update.updatedFragmentOffsets()without an O(n rows)expansion on the executor→driver path.
FragmentUpdateResult.getUpdatedRowOffsetBytes()— primary accessor; values are the sameportable RoaringBitmap byte format as
Update.updatedFragmentOffsets().java/lance-jni/src/fragment.rs—update_columns_with_offsetsserializesmatched_offsetsonce withRoaringBitmap::serialize_into; JNI constructs results viathe private
(FragmentMetadata, long[], byte[])constructor (JNI can access private ctors).FragmentUpdateResult.create(FragmentMetadata, long[], byte[])— public static factory;primary construction path for callers using the bytes API.
@Deprecated getUpdatedRowOffsets()— retained for backward compatibility; expands bytes viaexpandRowOffsetsFromBytesonly when called (lazy O(n rows)).@Deprecated3-arg(FragmentMetadata, long[] fieldsModified, long[] updatedRowOffsets)constructor — encodes offsets via
encodeRowOffsetsToBytesfor source compat.FragmentUpdateResultTest— round-trip bytes, deprecated constructor encode, andupdateColumns()integration asserting matched offsets{0,1,2,3}on the test fixture.Test plan
UpdateTest#testUpdatedFragmentOffsetsRoundTrip— commits anUpdatewith a non-emptyupdatedFragmentOffsetsmap throughCommitBuilder.execute(exercises the FromJava JNIpath), reads the transaction back via
Dataset.readTransaction()(exercises the IntoJavaJNI path), and asserts the offsets match. Map value is hardcoded portable RoaringBitmap
bytes encoding {1, 3, 5}; verified with
assertArrayEqualsafter the round-trip.FragmentUpdateResultTest— see Additional change.Compatibility
Update— theupdatedFragmentOffsetsfield did not exist in anyprior release. The builder setter is optional and defaults to
Collections.emptyMap(), soexisting
Update.builder()...build()call sites compile and behave identically.equals/hashCode:equalsusesoffsetMapsEqualto deep-comparebyte[]values via
Arrays.equals;hashCodeis added per the Java contract.new_objectcall is updated from the 6-arg tothe 7-arg form in the same PR. Both files must ship together; within that atomic change
there is no compatibility gap.
updated_fragment_offsetsproto field and Rust structfield were already added in fix: propagate update_columns offsets and partial last_updated for RewriteColumns #6650.
FragmentUpdateResult—@Deprecatedlong[] getter and constructor retained; new bytes getter is the supported path for new callers(see Additional change).