feat(java): expose aggregate fragment summary

[everySympathy]

Summary

• aggregate fragment count and min/max live row counts in one native pass over the loaded manifest
• aggregate min/max data-file counts per fragment
• return a fixed-size typed Java result instead of materializing per-fragment Java objects
• define zero-valued min/max fields for empty datasets

Live row counts match the existing Java fragment-statistics semantics: physical rows minus deleted rows.

Use case

Aggregate-only consumers such as dataset profiling, health checks, and optimization admission need fragment count and min/max values, but do not consume per-fragment IDs or arrays. For these callers, getFragmentSummary() keeps the result size constant as the manifest grows.

100k-fragment benchmark

Reproduce with:

cargo bench -p lance --bench fragment_summary -- --sample-size 10

Measured on this branch:

Path	Median
native aggregate summary	770.73 µs
materialize flattened per-fragment statistics	812.23 µs

Both paths must scan the same 100,000 manifest entries, so native traversal is only about 5% faster. The main benefit is allocation and JNI transfer size.

Allocation and Java/JNI validation

I also ran local measurement instrumentation against a real 100,000-fragment dataset. The probe used the public Java APIs through JNI, with 20 warm-up calls followed by seven alternating rounds of 20 calls. The table reports the median of three complete probe runs:

Public Java API	Current-thread Java heap allocation/op	End-to-end median/op
getFragmentStatistics()	4,000,104 bytes	20.94 ms
getFragmentSummary()	72 bytes	2.16 ms

A separate native allocator probe measured 2,400,000 bytes/op for the flattened Vec<i64> and 0 bytes/op for the aggregate result. The Java allocation measurement uses ThreadMXBean.getThreadAllocatedBytes, so it deliberately excludes native allocations; the two measurements validate different sides of the JNI boundary.

On this fixture, the new API therefore removes the fragment-count-scaled native and Java allocations and is about 9.7x faster end to end. The temporary allocation probes are not part of this PR; the committed Criterion benchmark and API tests remain small and maintainable.

The committed Criterion benchmark intentionally measures only native work, so it does not count the additional JNI copy and Java array-splitting time.

Testing

• cd java && ./mvnw test -Dtest=FragmentTest
• cargo check -p lance --bench fragment_summary
• cargo clippy -p lance --bench fragment_summary -- -D warnings
• cargo bench -p lance --bench fragment_summary -- --sample-size 10
• cargo clippy -p lance --tests --benches -- -D warnings
• cd java && cargo clippy --tests --manifest-path lance-jni/Cargo.toml -- -D warnings
• cd java && ./mvnw spotless:check

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The fixed-size native aggregation is the right boundary: it keeps the Java binding thin and removes O(fragment-count) JNI payload and allocation when callers need only extrema. The non-blocking risk is evidence debt: the PR does not identify an aggregate-only production consumer or benchmark the new path, and coverage does not exercise unequal data-file counts or deletion metadata. A focused scale benchmark plus those edge cases would clarify the adoption benefit and protect the contract.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The fixed-size native aggregation remains the right boundary, but the new revision makes missing row metadata from frozen legacy manifests part of a new public Rust/Java contract. Keep this API scoped to current-format metadata: return a descriptive error when a fragment lacks a physical or deletion row count instead of exposing partial optional extrema and an unknown-count field.

[lance-gatekeeper]

Missing row metadata is legacy-only (Fragment::physical_rows documents that all new tables set it), so these Option extrema and the paired unknown-count field permanently extend a frozen legacy surface into a new API. That conflicts with the repository contract to preserve legacy reads without using legacy paths for new feature design, and it leaves callers with a partial-result mode current writers never produce. Keep row extrema required for this metadata-only summary and make fragment_summary return a descriptive error naming the fragment when num_rows() is unavailable; JNI can propagate that exception. The historical fixture should assert this error rather than define legacy success.

[lance-gatekeeper]

Fixed in 226857c: fragment_summary now returns a Result, reports missing row metadata with the fragment ID, JNI propagates the exception, and the historical fixture asserts rejection. The optional extrema and unknown-count fields are removed.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The legacy-contract blocker is fixed: missing row metadata now fails with a fragment-specific error through Rust, JNI, and Java, and the historical fixture covers the boundary. The fixed-size native aggregation now has the right current-format contract.

The remaining non-blocking risk is adoption evidence: the PR still does not identify an aggregate-only production consumer or benchmark the scale benefit. A focused high-fragment-count benchmark would validate the intended JNI and allocation reduction.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The added 100k-fragment benchmark closes the remaining adoption-evidence gap: fixed-size native aggregation is faster than materializing the flattened per-fragment payload while avoiding allocation and JNI transfer that scale with fragment count. The current-format metadata contract and its empty, deletion, unequal-file-count, and legacy-error coverage remain intact.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!