Conversation
zjw1111
marked this pull request as ready for review
September 22, 2026 10:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
V2 file-index containers now support table writes and index rewrites when an index payload or the container exceeds 2 GiB. Previously, both production paths assembled the complete container in a
ByteArrayOutputStream, so selecting V2 did not remove that size limit.The container writer accepts payloads through
writeIndexandFileIndexWriter.writeTo(OutputStream).DataFileIndexWriterandFileIndexProcessorwrite through a spillable output stream: small containers stay embedded in file metadata, while larger ones become independent index files. V2 records 64-bit payload positions and lengths in its footer; readers continue to accept V1 and V2. The plugin reader interface now takes alongpayload length. Existing built-in index writers still serialize individual payloads as byte arrays, and V1 keeps its header-first, in-memory write behavior. V1 remains the default format.Tests
FileIndexFormatFormatTest:testV2TotalSizeOverTwoGiBAndLegacyBackendAtLargePosition,testV2StreamedPayloadOverInt32, andtestV2CopyPayloadOverInt32cover large positions, a single large streamed payload, and copying that payload without a large byte array.testIncrementalWriteAndCopyPayloadcovers V1/V2 incremental writes, andtestV2RejectInvalidTailMagiccovers a damaged trailer.DataFileIndexWriterTest:testSpillableIndexOutputStreamcovers the embed/spill boundary;testV2TableWriterStreamsPayloadOverTwoGiBwrites a table index above 2 GiB;testV2FailedWriteDeletesPartialIndexFilechecks cleanup after a failed write.FileIndexProcessorTest:testV2RewritesContainerOverTwoGiBrewrites a table index above 2 GiB. The map-key rewrite test also runs with V1 and V2.CoreOptionsTest.testFileIndexInManifestThresholdcovers threshold bounds, andFileIndexPredicateCloseTest.testFileStatusFailureDoesNotOpenStreamcovers reader construction failure.FileIndexesTableTestruns embedded and external index cases with both V1 and V2.