Skip to content

[file-index] Support stream file index containers - #10101

Open
zjw1111 wants to merge 5 commits into
apache:masterfrom
zjw1111:codex/file-index-v2-format
Open

zjw1111 wants to merge 5 commits into
apache:masterfrom
zjw1111:codex/file-index-v2-format

Conversation

@zjw1111

@zjw1111 zjw1111 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Purpose

V2 file-index containers now support table writes and index rewrites when an index payload or the container exceeds 2 GiB. Previously, both production paths assembled the complete container in a ByteArrayOutputStream, so selecting V2 did not remove that size limit.

The container writer accepts payloads through writeIndex and FileIndexWriter.writeTo(OutputStream). DataFileIndexWriter and FileIndexProcessor write through a spillable output stream: small containers stay embedded in file metadata, while larger ones become independent index files. V2 records 64-bit payload positions and lengths in its footer; readers continue to accept V1 and V2. The plugin reader interface now takes a long payload length. Existing built-in index writers still serialize individual payloads as byte arrays, and V1 keeps its header-first, in-memory write behavior. V1 remains the default format.

Tests

  • FileIndexFormatFormatTest: testV2TotalSizeOverTwoGiBAndLegacyBackendAtLargePosition, testV2StreamedPayloadOverInt32, and testV2CopyPayloadOverInt32 cover large positions, a single large streamed payload, and copying that payload without a large byte array. testIncrementalWriteAndCopyPayload covers V1/V2 incremental writes, and testV2RejectInvalidTailMagic covers a damaged trailer.
  • DataFileIndexWriterTest: testSpillableIndexOutputStream covers the embed/spill boundary; testV2TableWriterStreamsPayloadOverTwoGiB writes a table index above 2 GiB; testV2FailedWriteDeletesPartialIndexFile checks cleanup after a failed write.
  • FileIndexProcessorTest: testV2RewritesContainerOverTwoGiB rewrites a table index above 2 GiB. The map-key rewrite test also runs with V1 and V2.
  • CoreOptionsTest.testFileIndexInManifestThreshold covers threshold bounds, and FileIndexPredicateCloseTest.testFileStatusFailureDoesNotOpenStream covers reader construction failure.
  • FileIndexesTableTest runs embedded and external index cases with both V1 and V2.

@zjw1111
zjw1111 marked this pull request as ready for review September 22, 2026 10:42
@zjw1111 zjw1111 changed the title [file-index] Support V2 file indexes over 2 GiB [file-index] Stream V2 index containers beyond 2 GiB Sep 22, 2026
@zjw1111 zjw1111 changed the title [file-index] Stream V2 index containers beyond 2 GiB [file-index] Stream V2 index containers Sep 22, 2026
@zjw1111 zjw1111 changed the title [file-index] Stream V2 index containers [file-index] Support stream file index containers Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant