parquet: bounded skip chunk in DeltaBitPackDecoder::skip - #2
Merged
Vedin merged 1 commit intoSep 12, 2026
Merged
Conversation
…page header 3007f88 sized the skip buffer from `values_per_mini_block` so that mini-block sizes other than 32/64 (as written by Snowflake) could be skipped. That value is derived from the page header (`block_size / mini_blocks_per_block`, both unbounded ULEB128, only checked for divisibility), so a malformed file could make `skip` allocate an arbitrarily large Vec and abort the process (`memory allocation of 8796093022208 bytes failed`). Replace the header-sized Vec with a fixed 64-value stack array (`SKIP_CHUNK`) and decode each `bit_width > 0` mini-block through it in chunks, updating `last_value` exactly as before. The bit reader is positional, so chunking consumes the same bits; the `bit_width == 0` fast path is unchanged. Any mini-block size that `set_data` accepts is now handled with no allocation at all. Tests (parquet/src/encodings/decoding.rs): - a test-only DELTA_BINARY_PACKED encoder with configurable block_size / mini_blocks_per_block (the built-in encoder always writes 128/4), sanity-checked against the built-in layout; - Snowflake-like layouts 256/2 (128 per mini-block), 512/1 and 256/1 (i32): full decode in uneven `get` calls, `skip(n)` + tail `get` for n inside a mini-block, across mini-block and block boundaries, larger than a mini-block/block, and past the end, plus consecutive partial skips; - an oversized header (block_size = 2^40, one mini-block, total_count = 1000, one payload byte): `set_data` accepts it, `skip(10)` and `get` return `Err` promptly instead of allocating. With the previous code this test aborts with an 8 TiB allocation failure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012m5Yx6yEpZsacqAhZotgkT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
DeltaBitPackDecoder::skipno longer sizes its buffer fromvalues_per_mini_block(an unbounded ULEB128 page-header field, only divisibility-checked): a malformed file could force an arbitrarily large allocation and abort the process.It now decodes each mini-block through a fixed
SKIP_CHUNK = 64stack array inmin(remaining, 64)pieces, updatinglast_valueexactly as before; thebit_width == 0fast path is untouched. Any mini-block size accepted byset_data(Snowflake-written files use sizes other than 32/64) is handled with zero heap allocation; for 32/64 the behaviour is bit-identical.Tests
block_size/mini_blocks_per_block(sanity-checked against the built-in 128/4 layout): 256/2, 512/1, 256/1 — full decode,skip(n)+ tailgetfor n = 1, 5, 100, 129, 300, block/mini-block boundaries ±1, past the end, consecutive partial skips.block_size = 2^40,total_count = 1000, one payload byte):skip(10)andgetreturnErrpromptly. With the previous code this test aborts with an 8 TiB allocation failure.cargo test -p parquet --lib encodings::decoding: 85 passed; fmt and clippy clean.Pinned by Embucket/rustice (
[patch.crates-io], whole arrow 59.2.0 workspace at this rev).🤖 Generated with Claude Code
https://claude.ai/code/session_012m5Yx6yEpZsacqAhZotgkT