datafusion_iceberg: re-port the adaptive page-index cache and parquet metadata size hint onto DF55 - #74
Merged
Conversation
…t cache behaviour Two gaps in the parquet metadata cache, found by measuring the deployed service across five benchmark tiers. 1. No accounting. A cold cache and one too small for the working set look identical from outside -- both are just slow metadata loading. Hits, misses and evictions are now plan metrics on the scan node, so `EXPLAIN ANALYZE VERBOSE` reports them against any deployed binary with no wire change. `evictions` counts entries pushed out by a scan's own inserts, which is what separates the two cases. 2. The page index was never cached. DataFusion's opener loads the footer with `PageIndexPolicy::Skip`, so the cached entry is footer-only; its own `load_page_index` then rebuilds an enriched `ParquetMetaData` and drops it. Every query therefore re-fetched the page index of every file it touched. Populating the cache with it takes SF100 Q8's warm `metadata_load_time` from 15.1s to 0.2s (75x) at an unchanged 100% footer hit rate -- that residual was entirely this. The trade is scale-dependent, which is why it is guarded rather than unconditional. Where the working set fits, end to end: SF1 -29%, SF10 -15%, ClickBench -34%, SF100 -18% of server time. Where it does not, it inverts -- at TPC-H SF1000 the cache churns, so the extra first-touch range read is paid on nearly every load and the warm payback never arrives. Attribution on Q10, one variable at a time: page index off 393s (baseline 383s) page index on, 64 MiB 511s (+33%) page index on, 2048 MiB >1860s (killed) Across that suite it took Q18 from 2078s to a 7200s timeout and Q21 +62%. Raising the cap does not rescue it; it worsens it, because gigabytes of cached metadata inside a fixed container compete with the query budget. So widening is on by default and latches off once cumulative evictions pass PAGE_INDEX_BACKOFF_EVICTIONS -- eviction being direct evidence that the working set does not fit. Verified end to end: at SF1000 every cold load is suppressed (34 misses, 34 suppressed, 404 evictions) and Q10 returns to 394s; on a fresh process at SF100 the guard never engages (0 evictions, 0 suppressed) and the 75x warm cut is fully retained. The latch is deliberately one-way. Backing off makes subsequent entries footer-sized, which stops the eviction that triggered it, so a recovering guard would re-enable widening, grow entries, evict again, and oscillate. One-way also fails safe: the worst case is losing an optimization for the life of the process, never reintroducing a timeout. The cost is that a process serving a very large workload and then a small one keeps the back-off; that is the intended direction. `parquet_metadata_cache_page_index_suppressed` reports when the guard has engaged, so "did it back off?" is answerable rather than inferred. 8 new tests: hit/miss/eviction accounting through `load`, eviction counting in `ByteCappedCache::insert`, the widening policy (inert when disabled, widens Skip and absent options, skips a caller that already asked), and the guard (a cache with room never latches, a churning cache latches and stays latched, a single eviction does not latch). 18/18 in the module. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…servability Iceberg scans never set metadata_size_hint (neither on ParquetSource nor on PartitionedFile), so the parquet reader prefetches only the 8-byte footer tail and every cold metadata load costs two sequential ranged GETs (tail probe, then footer body) plus a third for the page index. At TPC-H SF1000 that is 136.5 ms of metadata per open against 15 ms at SF100, 44% of the tier's partition-time. ICEBERG_PARQUET_METADATA_SIZE_HINT_KB (read once per process, default 0 = off, byte-identical to before) stamps a clamped per-file hint in generate_partitioned_file, so one suffix GET fetches footer + page index and the retained remainder serves the index ranges with no further fetch. The new size_hint_collapses_cold_metadata_load_to_one_get test locks the 1-GET behavior in. Also surface the cache working set - (entries, total_bytes, mean weight) logged every 4096 inserts and on the page-index back-off latch - so the cap can be sized per node from measurement instead of guesses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…delete files too The DF55 branch scans v2 position-delete files through the same parquet reader factory as data files; without the per-file hint their cold footer loads still take two or three sequential ranged reads.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
perf/metadata-size-hint(DF53 base) onto the DF55 sync branch, as faithful cherry-picks:6ee5588dadaptive page-index caching + cache-behaviour reportingd88f560bICEBERG_PARQUET_METADATA_SIZE_HINT_KBper-file footer prefetch hint + cache working-set observabilityPartitionedFileliteral indatafusion_iceberg/src/table/mod.rs(DF55 usesextensions: Default::default()); resolved by keeping DF55's field set plus the hintedmetadata_size_hint. DF55's parquet opener still prefers the per-file hint over the source-level one (datasource-parquet/src/opener/mod.rs), so the hint is live end to end.Why
The DF55 migration carried the July footer/manifest caches but not these two commits, so
ICEBERG_PARQUET_METADATA_SIZE_HINT_KB(512 in every measured SPCS configuration) became a silent no-op on rusticemain: every cold footer load went back to 2-3 sequential ranged GETs.Verification (local, rustfmt/clippy 1.95 as in CI)
cargo fmt --all -- --check(=make fmt): cleancargo clippy --all-targets --all-features -- -D warningson the workspace (=make clippy): cleancargo test -p datafusion_iceberg --lib parquet_metadata_cache: 20 passed, includingsize_hint_collapses_cold_metadata_load_to_one_getand the page-index widening / back-off testscargo test -p datafusion_iceberg --tests -j 2(=make test-datafusion_iceberg): every suite passed exceptintegration_trino_rest, which cannot run on this host at all: it bind-mounts a temp file from/tmpinto the Trino container and snap Docker cannot see/tmp(bind source path does not exist: /tmp/.tmp…/iceberg-properties); unrelated to this change95236d32(Fix MAP metrics and nested field IDs #73): fmt, strict clippy and the lib tests re-run cleanAlso validated end to end on rustice with this port pinned: a local embucketd against the Horizon IRC bootstraps and scans
RUSTICE_DEV.SANDBOX(TPC-H SF1, HITS_10M).GitHub Actions in this org is currently not starting jobs ("recent account payments have failed or your spending limit needs to be increased"), so the check status on this PR is not meaningful; the commands above replicate
make fmt,make clippyand the module's tests.