Skip to content

feat: union nested Parquet leaves across projection accesses - #24130

Open
goutamadwant wants to merge 1 commit into
apache:mainfrom
goutamadwant:gh-24121-union-required-leaves
Open

feat: union nested Parquet leaves across projection accesses#24130
goutamadwant wants to merge 1 commit into
apache:mainfrom
goutamadwant:gh-24121-union-required-leaves

Conversation

@goutamadwant

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

When one projected nested column is consumed through multiple narrowing casts, or through both a narrowing cast and get_field, the Parquet read plan currently falls back to reading the entire root column. The leaves required by those consumers can be combined, so reading the full root performs unnecessary I/O.

What changes are included in this PR?

  • Accumulate and deduplicate the leaf offsets required by every narrowing cast on a root column.
  • Add leaves required by get_field accesses to the same union.
  • Derive the Arrow type emitted by the reader from the merged leaf set while retaining a full-read fallback for unsupported partial wrapper shapes.
  • Add coverage for disjoint, overlapping, and repeated casts, cast plus get_field, and casts that diverge below List<Struct>.

Are these changes tested?

Yes. The following checks pass:

  • cargo fmt --all -- --check
  • cargo test -p datafusion-datasource-parquet --lib
  • cargo test --profile=ci --test sqllogictests -- parquet_nested_schema_pruning.slt
  • cargo clippy -p datafusion-datasource-parquet --all-targets --all-features -- -D warnings

The SQL logic test also verifies that the merged projections return the expected values and scan fewer bytes than a full-root read.

Are there any user-facing changes?

No API or SQL behavior changes. Queries with multiple nested accesses to the same Parquet root can read fewer leaf columns.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.05556% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.04%. Comparing base (f27e50c) to head (943a24a).

Files with missing lines Patch % Lines
...ion/datasource-parquet/src/projection_read_plan.rs 92.20% 6 Missing ⚠️
...on/datasource-parquet/src/nested_schema_pruning.rs 94.02% 1 Missing and 3 partials ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #24130    +/-   ##
========================================
  Coverage   81.04%   81.04%            
========================================
  Files        1105     1105            
  Lines      380092   380201   +109     
  Branches   380092   380201   +109     
========================================
+ Hits       308041   308139    +98     
- Misses      53835    53840     +5     
- Partials    18216    18222     +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Union the required leaves when a nested root is reached by multiple casts (or a cast plus get_field) instead of reading the whole column

2 participants