fix: apply struct field filters when the file schema needs adaptation

[adriangb]

Which issue does this PR close?

• Closes Parquet filter pushdown silently drops a get_field predicate when the file needs schema adaptation (wrong results) #24109.

Rationale for this change

With datafusion.execution.parquet.pushdown_filters = true, a filter on a struct field returns all rows when the declared table schema differs from the physical file schema for that column:

-- file stores s as Struct<x: Int32>, table declares Struct<x: BIGINT>
SELECT id, s['x'] FROM t WHERE s['x'] = 200;
-- returns 3 rows instead of 1

The planning-time decision and the runtime construction disagree:

1. ParquetSource::try_pushdown_filters evaluates can_expr_be_pushed_down_with_schemas against the table schema. get_field(s, 'x') has a bare column under the get_field, so it reports the predicate as fully handled and FilterExec is removed from the plan.
2. At open time the expression adapter rewrites the predicate against the file schema. Because the struct types differ, rewrite_column wraps the whole column in a cast, giving get_field(cast(s AS Struct<x: Int64>), 'x').
3. PushdownChecker only recognizes get_field whose first argument is a Column. It now sees a CastExpr, falls through to normal traversal, hits the struct Column, and rejects pushdown — so no row filter is built and the conjunct is silently dropped.

Nothing applies the predicate, and the scan returns unfiltered rows.

What changes are included in this PR?

Narrow the cast to the field that is actually read, in DefaultPhysicalExprAdapter:

get_field(cast(s AS Struct<x: Int64>), 'x')  ->  cast(get_field(s, 'x') AS Int64)

Expressions are rewritten bottom-up, so the new try_narrow_struct_cast matches the get_field node after its struct argument has already been wrapped, and rebuilds the get_field over the uncast struct (recomputing its return field from the physical field type) with the cast moved outside. This keeps the column visible under the get_field, so the Parquet row filter builder makes good on what planning promised.

Two details worth calling out:

• A field that is missing from the file collapses to a typed null literal, matching what the struct cast would have produced (DataFusion's struct casts match by name and fill missing target fields with nulls).
• get_field on a Map column is a runtime key lookup rather than a schema-level field access, so map values keep the whole-column cast.

As a side effect this also avoids materializing an entire cast struct just to read one field, which is a small win for any struct-field access over an evolved schema — not only for filters.

Not addressed here

The issue also raises the broader concern that "a static determination made at planning time about what the scan can do, and the runtime construction that has to make good on it, are computed by different code against different schemas, and there is no mechanism forcing them to agree." This PR fixes the reported wrong-results bug; it does not add a mechanism (e.g. post-decode filtering in ParquetOpener) that would make any future divergence safe by construction. That seems worth doing separately.

Are these changes tested?

Yes.

• datafusion/physical-expr-adapter/src/schema_rewriter.rs: unit tests for the narrowed cast (flat and nested field access), the missing-field null literal, and that Map columns keep their cast.
• datafusion/datasource-parquet/src/opener/mod.rs: end-to-end opener tests reading a Struct<x: Int32> file through a Struct<x: Int64> table schema with pushdown enabled, plus a matching-schema control.
• datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt: a SQL-level regression test. Verified that it fails on main (returns all 3 rows) and passes with the fix.

Full runs: cargo clippy --all-targets --all-features -- -D warnings, the complete sqllogictest suite (498 files), datafusion-physical-expr-adapter, datafusion-datasource-parquet, and the datafusion core_integration / parquet_integration suites all pass.

Are there any user-facing changes?

A wrong-results bug fix: struct-field predicates are now applied when the scan needs schema adaptation. No public API changes.

---

Generated by Claude Code

[adriangb]

Prefer SLT tests if possible

[Copilot]

Pull request overview

Fixes a wrong-results bug in Parquet filter pushdown when struct-field predicates are used and the declared table schema differs from the physical file schema. The change ensures the runtime schema adaptation preserves a get_field(Column(..), ...) shape so Parquet’s row-filter builder can still recognize and apply the predicate that planning-time pushdown claimed would be handled.

Changes:

• Add a physical-expr rewrite that narrows cast(struct) under get_field into cast(get_field(..)), preserving pushdown compatibility and avoiding casting whole structs unnecessarily.
• Add unit tests covering flat, nested, and flattened multi-key get_field paths, missing-field-to-typed-null behavior, and Map behavior (no narrowing).
• Add a SQL logic regression test reproducing issue #24109 and validating correct filtering for both schema-adapted and matching-schema reads.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
datafusion/physical-expr-adapter/src/schema_rewriter.rs	Introduces try_narrow_struct_cast + field-path resolution and adds focused unit tests for the new rewrite behavior.
datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt	Adds an end-to-end SQL regression test to prevent the struct-field predicate from being silently dropped under schema adaptation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 95.62290% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.99%. Comparing base (39d5064) to head (f7bdbd0).
⚠️ Report is 60 commits behind head on main.

Files with missing lines	Patch %	Lines
...usion/physical-expr-adapter/src/schema_rewriter.rs	95.62%	9 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24125      +/-   ##
==========================================
+ Coverage   80.85%   80.99%   +0.14%     
==========================================
  Files        1099     1104       +5     
  Lines      374304   379123    +4819     
  Branches   374304   379123    +4819     
==========================================
+ Hits       302642   307075    +4433     
- Misses      53607    53825     +218     
- Partials    18055    18223     +168     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.