[EPIC] Improvements to struct access in Parquet

[adriangb]

This EPIC tracks work on making queries over struct columns in Parquet read and evaluate only what they need — projection pruning, filter pushdown, row-group pruning, and the statistics that feed them.

Scope is the concrete Parquet-side machinery: datafusion/datasource-parquet/src/{projection_read_plan,row_filter,nested_schema_pruning}.rs and the physical expression adapter's interaction with it. Variant support builds on some of the same primitives but is tracked separately in #21308.

Motivation

Two shapes reach the Parquet scan for struct access, and they are handled by unrelated code:

1. get_field expressions — WHERE s['x'] = 200, SELECT s['a']. Handled by PushdownChecker / StructFieldAccess.
2. A narrowed declared schema — the table declares s STRUCT<a> where the file has STRUCT<a, b, c>. The physical expression adapter rewrites the projected column into CAST(s AS STRUCT<a>), and engines like Spark/Comet communicate nested projection pruning this way, as a clipped read schema rather than as expressions.

Until #24090, shape (2) read every physical leaf and discarded the extras in memory. apache/datafusion-comet#4859 reported a production query reading 1.35 TB where plain Spark read 30.9 GB for the same pruned ReadSchema. Any embedder that hands DataFusion a pre-pruned schema — Comet, delta-rs, Iceberg integrations — hits the same gap.

#24090 fixes that for the projection path. The items below are what it does not cover.

Note that the cast side of shape (2) is already settled: CastColumnExpr was removed in 54.0.0 (#21563) and struct-aware casting is unified into the field-aware CastExpr, which routes through nested_struct::cast_column via ColumnarValue::cast_to. That single cast node is what nested pruning clips against.

Correctness

• Parquet filter pushdown silently drops a get_field predicate when the file needs schema adaptation (wrong results) #24109 — Parquet filter pushdown silently drops a get_field predicate when the file needs schema adaptation (wrong results). Blocks the filter-pushdown work below: on exactly the tables nested pruning targets, WHERE s['field'] = ... with pushdown_filters = true is wrong today.

Projection & filter pruning

• feat: prune unread Parquet leaves when a nested column is cast to a narrower type #24090 — prune unread Parquet leaves when a nested column is cast to a narrower type (projection path)
• Extend Parquet nested schema pruning to filter pushdown #24120 — extend nested schema pruning to filter pushdown
• Union the required leaves when a nested root is reached by multiple casts (or a cast plus get_field) instead of reading the whole column #24121 — union the required leaves when a root is reached by multiple casts, or by a cast and a get_field, instead of falling back to a full read
• Clip ListView / Dictionary / FixedSizeList wrappers in nested schema pruning #24122 — clip ListView / LargeListView / Dictionary / FixedSizeList / RunEndEncoded wrappers, which are conservatively left unclipped today
• Push down IS NULL / IS NOT NULL on struct columns into Parquet scan #21795 — push down IS NULL / IS NOT NULL on struct columns: PushdownChecker rejects whole-struct references, so the predicate materializes every leaf when only the definition levels are needed
• Refactor: Build a reusable struct-access path tree for Parquet row-filter planning #23156 — refactor: build a reusable struct-access path tree for row-filter planning, so projection-mask selection and projected-schema pruning cannot drift
• Introduce ProjectionMask To Allow Nested Projection Pushdown #2581 — introduce ProjectionMask to allow nested projection pushdown (long-standing parent issue for this area)

Row-group pruning & statistics

• Support row group pruning for struct field predicates #20871 — support row group pruning for struct field predicates
• Support parquet statistics for struct columns #8334 — support Parquet statistics for struct columns
• Incorrect statistics read for struct array in parquet #10609 — incorrect statistics read for struct array in Parquet (concrete reproducer for Support parquet statistics for struct columns #8334; the bug that makes struct row-group pruning impossible today)
• Bloom filters and statistics not being used for Map keys/values #17221 — bloom filters and statistics not being used for Map keys/values
• Nested types and extension types are not supported in Statistics / ColumnStatistics #20707 — nested types and extension types are not supported in Statistics / ColumnStatistics

Schema evolution

• Support additive schema evolution for List<Struct> / nested container types in Parquet scans #20835 — support additive schema evolution for List<Struct> / nested container types in Parquet scans (in flight: Add nested struct schema evolution support for Map types #23914 for Map types)
• Feature is not implemeneted: Unsupported cast with list of structs #15338 — Unsupported cast with list of structs: user-facing failure of the nested_struct::cast_column / validate_struct_compatibility path on List<Struct> drift
• Add support for reading Parquet field ids for complex datatypes #20475 — read Parquet field ids for complex datatypes, so nested schema adaptation can match by field id rather than by name (Iceberg / Comet embedders)

Shared with Variant

• All struct-aware optimizations are hardcoded to GetFieldFunc #21306 — all struct-aware optimizations are hardcoded to GetFieldFunc; a custom UDF like variant_get gets none of them. Relevant to both this EPIC and [EPIC] first class support for struct field / Variant access in Parquet #21308.
• ProjectionExec produces unknown statistics for all ScalarFunctionExpr outputs #21307 — ProjectionExec produces unknown statistics for all ScalarFunctionExpr outputs, so every column reached through get_field / variant_get is invisible to the cost model.

Related

• Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema datafusion-comet#4859 — the production report that motivated the projection-side work
• [EPIC] first class support for struct field / Variant access in Parquet #21308 — first-class support for Variant access in Parquet
• Datafusion listing table evolution is dependent on file order #14755 — listing-table schema evolution depends on file order, which affects how struct schemas merge across files
• [EPIC] Improved support for nested / structured types (Struct , List, ListArray, and other Composite types) #2326 (closed) and Support deep schema pruning and projection #11745 (closed) — earlier attempts at this area, kept for history