introduce optional rle reads from parquet by Rich-T-kid · Pull Request #24227 · apache/datafusion

Rich-T-kid · 2026-08-10T15:21:19Z

Which issue does this PR close?

Rationale for this change

When DataFusion reads a parquet file with dictionary-encoded string or binary columns, it currently decodes the dictionary and returns plain Utf8/Binary arrays, discarding the encoding. For low-cardinality columns (status, country, category, etc.) this doesn't take full advantage of the compacted format parquet gives the engine Preserving the dictionary encoding reduces memory usage and can improve aggregation performance on these columns.

What changes are included in this PR?

Adds datafusion.execution.parquet.enable_rle_to_dictionary (default false). When enabled:

DFParquetMetadata::fetch_schema rewrites dictionary-encoded string/binary fields to Dictionary(Int32, Utf8/Binary) at schema inference time, so physical plans are built with the correct types
prepare_filters in the parquet opener detects the same columns at execution time and passes the rewritten schema to ArrowReaderOptions::with_schema, telling the arrow-rs reader to produce dictionary arrays directly

Are these changes tested?

Unit tests in opener/mod.rs verify string and binary columns are read as Dictionary(Int32, Utf8) / Dictionary(Int32, Binary) when the flag is enabled, and as plain types when disabled
Unit tests in source.rs verify flag propagation through config and end-to-end type correctness through a DataSourceExec -> FilterExec -> ProjectionExec plan
A sqllogictest file writes a real parquet file and uses arrow_typeof() to assert the physical Arrow type changes between the two flag states

Are there any user-facing changes?

New session config option: SET datafusion.execution.parquet.enable_rle_to_dictionary = true. Default is false so existing behavior is unchanged.

Rich-T-kid · 2026-08-10T15:28:14Z

+        // to Dictionary(Int32, ...) so the parquet reader produces dictionary arrays.
+        // All three schemas are updated together so the reader, expr-adapter, and
+        // downstream operators all agree on the output type.
+        if prepared.enable_rle_to_dictionary {


All three schemas are updated together so the reader, expr-adapter, and downstream operators all agree on the output type.

Rich-T-kid · 2026-08-10T15:33:05Z

        pub max_in_list_size: usize, default = 20

+        /// (reading) If true, string and binary columns that are dictionary-encoded in
+        /// the parquet file are read as `Dictionary<Int32, Utf8>` / `Dictionary<Int32, Binary>`


I choose to keep the key type hard coded to int32 instead of u64, would not mind changing

github-actions · 2026-08-10T15:34:06Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [  61.602s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.036s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [  60.426s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.576s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 124.939s] datafusion
    Building datafusion-common v54.1.0 (current)
       Built [  33.498s] (current)
     Parsing datafusion-common v54.1.0 (current)
      Parsed [   0.060s] (current)
    Building datafusion-common v54.1.0 (baseline)
       Built [  33.094s] (baseline)
     Parsing datafusion-common v54.1.0 (baseline)
      Parsed [   0.061s] (baseline)
    Checking datafusion-common v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.656s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1196

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  68.728s] datafusion-common
    Building datafusion-datasource-parquet v54.1.0 (current)
       Built [  47.573s] (current)
     Parsing datafusion-datasource-parquet v54.1.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion-datasource-parquet v54.1.0 (baseline)
       Built [  47.469s] (baseline)
     Parsing datafusion-datasource-parquet v54.1.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion-datasource-parquet v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.159s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  97.066s] datafusion-datasource-parquet
    Building datafusion-proto-common v54.1.0 (current)
       Built [  21.877s] (current)
     Parsing datafusion-proto-common v54.1.0 (current)
      Parsed [   0.045s] (current)
    Building datafusion-proto-common v54.1.0 (baseline)
       Built [  21.742s] (baseline)
     Parsing datafusion-proto-common v54.1.0 (baseline)
      Parsed [   0.047s] (baseline)
    Checking datafusion-proto-common v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.076s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:868
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:868
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:868

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  46.045s] datafusion-proto-common
    Building datafusion-proto-models v54.1.0 (current)
       Built [  24.581s] (current)
     Parsing datafusion-proto-models v54.1.0 (current)
      Parsed [   0.127s] (current)
    Building datafusion-proto-models v54.1.0 (baseline)
       Built [  24.492s] (baseline)
     Parsing datafusion-proto-models v54.1.0 (baseline)
      Parsed [   0.129s] (baseline)
    Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.717s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:868
  field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:868

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  52.306s] datafusion-proto-models
    Building datafusion-sqllogictest v54.1.0 (current)
       Built [  98.856s] (current)
     Parsing datafusion-sqllogictest v54.1.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-sqllogictest v54.1.0 (baseline)
       Built [  99.221s] (baseline)
     Parsing datafusion-sqllogictest v54.1.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-sqllogictest v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.098s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 201.303s] datafusion-sqllogictest

codecov-commenter · 2026-08-10T19:42:57Z

Codecov Report

❌ Patch coverage is 93.11224% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.18%. Comparing base (1164d60) to head (46ee5ac).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto-common/src/generated/pbjson.rs	0.00%	14 Missing ⚠️
datafusion/datasource-parquet/src/source.rs	96.64%	5 Missing and 1 partial ⚠️
datafusion/datasource-parquet/src/metadata.rs	92.30%	4 Missing ⚠️
datafusion/datasource-parquet/src/opener/mod.rs	97.81%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24227      +/-   ##
==========================================
+ Coverage   81.14%   81.18%   +0.04%     
==========================================
  Files        1112     1109       -3     
  Lines      386933   388424    +1491     
  Branches   386933   388424    +1491     
==========================================
+ Hits       313978   315345    +1367     
- Misses      54462    54534      +72     
- Partials    18493    18545      +52

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Rich-T-kid · 2026-08-12T23:58:21Z

@adriangb pinging you since you seem interested in parquet related speed ups 👍

adriangb · 2026-08-13T12:49:06Z

If I understand correctly the goal is to evaluate filters during filter pushdown against dictionary / RLE encoded columns? We can't propagate these dynamic type changes to the rest of the query plan / scan. Is that right?

Rich-T-kid · 2026-08-13T13:55:45Z

@adriangb

If I understand correctly the goal is to evaluate filters during filter pushdown against dictionary / RLE encoded columns?

no not exactly. The goal of this PR is to keep RLE parquet columns in their compacted form by materializing them as dictionary arrays instead of regular strings.

We can't propagate these dynamic type changes to the rest of the query plan / scan. Is that right?

exactly! This is why it needs to be done as early in the plan as possible. we inspect DFParquetMetadata::fetch_schema to see if any RLE columns exist, if so change the plan type from utf8/binary to dict<_,utf8/binary>. Thanks to some plumbing in arrow-rs it will handle the conversions correctly returning a dictionary array, this can cause a 60x perf boost

adriangb · 2026-08-13T15:47:56Z

no not exactly. The goal of this PR is to keep RLE parquet columns in their compacted form by materializing them as dictionary arrays instead of regular strings.

Why only RLE and not dictionaries as well? How does this compare to / relate to the schema_force_view_types option?

It also looks like this goes through infer_schema right? A lot of code paths never touch that (CREATE EXTERNAL TABLE (a VARCHAR), any custom table providers, etc.).

I'd be more interested in seeing something at the parquet scan level that was able to e.g. optimize how row filters are applied by applying them to the dictionary instead of expanding into Utf8View. That would be applicable to all DataFusion users.

Rich-T-kid · 2026-08-13T17:13:11Z

Why only RLE and not dictionaries as well? How does this compare to / relate to the schema_force_view_types option?

my bad when I say RLE i'm referring to RLE_DICTIONARY

It also looks like this goes through infer_schema right? A lot of code paths never touch that (CREATE EXTERNAL TABLE (a VARCHAR), any custom table providers, etc.).

I'd be more interested in seeing something at the parquet scan level that was able to e.g. optimize how row filters are applied by applying them to the dictionary instead of expanding into Utf8View. That would be applicable to all DataFusion users.

I agree, ill update the PR to target all parquet scans.

Rich-T-kid · 2026-08-13T18:52:20Z

@adriangb your comments are addressed in 8bc183b
Previously the promotion only applied when the schema was inferred via CREATE EXTERNAL TABLE (no explicit columns). Explicit schemas (CREATE EXTERNAL TABLE (col VARCHAR)) and custom TableProvider implementations never touched infer_schema so the flag had no effect on those paths.

ParquetSource::with_table_parquet_options now promotes all string/binary fields to Dictionary(Int32, …) at planning time regardless of how the schema was established, since every ParquetSource passes through this method.

To keep planning and execution in sync, the opener also rewrites the physical read schema so arrow-rs produces dict arrays. Without the planning-time promotion, downstream operators (FilterExec, AggregateExec) would be compiled against Utf8 while the scan produces Dictionary(Int32, Utf8), a type mismatch at execution time.

test

Added tests covering:

Unit tests in datasource-parquet: flag propagation, RLE-encoded and plain-encoded files with the flag on/off
Integration tests in core/tests/parquet/schema_coercion.rs: scan -> filter -> project using ParquetSource directly (no infer_schema) to validate the explicit-schema path; end-to-end scan -> filter -> aggregate via SessionContext and SQL to validate the full planning pipeline
SLT (parquet_rle_to_dictionary.slt): arrow_typeof assertions with the flag on/off, GROUP BY over dict columns to confirm aggregation works correctly

Rich-T-kid · 2026-08-13T18:59:51Z

ideally we surface columns that are physically RLE_DICTIONARY-encoded in the parquet file as Arrow Dictionary(Int32, Utf8) arrays rather than decoding them back to plain Utf8.

To know whether a specific column is RLE_DICTIONARY-encoded you need to read the parquet file footer. For the infer_schema path this happens at table registration, but for explicit schemas (CREATE EXTERNAL TABLE (col VARCHAR)) and direct ParquetSource construction no footer is ever read during planning, so per-column encoding information isn't available for all paths.

Downstream physical operators (FilterExec, AggregateExec) are compiled against the scan's declared output schema during physical planning, before any files are opened. If the scan declares Utf8 but produces Dictionary(Int32, Utf8) at execution time that's a type mismatch.

So when the flag is enabled we promote all string/binary columns to dict at planning time, not just the ones that are actually RLE-encoded, because that's the only way to guarantee schema consistency across all parquet scan paths without introducing file I/O into the planning stage.

I feel like i'm missing something here. if we could take a peak at the parquets metadata before physical planning and change the schema for all operators from the point forward that would be perfect. Im not sure this is currently possible

github-actions Bot added sqllogictest SQL Logic Tests (.slt) common Related to common crate datasource Changes to the datasource crate labels Aug 10, 2026

Rich-T-kid commented Aug 10, 2026

View reviewed changes

github-actions Bot added the auto detected api change Auto detected API change label Aug 10, 2026

Rich-T-kid marked this pull request as ready for review August 10, 2026 15:45

github-actions Bot added proto Related to proto crate documentation Improvements or additions to documentation labels Aug 10, 2026

Rich-T-kid added 6 commits August 13, 2026 10:01

introduce optional rle reads from parquet

10d3b56

fix lint

e152f63

thread flag to protos

c1d232c

update docs/ fix ci

c734b15

fix lint

3d4cba6

regen proto to fix CI formatting check

6e1cbc1

Rich-T-kid force-pushed the rich-T-kid/introduce-rle-parquet-flag branch from 0479f9c to 6e1cbc1 Compare August 13, 2026 15:10

trigger CI

d1273bf

Rich-T-kid force-pushed the rich-T-kid/introduce-rle-parquet-flag branch from 6a4e89d to d1273bf Compare August 13, 2026 15:48

address PR comments

8bc183b

github-actions Bot added the core Core DataFusion crate label Aug 13, 2026

avoid casting non rle-columns

99a9f11

Rich-T-kid marked this pull request as draft August 13, 2026 19:53

trim pr

46ee5ac

Navigation Menu

Uh oh!

introduce optional rle reads from parquet - #24227

introduce optional rle reads from parquet#24227
Rich-T-kid wants to merge 10 commits into
apache:mainfrom
Rich-T-kid:rich-T-kid/introduce-rle-parquet-flag

Rich-T-kid commented Aug 10, 2026 •

edited

Loading

Uh oh!

Rich-T-kid Aug 10, 2026

Uh oh!

Rich-T-kid Aug 10, 2026

Uh oh!

github-actions Bot commented Aug 10, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 10, 2026 •

edited

Loading

Uh oh!

Rich-T-kid commented Aug 12, 2026

Uh oh!

adriangb commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026 •

edited

Loading

Uh oh!

adriangb commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026 •

edited

Loading

Uh oh!

Rich-T-kid commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Rich-T-kid commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Rich-T-kid Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

Rich-T-kid Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Rich-T-kid commented Aug 12, 2026

Uh oh!

adriangb commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adriangb commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026

Uh oh!

Rich-T-kid commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

test

Uh oh!

Rich-T-kid commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Rich-T-kid commented Aug 10, 2026 •

edited

Loading

github-actions Bot commented Aug 10, 2026 •

edited

Loading

codecov-commenter commented Aug 10, 2026 •

edited

Loading

Rich-T-kid commented Aug 13, 2026 •

edited

Loading

Rich-T-kid commented Aug 13, 2026 •

edited

Loading