perf: preserve dictionary encoding for lower/upper to avoid materializing low-cardinality columns by lyne7-sc · Pull Request #22905 · apache/datafusion

lyne7-sc · 2026-06-11T09:45:45Z

Which issue does this PR close?

Related to How Null/Dictionary/REE datatypes interact with Coercion API #19458
Related to String scalar functions should preserve Dictionary encoding for Dictionary-typed inputs #20935

Rationale for this change

When a Dictionary(K, Utf8) column is passed to a string scalar function, the type-coercion layer currently materializes it to flat Utf8/Utf8View before the function runs, so the operation is applied to every row instead of just the unique dictionary values, and the dictionary encoding is lost on the output. See #19458 for the underlying coercion behavior and #20935 for the string-function-specific impact.

This is wasteful for low-cardinality columns and inflates Arrow IPC/Flight message sizes downstream.

What changes are included in this PR?

Add EncodingPreservation { None, Dictionary } and opt-in constructors on Coercion (new_exact_preserving_encoding, with_encoding_preservation).
In get_valid_types, when a Coercible arg requests Dictionary preservation, run coercion against the dictionary's value type and re-wrap the result as Dictionary(K, V'), so the function receives a DictionaryArray.
Make lower/upper opt in and handle dictionary inputs (array + scalar) by converting only the dictionary values and re-wrapping with the original keys.

Are these changes tested?

Yes

Are there any user-facing changes?

Yes. upper/lower now return Dictionary(...) for dictionary inputs instead of the previously materialized Utf8View. New public API (EncodingPreservation, new Coercion constructors) is added — please add the api change label.

lyne7-sc · 2026-06-11T09:50:01Z

Hi @Jefffrey, following up on #19458. I prototyped one approach and wanted to check it with you before going further. Would you be interested in taking a look?

github-actions · 2026-06-11T10:05:58Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-expr v54.0.0 (current)
       Built [  27.621s] (current)
     Parsing datafusion-expr v54.0.0 (current)
      Parsed [   0.074s] (current)
    Building datafusion-expr v54.0.0 (baseline)
       Built [  26.909s] (baseline)
     Parsing datafusion-expr v54.0.0 (baseline)
      Parsed [   0.078s] (baseline)
    Checking datafusion-expr v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   1.348s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  57.057s] datafusion-expr
    Building datafusion-expr-common v54.0.0 (current)
       Built [  19.314s] (current)
     Parsing datafusion-expr-common v54.0.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-expr-common v54.0.0 (baseline)
       Built [  18.917s] (baseline)
     Parsing datafusion-expr-common v54.0.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-expr-common v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.215s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure enum_struct_variant_field_added: pub enum struct variant field added ---

Description:
An enum's exhaustive struct variant has a new field, which has to be included when constructing or matching on this variant.
        ref: https://doc.rust-lang.org/reference/attributes/type_system.html#the-non_exhaustive-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_struct_variant_field_added.ron

Failed in:
  field encoding_preservation of variant Coercion::Exact in /home/runner/work/datafusion/datafusion/datafusion/expr-common/src/signature.rs:1053
  field encoding_preservation of variant Coercion::Implicit in /home/runner/work/datafusion/datafusion/datafusion/expr-common/src/signature.rs:1063

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  39.179s] datafusion-expr-common
    Building datafusion-functions v54.0.0 (current)
       Built [  30.678s] (current)
     Parsing datafusion-functions v54.0.0 (current)
      Parsed [   0.085s] (current)
    Building datafusion-functions v54.0.0 (baseline)
       Built [  29.939s] (baseline)
     Parsing datafusion-functions v54.0.0 (baseline)
      Parsed [   0.085s] (baseline)
    Checking datafusion-functions v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.419s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  62.482s] datafusion-functions
    Building datafusion-sqllogictest v54.0.0 (current)
       Built [ 177.687s] (current)
     Parsing datafusion-sqllogictest v54.0.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-sqllogictest v54.0.0 (baseline)
       Built [ 176.598s] (baseline)
     Parsing datafusion-sqllogictest v54.0.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.089s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 357.278s] datafusion-sqllogictest

Jefffrey

just an initial comment, hope to fully review it soon

Jefffrey · 2026-06-13T05:23:32Z

+/// Controls whether a [`Coercion`] preserves an argument's physical encoding
+/// (e.g. dictionary) instead of materializing it to the coerced value type.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Hash)]
+pub enum EncodingPreservation {
+    /// Do not request preservation of a physical encoding.
+    None,
+    /// Preserve dictionary encoding and coerce only the dictionary values.
+    Dictionary,
+}


Something to consider is if an enum is best suited for this 🤔

For example, if we want to also include run encoded arrays as well (since they are similar to dictionaries), would this mean two new variants, one just for run arrays and one for dictionary + run arrays?

I don't know if arrow is planning to include any more types of encodings like this at the moment

So I was also thinking perhaps a bitflag approach (or just a struct with boolean flags) could be another approach:

struct Encoding { preserve_dictionary: bool, preserve_run: bool, }

But I don't know how common a use case would be to implement preservation only for dictionaries and not run arrays (or vice versa) so maybe thats overengineering 🤔

Considering RunEndEncoded and other potential encoding types, I agree that using a struct with boolean flags is a better approach. It keeps the complexity low while making it easier to add future preservation options without introducing enum variants for every possible combination.

Jefffrey · 2026-06-17T04:04:58Z

    }

+    #[test]
+    fn test_coercible_dictionary_preserves_encoding() -> Result<()> {


Could we also have a test for a TypeSignatureClass that isn't native? curious to see how it'll work, since #19458 highlighted that current behaviour for dictionaries is already different for TypeSignatureClass::Native and non-native (e.g. TypeSignatureClass::Integer)

Added tests for both cases:

The behavior now looks like this:

TypeSignatureClass no preservation dictionary preservation

Native(Int64) Int64 Dictionary(Int8, Int64)

Non-Native(Integer) Dictionary(Int8, Int32) Dictionary(Int8, Int32)

Thanks for checking this; I'm a little worried because of the discrepancy here now; is it a big task to try align them? Though we might need to check the UDFs of existing functions that use a non-native typesignatureclass in case they already knew this assumption and had dictionary paths 🤔

It looks like this discrepancy already exists today. Typed non-Native TypeSignatureClass values can pass Dictionary through, while Native coercions materialize to the target type. But this pr makes that contrast more visible by adding an explicit preservation API for the Native path.

If we want EncodingPreservation to be the general mechanism for deciding whether Dictionary is preserved, then I agree we should also think about typed non-Native TypeSignatureClass values. I took a quick pass over the affected built-ins. The good news is that I did not find an obvious core DataFusion function in the affected typed non-Native TypeSignatureClass path that intentionally depends on receiving Dictionary(_, T)🙂. Most seem to dispatch on T directly, and some would likely reject Dictionary(_, T) today after type coercion has accepted it. From that angle, aligning the default behavior may actually fix cases where dictionary inputs currently pass signature matching but fail later in the function implementation.

I did find a couple of Spark compatibility functions, such as spark hex and bitmap_count, that have dictionary paths and would likely need explicit opt-in if we align the default behavior.

From my side, the set of built-ins that may need changes looks bounded, so aligning the behavior seems manageable.

Thanks for checking this; I think we can separate that work into a followup PR 👍

lyne7-sc · 2026-06-24T08:31:08Z

@Jefffrey All feedback has been resolved, please take another look when you have a chance

Jefffrey

sorry i took so long to get back to this, ill try prioritize it now

Jefffrey · 2026-07-05T08:24:51Z

    }

+    #[test]
+    fn test_coercible_dictionary_preserves_encoding() -> Result<()> {


Thanks for checking this; I'm a little worried because of the discrepancy here now; is it a big task to try align them? Though we might need to check the UDFs of existing functions that use a non-native typesignatureclass in case they already knew this assumption and had dictionary paths 🤔

Jefffrey

I think we just need to add a note to the upgrade guide; other than that, this PR looks good to me

Jefffrey · 2026-07-06T14:51:06Z

    }

+    #[test]
+    fn test_coercible_dictionary_preserves_encoding() -> Result<()> {


Thanks for checking this; I think we can separate that work into a followup PR 👍

lyne7-sc · 2026-07-07T07:04:17Z

Thanks @Jefffrey , I added a note to the 55.0.0 upgrade guide for the new Coercion dictionary encoding preservation support.

As a follow-up, I will look into the differing coercion behavior, and check which remaining scalar functions can safely opt in to dictionary preservation. That should help address #20935 more completely.

Jefffrey · 2026-07-08T15:29:18Z

thanks @lyne7-sc

## Which issue does this PR close?  - Follow-up to apache#22905. - Part of apache#19458 ## Rationale for this change  apache#22905 introduced explicit dictionary encoding preservation for coercible function signatures, but dictionary inputs were still handled differently across `TypeSignatureClass` variants: | Signature category | Before | After | | --- | --- | --- | | `Native(...)` | Materialized by default; preserved when explicitly requested | Same default/opt-in contract | | Typed non-Native (e.g. `Integer`, `Numeric`, `Binary`) | Retained the physical dictionary type by default | Materialized by default; preserved when explicitly requested | | `Any` | Passed through the original physical input type | Unchanged | This PR makes the encoding preservation contract consistent across all typed signature classes: coercion operates on the dictionary value type, and the dictionary encoding is restored only when `EncodingPreservation::dictionary()` is enabled. An audit of the affected built-ins found two functions, Spark hex and bitmap_count, that intentionally handle dictionary inputs; both now opt in explicitly. Other affected functions generally expect materialized value arrays, so the new default also avoids cases where signature matching accepted a dictionary but the function implementation rejected it at execution time. Functions that continue to materialize dictionary inputs do not gain dictionary-aware execution efficiency yet, but they can opt in later if they add support for encoded inputs. ## What changes are included in this PR?  - Align dictionary coercion across typed signature classes and preserve dictionary encoding when explicitly requested. - Explicitly enable dictionary preservation for Spark `bitmap_count` and the binary variant of Spark `hex`. - Document the behavior change and migration guidance in the DataFusion 55.0.0 upgrade guide. ## Are these changes tested?  - Unit tests cover materialization and preservation for Native, non-Native, and `Any` inputs. - SLTs cover `to_hex` materialization and verify that `bitmap_count` preserves its dictionary input without an additional cast to `Binary`. - Existing Spark `hex` dictionary tests cover its opt-in preservation behavior. ## Are there any user-facing changes?   This is a behavioral API change for UDFs using typed non-Native classes such as `Integer` or `Binary`. UDFs relying on implicit dictionary preservation must now enable `EncodingPreservation::dictionary()` explicitly. `TypeSignatureClass::Any` is unaffected. The upgrade guide has been updated, and this PR should carry the `api change` label.

…and `ascii` (apache#23743) ## Which issue does this PR close?  - Follow-up to apache#22905 - Related to apache#19458 - Related to apache#20935 ## Rationale for this change  String functions normally materialize `Dictionary(K, Utf8)` inputs before evaluation. This loses the dictionary encoding and applies the function to every row instead of only the unique dictionary values. This pr extends the dictionary-preserving implementation from apache#22905 to `ascii`, `bit_length`, and `octet_length`. ## What changes are included in this PR?  - Preserve dictionary encoding for `ascii`, `bit_length`, and `octet_length`. - Preserve return types for regular and nested dictionaries. - Add slts and Dictionary cardinality benchmarks. ## Are these changes tested? Yes, covered by SLTs.  ## Are there any user-facing changes?   Yes. These functions now preserve Dictionary encoding in their output: `Dictionary(K, Utf8) -> Dictionary(K, Int32)` ## Benchmarks ``` group main new ----- ---------------- --- dictionary_string_functions/cardinality_10/ascii 28.71 5.0±0.05µs ? ?/sec 1.00 172.5±12.31ns ? ?/sec dictionary_string_functions/cardinality_10/bit_length 4.01 698.7±25.75ns ? ?/sec 1.00 174.1±2.45ns ? ?/sec dictionary_string_functions/cardinality_10/octet_length 3.63 651.3±58.51ns ? ?/sec 1.00 179.4±3.27ns ? ?/sec dictionary_string_functions/cardinality_100/ascii 18.92 4.9±0.03µs ? ?/sec 1.00 258.6±8.28ns ? ?/sec dictionary_string_functions/cardinality_100/bit_length 3.11 723.4±27.14ns ? ?/sec 1.00 232.5±35.47ns ? ?/sec dictionary_string_functions/cardinality_100/octet_length 3.06 651.6±26.97ns ? ?/sec 1.00 213.2±6.96ns ? ?/sec dictionary_string_functions/cardinality_1000/ascii 6.32 4.8±0.03µs ? ?/sec 1.00 766.7±70.66ns ? ?/sec dictionary_string_functions/cardinality_1000/bit_length 3.06 776.1±67.85ns ? ?/sec 1.00 253.9±6.93ns ? ?/sec dictionary_string_functions/cardinality_1000/octet_length 3.10 807.9±188.65ns ? ?/sec 1.00 261.0±22.42ns ? ?/sec dictionary_string_functions/cardinality_8192/ascii 1.00 5.1±0.15µs ? ?/sec 1.00 5.1±0.24µs ? ?/sec dictionary_string_functions/cardinality_8192/bit_length 1.00 699.1±17.54ns ? ?/sec 1.09 763.7±36.57ns ? ?/sec dictionary_string_functions/cardinality_8192/octet_length 1.00 677.3±62.91ns ? ?/sec 1.11 750.7±32.56ns ? ?/sec ```

… and `reverse` (apache#23930) ## Which issue does this PR close?  - Follow-up to apache#22905 - Related to apache#19458 and apache#20935 ## Rationale for this change  Previously, coercion materialized dictionary-encoded inputs for `character_length`, `initcap`, and `reverse`. This lost the encoding and evaluated the function for every row instead of once per dictionary value entry. This PR extends dictionary preservation to these functions. `character_length` and `reverse` now use `Coercible` signatures to explicitly model string inputs and binary-to-string coercion while preserving dictionary encoding. ## What changes are included in this PR?  - Preserve dictionary encoding for `character_length`, `initcap`, and `reverse`. - Evaluate only dictionary values while reusing the original keys. - Migrate `character_length` and `reverse` from `Uniform` to `Coercible`. - Add tests and benchmarks. ## Are these changes tested?  Yes, covered by SQL logic tests. ## Are there any user-facing changes?   Yes. These functions now preserve dictionary encoding in their output. `character_length` and `reverse` now accept logical string and binary inputs instead of implicitly converting unrelated types to strings. ## Benchmark ``` group branch main ----- ------ ---- dictionary_encoding/string/cardinality_10/character_length 1.00 218.5±32.62ns ? ?/sec 31.04 6.8±0.07µs ? ?/sec dictionary_encoding/string/cardinality_10/initcap 1.00 292.2±4.75ns ? ?/sec 359.88 105.2±0.91µs ? ?/sec dictionary_encoding/string/cardinality_10/reverse 1.00 501.6±15.71ns ? ?/sec 140.60 70.5±1.05µs ? ?/sec dictionary_encoding/string/cardinality_100/character_length 1.00 307.5±24.93ns ? ?/sec 21.85 6.7±0.09µs ? ?/sec dictionary_encoding/string/cardinality_100/initcap 1.00 1486.1±65.36ns ? ?/sec 70.56 104.9±1.57µs ? ?/sec dictionary_encoding/string/cardinality_100/reverse 1.00 1410.9±52.69ns ? ?/sec 49.28 69.5±0.68µs ? ?/sec dictionary_encoding/string/cardinality_1000/character_length 1.00 1007.4±93.42ns ? ?/sec 6.64 6.7±0.07µs ? ?/sec dictionary_encoding/string/cardinality_1000/initcap 1.00 12.8±0.17µs ? ?/sec 8.41 107.5±8.55µs ? ?/sec dictionary_encoding/string/cardinality_1000/reverse 1.00 10.5±0.24µs ? ?/sec 6.56 69.1±0.58µs ? ?/sec dictionary_encoding/string/cardinality_8192/character_length 1.00 6.6±0.09µs ? ?/sec 1.02 6.7±0.18µs ? ?/sec dictionary_encoding/string/cardinality_8192/initcap 1.00 104.2±1.36µs ? ?/sec 1.00 104.2±1.63µs ? ?/sec dictionary_encoding/string/cardinality_8192/reverse 1.24 85.4±2.99µs ? ?/sec 1.00 69.0±0.58µs ? ?/sec ``` --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

github-actions Bot added logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Jun 11, 2026

github-actions Bot added the auto detected api change Auto detected API change label Jun 11, 2026

Jefffrey reviewed Jun 13, 2026

View reviewed changes

github-actions Bot added the documentation Improvements or additions to documentation label Jun 15, 2026

Jefffrey added the api change Changes the API exposed to users of the crate label Jun 17, 2026

Jefffrey reviewed Jun 17, 2026

View reviewed changes

lyne7-sc requested a review from Jefffrey June 24, 2026 08:28

Jefffrey reviewed Jul 5, 2026

View reviewed changes

lyne7-sc and others added 6 commits July 6, 2026 22:09

perf: preserve dictionary encoding for case conversion

1896e06

Update signature.rs

4d32c12

docs: use Coercion constructor in UDF examples

8f0f701

use boolean flags for encoding preservation

b1bb38d

address dictionary preservation review feedback

789014d

add tests

adbdd13

lyne7-sc force-pushed the perf/dictionary_scalar branch from 6fc55ee to adbdd13 Compare July 6, 2026 14:21

Jefffrey reviewed Jul 6, 2026

View reviewed changes

added upgrading documentation

364d9d0

Jefffrey approved these changes Jul 8, 2026

View reviewed changes

Jefffrey added this pull request to the merge queue Jul 8, 2026

Merged via the queue into apache:main with commit 3089ace Jul 8, 2026
40 checks passed

lyne7-sc deleted the perf/dictionary_scalar branch July 9, 2026 06:40

lyne7-sc mentioned this pull request Jul 14, 2026

fix: align dictionary coercion across typed signatures #23549

Merged

Jefffrey mentioned this pull request Jul 18, 2026

Support lower and upper scalar udf on dict arrays #22647

Closed

lyne7-sc mentioned this pull request Jul 21, 2026

perf: preserve dictionary encoding for bit_length, octet_length, and ascii #23743

Merged

lyne7-sc mentioned this pull request Jul 28, 2026

perf: preserve dictionary encoding for character_length, initcap, and reverse #23930

Merged

TypeSignatureClass	no preservation	dictionary preservation
`Native(Int64)`	`Int64`	`Dictionary(Int8, Int64)`
`Non-Native(Integer)`	`Dictionary(Int8, Int32)`	`Dictionary(Int8, Int32)`

Uh oh!

Conversation

lyne7-sc commented Jun 11, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

lyne7-sc commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lyne7-sc Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lyne7-sc commented Jun 24, 2026

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lyne7-sc commented Jul 7, 2026

Uh oh!

Jefffrey commented Jul 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jun 11, 2026 •

edited

Loading

lyne7-sc Jun 18, 2026 •

edited

Loading