Skip to content

Optimize Hash Aggregation for Multiple Dictionary-Encoded Group Keys #21878

Description

@Rich-T-kid

Is your feature request related to a problem or challenge?

#21765 introduced GroupValuesDictionary, a specialized GroupValues implementation that operates directly on the structure of a single dictionary-encoded column during hash aggregation. Rather than deserializing dictionary arrays into a generic row-based format (as GroupValuesRows does), it exploits the inherently low-cardinality nature of dictionary arrays, maintaining a mapping of unique value hashes to group IDs so that repeated values are resolved in O(1) after the first encounter.
That optimization is limited to the single-column dictionary case. When a query groups by two or more columns, DataFusion falls back to GroupValuesRows, which serializes all columns regardless of their types, discarding the dictionary encoding properties entirely.
Real-world workloads frequently group by multiple low-cardinality string or categorical columns (e.g., GROUP BY country, product_category, status). These columns are natural candidates for dictionary encoding, and the combination of columns still has bounded cardinality, making the same key-caching insight from #21765 applicable at the multi-column level.

Describe the solution you'd like

Extend the dictionary grouping optimization introduced in #21765 to handle the case of 2+ dictionary-encoded group keys, introducing a multi-column GroupValuesDictionary path (or equivalent) in new_group_values. The approach will work solely on batches where all group-key columns are dictionary arrays, mixed dictionary + non-dictionary group keys will fall back to GroupValuesRows as before and are out of scope for this issue.
The implementation will mirror the strategy from #21765: exploit the compact dictionary representation across multiple columns to avoid redundant serialization and re-hashing of repeated value combinations.

  1. Implement the multi-column dictionary grouping path
  • Validate that there are no correctness regressions relative to GroupValuesRows
  1. Build dedicated benchmarks comparing the new path against GroupValuesRows across multiple axes:
  • Cardinality (low / medium / high number of unique group combinations)
  • Null frequency
  • Batch size
  • Number of group-key columns
  1. Confirm a meaningful performance improvement before merging

Describe alternatives you've considered

No response

Additional context

This is a follow-up to #21765 and continues progress toward #7000 and #7647.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions