IN LIST: reuse primitive filters for FixedSizeBinary - #23018
Draft
geoffreyclaude wants to merge 3 commits into
Draft
IN LIST: reuse primitive filters for FixedSizeBinary#23018geoffreyclaude wants to merge 3 commits into
geoffreyclaude wants to merge 3 commits into
Conversation
11 tasks
geoffreyclaude
force-pushed
the
perf/in_list_fixed_size_binary_filter
branch
3 times, most recently
from
June 18, 2026 10:29
70c420f to
098e0a6
Compare
geoffreyclaude
changed the title
Contributor
Author
|
run benchmark in_list_strategy |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/in_list_fixed_size_binary_filter (098e0a6) to c7e9284 (merge-base) diff using: in_list_strategy File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark in_list |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/in_list_fixed_size_binary_filter (098e0a6) to c7e9284 (merge-base) diff using: in_list File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagein_list — base (merge-base)
in_list — branch
File an issue against this benchmark runner |
geoffreyclaude
force-pushed
the
perf/in_list_fixed_size_binary_filter
branch
from
June 18, 2026 20:55
098e0a6 to
ce699c4
Compare
This was referenced Jun 18, 2026
geoffreyclaude
force-pushed
the
perf/in_list_fixed_size_binary_filter
branch
9 times, most recently
from
June 25, 2026 09:42
2e036e0 to
e49202b
Compare
geoffreyclaude
force-pushed
the
perf/in_list_fixed_size_binary_filter
branch
5 times, most recently
from
June 30, 2026 06:34
ade2587 to
397944d
Compare
geoffreyclaude
force-pushed
the
perf/in_list_fixed_size_binary_filter
branch
3 times, most recently
from
July 2, 2026 08:54
0b737f4 to
49ed00e
Compare
geoffreyclaude
force-pushed
the
perf/in_list_fixed_size_binary_filter
branch
from
July 21, 2026 19:25
49ed00e to
e9cef90
Compare
geoffreyclaude
changed the title
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #23018 +/- ##
==========================================
+ Coverage 80.89% 80.91% +0.02%
==========================================
Files 1102 1105 +3
Lines 376093 377186 +1093
Branches 376093 377186 +1093
==========================================
+ Hits 304239 305200 +961
- Misses 53748 53773 +25
- Partials 18106 18213 +107 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
geoffreyclaude
force-pushed
the
perf/in_list_fixed_size_binary_filter
branch
from
August 4, 2026 09:14
e9cef90 to
821e2d7
Compare
geoffreyclaude
force-pushed
the
perf/in_list_fixed_size_binary_filter
branch
from
August 4, 2026 13:47
821e2d7 to
f69d4c0
Compare
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
INperformance with specialized implementations #19390.Rationale for this change
FixedSizeBinary(1),(2),(4),(8), and(16)values have the same fixed-width buffer shape as the primitive representations already optimized in this stack. Native equality and hashing preserve opaque byte equality, so these arrays can reuse the existing filters through a narrow representation adapter.What changes are included in this PR?
PrimitiveFrozenFilterfor larger lists.FixedSizeBinary(16)rather than broadening general Decimal128 routing.read_unaligned.ArrayStaticFilter; wrong-width or custom needles return errors rather than panicking.Are these changes tested?
Yes. Coverage includes every supported width, both sides of each branchless/FrozenSet boundary, slices, dictionaries,
IN/NOT IN, nulls, empty and all-null lists, duplicates, the FrozenSet sentinel member, unsupported/custom arrays, aligned zero-copy conversion, unaligned buffers, and end-to-endInListExprevaluation.cargo fmt --allcargo test -p datafusion-physical-expr --lib expressions::in_listcargo clippy --all-targets --all-features -- -D warnings./dev/rust_lint.shRUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryptionAre there any user-facing changes?
No. This is an internal performance optimization only.
Local benchmark snapshot
Built and run in separate target directories with identical benchmark source, filtered to the relevant fixed-size binary rows:
Method: directly compared Criterion median point estimates. Expression construction remains outside the timed loop. Lower is better; changes within +/-5% are treated as noise.
Compared baselines: #23016 -> #23018
Relevant scope:
FixedSizeBinary(1),(2),(4),(8), and(16)across bitmap, branchless, and FrozenSet paths.Summary: 20 relevant rows, 20 faster, 0 slower, 0 within +/-5%. Geometric-mean time decreased by 80.7%.
By path, geometric-mean time decreased by 90.8% for bitmap rows, 87.4% for branchless rows, and 66.5% for FrozenSet rows. Miss-only rows decreased by 68.7%; 50%-hit rows decreased by 88.2%.
Largest relevant deltas:
fixed_size_binary/fsb4/list=4/match=50%fixed_size_binary/fsb8/list=4/match=50%fixed_size_binary/fsb2/list=64/match=50%fixed_size_binary/fsb1/list=16/match=50%fixed_size_binary/fsb4/list=4/match=0%fixed_size_binary/fsb8/list=4/match=0%fixed_size_binary/fsb2/list=64/match=0%fixed_size_binary/fsb1/list=16/match=0%fixed_size_binary/fsb4/list=64/match=50%fixed_size_binary/fsb16/list=4/match=50%Full relevant table (20 rows)
fixed_size_binary/fsb1/list=16/match=0%fixed_size_binary/fsb1/list=16/match=50%fixed_size_binary/fsb2/list=64/match=0%fixed_size_binary/fsb2/list=64/match=50%fixed_size_binary/fsb4/list=4/match=0%fixed_size_binary/fsb4/list=4/match=50%fixed_size_binary/fsb4/list=64/match=0%fixed_size_binary/fsb4/list=64/match=50%fixed_size_binary/fsb8/list=4/match=0%fixed_size_binary/fsb8/list=4/match=50%fixed_size_binary/fsb8/list=64/match=0%fixed_size_binary/fsb8/list=64/match=50%fixed_size_binary/fsb16/list=4/match=0%fixed_size_binary/fsb16/list=4/match=50%fixed_size_binary/fsb16/list=64/match=0%fixed_size_binary/fsb16/list=64/match=50%fixed_size_binary/fsb16/list=256/match=0%fixed_size_binary/fsb16/list=256/match=50%fixed_size_binary/fsb16/list=10000/match=0%fixed_size_binary/fsb16/list=10000/match=50%