Repository navigation
fix(physical-plan): preserve Exact(0) in FilterExec for null_count, distinct_count and total_byte_size upon empty input - #24000
Merged
alamb merged 4 commits intoAug 4, 2026
Conversation
…d distinct_count at num_rows `cap_at_rows` demoted `null_count` and `distinct_count` to inexact unconditionally, so an exactly empty input came out of `FilterExec` with `Exact(0)` for `num_rows`, `total_byte_size` and `byte_size` but `Inexact(0)` for the two counts.
…dictory predicate A contradictory predicate already yielded `Exact(0)` for `num_rows` and per-column `byte_size`, but `total_byte_size` went through the selectivity estimate and came out inexact for any non-empty input.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24000 +/- ##
==========================================
+ Coverage 80.84% 80.85% +0.01%
==========================================
Files 1096 1101 +5
Lines 373832 375032 +1200
Branches 373832 375032 +1200
==========================================
+ Hits 302240 303248 +1008
- Misses 53563 53679 +116
- Partials 18029 18105 +76 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
cc: @kosiew and @tschwarzinger, in case you have bandwidth, as you have the full context from #23936 and #23670 |
tschwarzinger
approved these changes
Jul 31, 2026
tschwarzinger
left a comment
tschwarzinger
left a comment
Contributor
There was a problem hiding this comment.
Looks like a good change to me. Thanks again!
kosiew
requested changes
Jul 31, 2026
Contributor
There was a problem hiding this comment.
@asolimando
Thanks for the update! I found a correctness issue around preserving exact empty statistics.
…ty FilterExec output `total_byte_size` was exact only for a contradictory predicate, and the per-column `byte_size` was never exact, so an input with `num_rows: Exact(0)` produced an exact zero row count next to an inexact or `Absent` byte size. `scale_byte_size_at_rows` returns an exact zero when the filtered row count is an exact zero, which also covers the `is_infeasible` case, so that branch is no longer needed. It tests the row count, unlike the `scale_byte_size` removed in apache#23936, which tested the byte size itself.
kosiew
approved these changes
Aug 3, 2026
Contributor
|
Thanks @asolimando and @kosiew |
kosiew
pushed a commit
to kosiew/datafusion
that referenced
this pull request
Aug 12, 2026
…istinct_count and total_byte_size upon empty input (apache#24000) ## Which issue does this PR close? - Part of apache#8227 ## Rationale for this change Follow-up to apache#23936 (suggested [here](apache#23936 (comment))). apache#23936 made `FilterExec` report `num_rows: Exact(0)` for a provably empty result. Two places in the same operator still turn the same proof into an estimate: - `cap_at_rows` demotes `null_count` and `distinct_count` unconditionally - `total_byte_size` is computed after the "infeasibility branch", so a contradictory predicate (`a = 1 AND a = 2`) gives `Exact(0)` for `num_rows` and per-column `byte_size` but an inexact `total_byte_size` ## What changes are included in this PR? - `cap_at_rows` now maps an `Exact(0)` row bound to `Exact(0)` - `statistics_helper` reports `total_byte_size: Exact(0)` for a contradictory predicate instead of routing it through the selectivity estimate ## Are these changes tested? Yes, both by extending `test_filter_statistics_preserves_exactly_empty_input` with two assertions. Both assertions were confirmed to fail without the respective fix. No sqllogictest baselines change. ## Are there any user-facing changes? No breaking changes and no signature changes. `FilterExec` reports exact rather than inexact zeros for these statistics, visible in `EXPLAIN` output that shows statistics. ---- Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Follow-up to #23936 (suggested here).
#23936 made
FilterExecreportnum_rows: Exact(0)for a provably empty result. Two places in the same operator still turn the same proof into an estimate:cap_at_rowsdemotesnull_countanddistinct_countunconditionallytotal_byte_sizeis computed after the "infeasibility branch", so a contradictory predicate (a = 1 AND a = 2) givesExact(0)fornum_rowsand per-columnbyte_sizebut an inexacttotal_byte_sizeWhat changes are included in this PR?
cap_at_rowsnow maps anExact(0)row bound toExact(0)statistics_helperreportstotal_byte_size: Exact(0)for a contradictory predicate instead of routing it through the selectivity estimateAre these changes tested?
Yes, both by extending
test_filter_statistics_preserves_exactly_empty_inputwith two assertions.Both assertions were confirmed to fail without the respective fix. No sqllogictest baselines change.
Are there any user-facing changes?
No breaking changes and no signature changes.
FilterExecreports exact rather than inexact zeros for these statistics, visible inEXPLAINoutput that shows statistics.Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.