Repository navigation
fix(common): preserve an exact zero through filter selectivity estimation - #23936
Merged
kosiew merged 1 commit intoJul 30, 2026
Merged
Conversation
…tion `Precision::with_estimated_selectivity` documented itself as always returning inexact statistics, so `Exact(0)` became `Inexact(0)`. Scaling zero by any selectivity is still exactly zero: filtering an input that provably has no rows cannot produce rows, whatever the predicate is. The consequence is not just a weaker estimate. Because operators widen exact to inexact as statistics flow up a plan, emptiness could not be propagated upwards through statistics at all, it was lost at the first `FilterExec`. `FilterExec` already special-cased this for a contradictory predicate, which returns `Exact(0)` directly, and for column byte sizes via a local `scale_byte_size` helper, so the two disagreed about the same fact depending on the path taken. Preserve `Exact(0)` in `with_estimated_selectivity` itself, which is only called from `FilterExec`, and drop `scale_byte_size` now that it is subsumed. `Inexact(0)` is untouched: an estimate of zero is not a proof of zero.
github-actions
Bot
added
common
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #23936 +/- ##
========================================
Coverage 80.67% 80.68%
========================================
Files 1095 1095
Lines 372460 372576 +116
Branches 372460 372576 +116
========================================
+ Hits 300488 300615 +127
+ Misses 54045 54015 -30
- Partials 17927 17946 +19 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
tschwarzinger
approved these changes
Jul 28, 2026
Member
Author
|
cc:
|
kosiew
approved these changes
Jul 29, 2026
asolimando
added a commit
to asolimando/datafusion
that referenced
this pull request
Jul 31, 2026
…ty FilterExec output `total_byte_size` was exact only for a contradictory predicate, and the per-column `byte_size` was never exact, so an input with `num_rows: Exact(0)` produced an exact zero row count next to an inexact or `Absent` byte size. `scale_byte_size_at_rows` returns an exact zero when the filtered row count is an exact zero, which also covers the `is_infeasible` case, so that branch is no longer needed. It tests the row count, unlike the `scale_byte_size` removed in apache#23936, which tested the byte size itself.
alamb
pushed a commit
to Punisheroot/datafusion
that referenced
this pull request
Aug 4, 2026
…istinct_count and total_byte_size upon empty input (apache#24000) ## Which issue does this PR close? - Part of apache#8227 ## Rationale for this change Follow-up to apache#23936 (suggested [here](apache#23936 (comment))). apache#23936 made `FilterExec` report `num_rows: Exact(0)` for a provably empty result. Two places in the same operator still turn the same proof into an estimate: - `cap_at_rows` demotes `null_count` and `distinct_count` unconditionally - `total_byte_size` is computed after the "infeasibility branch", so a contradictory predicate (`a = 1 AND a = 2`) gives `Exact(0)` for `num_rows` and per-column `byte_size` but an inexact `total_byte_size` ## What changes are included in this PR? - `cap_at_rows` now maps an `Exact(0)` row bound to `Exact(0)` - `statistics_helper` reports `total_byte_size: Exact(0)` for a contradictory predicate instead of routing it through the selectivity estimate ## Are these changes tested? Yes, both by extending `test_filter_statistics_preserves_exactly_empty_input` with two assertions. Both assertions were confirmed to fail without the respective fix. No sqllogictest baselines change. ## Are there any user-facing changes? No breaking changes and no signature changes. `FilterExec` reports exact rather than inexact zeros for these statistics, visible in `EXPLAIN` output that shows statistics. ---- Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.
kosiew
pushed a commit
to kosiew/datafusion
that referenced
this pull request
Aug 12, 2026
…tion (apache#23936) ## Which issue does this PR close? - Part of apache#8227. ## Rationale for this change `Precision::with_estimated_selectivity` always demoted to inexact, so `Exact(0)` became `Inexact(0)`. Scaling zero by any selectivity is still exactly zero: filtering a provably empty input cannot produce rows. Because operators widen exact to inexact, emptiness could not propagate up a plan through statistics at all, it was lost at the first `FilterExec`. That operator already preserved the exact zero for a contradictory predicate and for column byte sizes, so the same fact came out exact or inexact depending on the path taken. This is the same reasoning as apache#23670, which stops `FileScanConfig::statistics()` demoting an exact zero when a filter is present, applied one level up. The two are independent, but together they let a proof of emptiness survive from the scan through the filter. ## What changes are included in this PR? - `with_estimated_selectivity` returns an exact zero unchanged. `Inexact(0)` is untouched: an estimate of zero is not a proof of zero. - `scale_byte_size` in `filter.rs` is removed, subsumed by the above. ## Are these changes tested? Yes. A unit test for the method, and a `FilterExec` test that a satisfiable predicate over an exactly empty input keeps `num_rows`, `total_byte_size` (proving that `scale_byte_size` can be safely removed) and column `byte_size` exact. `test_filter_statistics_empty_input_equality_ndv_zero` asserted `Inexact(0)` and now asserts `Exact(0)`, which is expected. ## Are there any user-facing changes? No breaking changes. `Precision::with_estimated_selectivity` is public and keeps its signature, but its documented contract changes: it previously stated it would always return inexact statistics, and it now preserves an exact zero. Callers relying on the old wording will see `Exact(0)` where they saw `Inexact(0)`. This is an improvement and could be arguably considered a bug-fix, so I consider this contract change justified. `FilterExec` over a provably empty input therefore reports exact statistics, visible in `EXPLAIN` output that shows statistics. ---- Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.
kosiew
pushed a commit
to kosiew/datafusion
that referenced
this pull request
Aug 12, 2026
…istinct_count and total_byte_size upon empty input (apache#24000) ## Which issue does this PR close? - Part of apache#8227 ## Rationale for this change Follow-up to apache#23936 (suggested [here](apache#23936 (comment))). apache#23936 made `FilterExec` report `num_rows: Exact(0)` for a provably empty result. Two places in the same operator still turn the same proof into an estimate: - `cap_at_rows` demotes `null_count` and `distinct_count` unconditionally - `total_byte_size` is computed after the "infeasibility branch", so a contradictory predicate (`a = 1 AND a = 2`) gives `Exact(0)` for `num_rows` and per-column `byte_size` but an inexact `total_byte_size` ## What changes are included in this PR? - `cap_at_rows` now maps an `Exact(0)` row bound to `Exact(0)` - `statistics_helper` reports `total_byte_size: Exact(0)` for a contradictory predicate instead of routing it through the selectivity estimate ## Are these changes tested? Yes, both by extending `test_filter_statistics_preserves_exactly_empty_input` with two assertions. Both assertions were confirmed to fail without the respective fix. No sqllogictest baselines change. ## Are there any user-facing changes? No breaking changes and no signature changes. `FilterExec` reports exact rather than inexact zeros for these statistics, visible in `EXPLAIN` output that shows statistics. ---- Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Precision::with_estimated_selectivityalways demoted to inexact, soExact(0)becameInexact(0). Scaling zero by any selectivity is still exactly zero: filtering a provably empty input cannot produce rows.Because operators widen exact to inexact, emptiness could not propagate up a plan through statistics at all, it was lost at the first
FilterExec. That operator already preserved the exact zero for a contradictory predicate and for column byte sizes, so the same fact came out exact or inexact depending on the path taken.This is the same reasoning as #23670, which stops
FileScanConfig::statistics()demoting an exact zero when a filter is present, applied one level up. The two are independent, but together they let a proof of emptiness survive from the scan through the filter.What changes are included in this PR?
with_estimated_selectivityreturns an exact zero unchanged.Inexact(0)is untouched: an estimate of zero is not a proof of zero.scale_byte_sizeinfilter.rsis removed, subsumed by the above.Are these changes tested?
Yes. A unit test for the method, and a
FilterExectest that a satisfiable predicate over an exactly empty input keepsnum_rows,total_byte_size(proving thatscale_byte_sizecan be safely removed) and columnbyte_sizeexact.test_filter_statistics_empty_input_equality_ndv_zeroassertedInexact(0)and now assertsExact(0), which is expected.Are there any user-facing changes?
No breaking changes.
Precision::with_estimated_selectivityis public and keeps its signature, but its documented contract changes: it previously stated it would always return inexact statistics, and it now preserves an exact zero. Callers relying on the old wording will seeExact(0)where they sawInexact(0). This is an improvement and could be arguably considered a bug-fix, so I consider this contract change justified.FilterExecover a provably empty input therefore reports exact statistics, visible inEXPLAINoutput that shows statistics.Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.