Support predicate subqueries in projections - #59
Merged
osipovartem merged 2 commits intoSep 4, 2026
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
IN/EXISTSprojections.Rationale for this change
DataFusion decorrelates predicate subqueries in filters, but leaves
IN,NOT IN,EXISTS, andNOT EXISTSin projection expressions for the physical planner, which rejects them. ProjectedINmust also preserve SQL three-valued logic for NULL values and empty subqueries.What changes are included in this PR?
INwith mark joins for equality matches, NULL presence, and non-empty input so FALSE and UNKNOWN remain distinct.The correctness-first
INrewrite uses three mark joins. A future optimization can combine those facts in one physical operator without changing the logical contract introduced here.Are these changes tested?
Yes. Added optimizer-plan and SQL logic coverage for:
INandNOT INwith an empty subqueryINwith match, miss, NULL, and empty per-row resultsEXISTSandNOT EXISTSLocal checks:
cargo +1.97.0 test -p datafusion-optimizer --libdatafusion-sqllogictestfilescargo +1.97.0 clippy --all-targets --all-features -- -D warningscargo +1.97.0 fmt --all -- --checkThe extended workspace suite reached
core_integration. Its remaining local failures were environmental: fixture symlinks exposed absolute paths in snapshots, and the known memory-limit test started above its fixed 152.6 MB limit (158.2 MB baseline RSS).Are there any user-facing changes?
Queries can now return
IN,NOT IN,EXISTS, andNOT EXISTSexpressions from the SELECT list, including correlated forms and SQL-compliant NULL behavior. There are no public API changes.