Skip to content

Support predicate subqueries in projections - #59

Merged
osipovartem merged 2 commits into
embucket-sync-df55.0.0from
snowflake-in-subquery-projection
Sep 4, 2026
Merged

Support predicate subqueries in projections#59
osipovartem merged 2 commits into
embucket-sync-df55.0.0from
snowflake-in-subquery-projection

Conversation

@osipovartem

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

  • Closes a Snowflake compatibility gap in Rustice for scalar IN/EXISTS projections.

Rationale for this change

DataFusion decorrelates predicate subqueries in filters, but leaves IN, NOT IN, EXISTS, and NOT EXISTS in projection expressions for the physical planner, which rejects them. Projected IN must also preserve SQL three-valued logic for NULL values and empty subqueries.

What changes are included in this PR?

  • Decorrelate predicate subqueries embedded in projection expressions.
  • Represent projected IN with mark joins for equality matches, NULL presence, and non-empty input so FALSE and UNKNOWN remain distinct.
  • Preserve the original projected column name.
  • Keep the existing semi/anti join path for top-level filter predicates unchanged.

The correctness-first IN rewrite uses three mark joins. A future optimization can combine those facts in one physical operator without changing the logical contract introduced here.

Are these changes tested?

Yes. Added optimizer-plan and SQL logic coverage for:

  • IN and NOT IN with an empty subquery
  • matches, misses, NULL left-hand values, and NULL values in the subquery
  • multiple predicate subqueries in one projection
  • correlated IN with match, miss, NULL, and empty per-row results
  • projected EXISTS and NOT EXISTS

Local checks:

  • cargo +1.97.0 test -p datafusion-optimizer --lib
  • focused datafusion-sqllogictest files
  • cargo +1.97.0 clippy --all-targets --all-features -- -D warnings
  • cargo +1.97.0 fmt --all -- --check

The extended workspace suite reached core_integration. Its remaining local failures were environmental: fixture symlinks exposed absolute paths in snapshots, and the known memory-limit test started above its fixed 152.6 MB limit (158.2 MB baseline RSS).

Are there any user-facing changes?

Queries can now return IN, NOT IN, EXISTS, and NOT EXISTS expressions from the SELECT list, including correlated forms and SQL-compliant NULL behavior. There are no public API changes.

@osipovartem
osipovartem merged commit 7c4f778 into embucket-sync-df55.0.0 Sep 4, 2026
42 of 69 checks passed
@osipovartem
osipovartem removed the request for review from rampage644 September 5, 2026 11:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant