fix: keep join filter columns when pushing a projection below SortMergeJoinExec

[jayzhan211]

Rationale for this change

A sort merge join with a non-equi filter panics (index out of bounds in get_filter_columns) when the query does not select a column the filter references:

SET datafusion.optimizer.prefer_hash_join = false;
SET datafusion.execution.target_partitions = 2;
CREATE TABLE l (k VARCHAR, a INT, b INT, c INT) AS VALUES ('x', 1, 1, 1), ('y', 2, 2, 2), ('x', 3, 3, 3);
CREATE TABLE r (k VARCHAR, a INT, b INT, c INT) AS VALUES ('x', 1, 1, 1), ('y', 2, 2, 2), ('z', 3, 3, 3);
SELECT l.c, l.k, l.b, r.k, r.b, r.a FROM l RIGHT JOIN r ON l.k = r.k AND l.c >= r.c;

thread 'tokio-rt-worker' panicked at datafusion/physical-plan/src/joins/sort_merge_join/filter.rs:169:34:
index out of bounds: the len is 3 but the index is 3

The projection pushdown added in #24517 pushes the projection into the join's children but passes self.filter.clone() through unchanged. The right child no longer produces c, but the filter's ColumnIndex still points at it.

What changes are included in this PR?

SortMergeJoinExec::try_swapping_with_projection now uses the shared try_pushdown_through_join_with_column_indices helper, as HashJoinExec and NestedLoopJoinExec already do. It remaps both the on keys and the filter's column indices to the projected children, and declines the pushdown — embedding the projection in the join instead — when the filter needs a column the projection drops.

One behavioral side effect: projections containing expressions (e.g. length(w.p)) are now also embedded into the join, so the join only emits the columns they reference. This matches the existing behavior of the other join operators; the affected EXPLAIN ANALYZE expectations in sort_merge_join_spill.slt are updated accordingly (projection=[...] on the SortMergeJoinExec line).

Are these changes tested?

Yes. A regression test in joins.slt covers the panicking case (explain + results). Existing SMJ unit tests, projection tests, proto round-trip tests, and joins.slt / range_partitioning.slt / sort_merge_join_spill.slt pass.

Are there any user-facing changes?

No API changes. Plans for SortMergeJoinExec under an expression projection now show projection=[...] on the join node.

[jayzhan211]

try_pushdown_through_join_with_column_indices includes what we have for the old code

[jayzhan211]

The overall idea of this PR

[codecov-commenter]

Codecov Report

❌ Patch coverage is 92.59259% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.44%. Comparing base (e14bd12) to head (c1bf726).
⚠️ Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
...on/physical-plan/src/joins/sort_merge_join/exec.rs	92.59%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24593      +/-   ##
==========================================
+ Coverage   81.39%   81.44%   +0.04%     
==========================================
  Files        1118     1119       +1     
  Lines      398680   400202    +1522     
  Branches   398680   400202    +1522     
==========================================
+ Hits       324517   325954    +1437     
+ Misses      55196    55192       -4     
- Partials    18967    19056      +89     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kosiew]

@jayzhan211, thanks for working on this. The shared schema-aware join helper looks like a good fit here, and the regression coverage addresses the reported SortMergeJoin projection and filter issue. I left one optional suggestion for a little more execution-level coverage. Nothing blocking from my side.

[kosiew]

Nice improvement here. One optional thought: could we add an execution-level unit test for the successful remapping path, where the filter columns are retained but reordered? The SQL regression covers the fallback with an embedded projection, while the current unit test verifies the rewritten indices without executing the returned SortMergeJoinExec. Executing it would give us a bit more confidence that the remapped JoinFilter schema and columns are consumed correctly. The broader SortMergeJoin spill SQL tests already exercise filtered projected joins, so I see this as a small coverage improvement rather than something that should block this PR.

[jayzhan211]

Thanks for your review, test added!