Improve hash-join liveness and allocation paths on DF55.1 - #95
Merged
Vedin merged 3 commits intoSep 16, 2026
Merged
Conversation
Publish complete CollectLeft build filters without waiting for unpolled probes. Add checked Decimal128 direct-map keys, reuse admitted map capacity, and update primitive equality masks in place. Includes focused liveness, join-oracle, memory-budget and mask regressions.
Remove the unused physical-plan num-traits dependency. Probe real fallback rows instead of asserting the number of hash buckets. Full df55-r4 gate passes: fmt, all-feature Clippy, 10572 extended Rust tests and 508 SLT files, 10381 forced-collision tests, three unchanged SMJ RSS guards, cargo machete and unmodified rust_lint.sh.
osipovartem
approved these changes
Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No separate issue. Selected interactive-execution follow-up, based on DF55.1 and
the separately merged SMJ fix in #94. This PR does not reintroduce the SMJ patch
or carry the local research journal.
Rationale for this change
A complete CollectLeft build could wait for probe streams that a parent has not
polled, preventing query progress. Small-key joins and primitive group comparisons
also incurred avoidable allocation/reservation work. These are general execution
paths, not table-name or benchmark-query special cases.
What changes are included in this PR?
keep the all-build-partitions barrier for Partitioned mode.
domain fits i64. Reject out-of-domain probes before conversion; do not materialize
a narrowed Arrow array or allow truncation aliases.
optional direct map. Fall back on ResourcesExhausted only; propagate other errors.
null handling and float canonicalization.
Remove the now-unused physical-plan
num-traitsdependency and its directlockfile entry. No new dependencies, SQL API or optimizer-default changes.
The lockfile also updates rustls to 0.23.45 and its required existing TLS
dependencies after the newly triggered audit found
RUSTSEC-2026-0285.
This is a narrow security follow-up, not a performance claim or an audit waiver.
Are these changes tested?
Fresh full verification
df55-r5, based on 8775702 (DF55.1), covers all fourruntime files, the test/dependency follow-up and the patched TLS lockfile:
recursive_protection,parquet_encryption): 10,572 Rust tests PASS, eight existing
ignored tests; 508/508 SQL-logic files PASS.
85.0 / 84.8 / 85.0 MiB query RSS deltas against 152.6 MiB. These are deltas,
not absolute process peak RSS. No test or threshold was skipped or weakened.
seven pre-existing ignored tests. The initial CI run exposed a test confusing
hash bucket count with row count; it now probes both keys, a miss and a
duplicate through the actual general-map lookup and retains memory checks.
Sep14 RustSec advisory database. Exactly the repository's two existing
quick-xml exclusions (RUSTSEC-2026-0194 and RUSTSEC-2026-0195) are retained;
no new exclusion was introduced for rustls.
Focused regressions cover unpolled CollectLeft probes and idempotent publication,
Decimal128 boundaries/nulls/duplicates/wide probes, all ten join types in both
partition modes against the general-map oracle, tight-budget admission/fallback,
and primitive mask/float equivalence.
The first local extended run lacked test-data submodule contents and failed.
All three pinned fixture submodules were then initialized; the entire gate was
rerun successfully. The failed log is retained locally.
Local dependency verification used the application's Arrow59.2 fork pin
cabd25a16b653a6ff6eafbc016545cde471c6c3e via a test-only Cargo patch.
There are no Cargo patches or test-runner changes in this PR.
Performance evidence is intentionally scoped: the prior DF55.0 research compared
six primitive-group Criterion cases over three paired runs (10.8–13.16% lower
times); its representative larger query screen was 1.11% slower and was retained.
The CollectLeft fix is a deterministic liveness correction, not a speedup computed
from a timed-out baseline. Fresh integrated Rustice/current-main large-suite and
SPCS qualification is tracked with the single application integration PR; no old
measurements are represented as DF55.1 results.
The initial remote head also reproduced the two failures already accepted for
#94: the removed MinIO image used by four CLI integration tests, and the missing
generated
array_reducedocumentation. The relevant files are unchanged from8775702. These are retained exceptions, not new test skips or a waiver for the
new checks: the unused-dependency and hash-collision regressions were fixed,
as was the newly exposed rustls advisory. The previous remote head completed
35 successful checks; the two accepted baseline failures and the now-patched
audit failure are not represented as green checks on that head.
All current-head checks must complete before merging.
Are there any user-facing changes?
Queries with a complete shared build can progress without waiting for unpolled
probe streams, and eligible joins/group comparisons avoid unnecessary work.
Result semantics, memory-limit errors and explicit general-map fallback remain
covered. No public API break and no benchmark-specific defaults.