Skip to content

Improve hash-join liveness and allocation paths on DF55.1 - #95

Merged
Vedin merged 3 commits into
embucket-sync-df55.0.0from
autoresearch/interactive-df55-integration
Sep 16, 2026
Merged

Vedin merged 3 commits into
embucket-sync-df55.0.0from
autoresearch/interactive-df55-integration

Conversation

@Vedin

@Vedin Vedin commented Sep 15, 2026

Copy link
Copy Markdown

Which issue does this PR close?

No separate issue. Selected interactive-execution follow-up, based on DF55.1 and
the separately merged SMJ fix in #94. This PR does not reintroduce the SMJ patch
or carry the local research journal.

Rationale for this change

A complete CollectLeft build could wait for probe streams that a parent has not
polled, preventing query progress. Small-key joins and primitive group comparisons
also incurred avoidable allocation/reservation work. These are general execution
paths, not table-name or benchmark-query special cases.

What changes are included in this PR?

  • Publish a CollectLeft dynamic filter after its one complete shared build report;
    keep the all-build-partitions barrier for Partitioned mode.
  • Use checked Decimal128 key conversion in the direct map when the complete build
    domain fits i64. Reject out-of-domain probes before conversion; do not materialize
    a narrowed Arrow array or allow truncation aliases.
  • Reuse the general hash table's already admitted capacity when selecting the
    optional direct map. Fall back on ResourcesExhausted only; propagate other errors.
  • Update primitive equality masks in place, preserving previous masks, tail bits,
    null handling and float canonicalization.

Remove the now-unused physical-plan num-traits dependency and its direct
lockfile entry. No new dependencies, SQL API or optimizer-default changes.

The lockfile also updates rustls to 0.23.45 and its required existing TLS
dependencies after the newly triggered audit found
RUSTSEC-2026-0285.
This is a narrow security follow-up, not a performance claim or an audit waiver.

Are these changes tested?

Fresh full verification df55-r5, based on 8775702 (DF55.1), covers all four
runtime files, the test/dependency follow-up and the patched TLS lockfile:

  • cargo fmt --all / --check: PASS.
  • All-target/all-feature Clippy with -D warnings: PASS.
  • Full extended workspace gate (avro,json,backtrace,extended_tests,
    recursive_protection,parquet_encryption): 10,572 Rust tests PASS, eight existing
    ignored tests; 508/508 SQL-logic files PASS.
  • Original unchanged SMJ RSS guard, three additional isolated executions:
    85.0 / 84.8 / 85.0 MiB query RSS deltas against 152.6 MiB. These are deltas,
    not absolute process peak RSS. No test or threshold was skipped or weakened.
  • Unmodified dev/rust_lint.sh: PASS, including docs and repository-wide lints.
  • Full forced-hash-collision workspace test matrix: 10,381 Rust tests PASS,
    seven pre-existing ignored tests. The initial CI run exposed a test confusing
    hash bucket count with row count; it now probes both keys, a miss and a
    duplicate through the actual general-map lookup and retains memory checks.
  • cargo machete --with-metadata: PASS after removing the unused dependency.
  • Security audit of the canonical publication lockfile: PASS against a fresh
    Sep14 RustSec advisory database. Exactly the repository's two existing
    quick-xml exclusions (RUSTSEC-2026-0194 and RUSTSEC-2026-0195) are retained;
    no new exclusion was introduced for rustls.
  • Source hashes and test executable hash checked before/after verification.

Focused regressions cover unpolled CollectLeft probes and idempotent publication,
Decimal128 boundaries/nulls/duplicates/wide probes, all ten join types in both
partition modes against the general-map oracle, tight-budget admission/fallback,
and primitive mask/float equivalence.

The first local extended run lacked test-data submodule contents and failed.
All three pinned fixture submodules were then initialized; the entire gate was
rerun successfully. The failed log is retained locally.

Local dependency verification used the application's Arrow59.2 fork pin
cabd25a16b653a6ff6eafbc016545cde471c6c3e via a test-only Cargo patch.
There are no Cargo patches or test-runner changes in this PR.

Performance evidence is intentionally scoped: the prior DF55.0 research compared
six primitive-group Criterion cases over three paired runs (10.8–13.16% lower
times); its representative larger query screen was 1.11% slower and was retained.
The CollectLeft fix is a deterministic liveness correction, not a speedup computed
from a timed-out baseline. Fresh integrated Rustice/current-main large-suite and
SPCS qualification is tracked with the single application integration PR; no old
measurements are represented as DF55.1 results.

The initial remote head also reproduced the two failures already accepted for
#94: the removed MinIO image used by four CLI integration tests, and the missing
generated array_reduce documentation. The relevant files are unchanged from
8775702. These are retained exceptions, not new test skips or a waiver for the
new checks: the unused-dependency and hash-collision regressions were fixed,
as was the newly exposed rustls advisory. The previous remote head completed
35 successful checks; the two accepted baseline failures and the now-patched
audit failure are not represented as green checks on that head.
All current-head checks must complete before merging.

Are there any user-facing changes?

Queries with a complete shared build can progress without waiting for unpolled
probe streams, and eligible joins/group comparisons avoid unnecessary work.
Result semantics, memory-limit errors and explicit general-map fallback remain
covered. No public API break and no benchmark-specific defaults.

Publish complete CollectLeft build filters without waiting for unpolled probes. Add checked Decimal128 direct-map keys, reuse admitted map capacity, and update primitive equality masks in place. Includes focused liveness, join-oracle, memory-budget and mask regressions.
Remove the unused physical-plan num-traits dependency. Probe real fallback rows instead of asserting the number of hash buckets.

Full df55-r4 gate passes: fmt, all-feature Clippy, 10572 extended Rust tests and 508 SLT files, 10381 forced-collision tests, three unchanged SMJ RSS guards, cargo machete and unmodified rust_lint.sh.
@Vedin
Vedin merged commit e52e5cc into embucket-sync-df55.0.0 Sep 16, 2026
36 of 39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants