Conversation
Flink's streaming planner is not a stable API, so targeting more than one release line normally means duplicating the planner integration. Measuring the actual divergence first showed it is far smaller than that: across everything StreamFusion links against, the 2.1 and 2.2 planners disagree on a handful of members, and most of those are renames that erase to the same bytecode. What genuinely differs is narrow — a watermark push-down argument added in 2.2, two changelog-normalize predicates that only exist in 2.2, a state-backend accessor absent from 2.1, and one helper whose Scala and Java collection return types are indistinguishable after erasure and so only surface when compiling. Concentrate that divergence in a single class compiled once per line and selected by a build profile, with the values it hands back carried in line-neutral holders. The rest of the tree compiles unchanged for either target, which keeps the seam reviewable and stops version drift from leaking into the matchers and execution nodes. The profile also pins the dependencies that track the Flink line rather than our own release cadence: the Kafka connector and the Delta connector publish a build per line, and Flink's Protobuf format generates against a different Protobuf runtime major on each, which fails at runtime inside Flink's own deserializer rather than at build time.
The loader shadows a private Flink class, so it has to match that class's exact signatures — and the two supported lines disagree on one. The accessor exposing the component classloader was narrowed to a concrete type in 2.2, so each line's callers are compiled against a different descriptor. Declaring either type alone turns a batch adaptive-join path into a missing-method failure on the other line. Declare the wider type in a base class and narrow it in the shim so the compiler emits a bridge, leaving one class that answers both descriptors. That keeps the shadow a signature detail rather than a second copy of the file to maintain. The fail-closed version whitelist is now compiled per line as well, so a build admits only the patch releases it was actually validated against instead of inheriting a list that happens to be true for a different target.
The suite installs StreamFusion by matching Flink classes by name at load time. A miss has always been silent: the suite runs stock Flink end to end, reports every test green, and proves nothing. That is tolerable while one Flink release is targeted and the names are known good, but it becomes actively misleading the moment the suite runs against more than one line, where a renamed or relocated injection point is exactly the failure being looked for. Record whether the injection point was instrumented and whether it was actually entered, then abort at shutdown if Flink's planner factory was loaded without it. A run that never installed the engine now fails loudly instead of passing.
The suite harness pinned one release throughout: the tag it cloned, the connector release it resolved, and the profile StreamFusion itself was built with were all fixed independently. Running it against another line therefore risked the worst possible outcome — an engine compiled for one planner exercised against another, producing either a confusing failure or, worse, a green run that measured a mismatched pair. Derive all of it from the requested Flink version instead, so choosing a line selects the source tag, the matching connector release, and the build profile together. The harness can now be pointed at any supported line without editing it.
Parity tests assert that a query produces identical results on stock Flink and on the native plan. That assertion is undefined when Flink itself cannot plan the query: an older line rejects a left-joined ordinality unnest outright, in its own optimizer, before the engine is ever installed. The failure looks like a StreamFusion defect and is not one. Gate such a case on the Flink release that fixed it, reading the version from the running Flink rather than a build property so a test can never be excused on a version it did not actually load. The exclusion is deliberately narrow: it covers only behavior the host lacks, never a coverage gap of ours, which must keep failing. Newer lines continue to run the case with nothing skipped.
Casts are evaluated through Flink's own cast rules, and the code generated for a non-nullable input dereferences the value with no null guard. Priming that code with a null therefore aborted operator startup rather than surfacing the per-row error the query expects, turning every cast of a non-nullable string into a job failure. Prime with a value the declared input type admits instead. This was invisible locally because the harnesses only ever exercised nullable columns; non-nullable types reach the engine through SQL literals, so only the upstream suite covered the shape that broke.
Day-time intervals cross the columnar boundary in two shapes. The engine canonicalises on integer milliseconds, but the query engine emits a native interval array whenever an expression's result type is an interval rather than a timestamp. The native side already absorbed both encodings; the Java boundary did not, so an interval-typed expression failed at runtime instead of returning a value. Read the native encoding as milliseconds at the boundary, mirroring how the TIME type already absorbs several Arrow encodings behind one SQL type. Boundary errors now name the Arrow vector and its type alongside the SQL type: dispatch happens on the vector, so the SQL type alone cannot explain a mismatch, and that blind spot hid this bug and an earlier one in the same code.
Flink validates a forced delta-join strategy after our pass has run, and raises an error only when an ordinary join survived optimization. Substituting that join away erased the very evidence the validation looks for, so a query the host intends to refuse ran silently instead. Any acceleration that changes whether a query is legal is a correctness bug, not a coverage question. Decline such plans wholesale. The condition mirrors the host's exactly rather than declining whenever the strategy is forced, so acceleration is retained for every plan the validation would have passed, including one that mixes a delta join with joins we do accelerate.
The upstream-suite harness hardcoded a single Calcite version while Flink pins one per release line, so on the older line the engine was built against a parser the host never uses and whole suite modes failed before a single test ran. Take the version from the Flink checkout under test rather than restating it, and fail loudly if it cannot be found. The generated classpath is named per version as well, so a reused build can no longer silently run one line's tests against the other line's jars.
The engine builds against two Flink lines, but every deployable module published one artifact name for both. Two builds with different bytecode claimed identical coordinates, so they overwrote each other in a local repository and only one could ever be published. A deployment had no way to ask for the line it runs on, and the image build silently paired one line's base image with the other line's payload. Name deployables after the line they target, following how the rest of the Flink ecosystem publishes per-line artifacts. Release, packaging, install, and image tooling now take the line explicitly rather than assuming the default, so a bundle holds one line's jars and an image cannot be assembled from a mismatched pair. The internal module that compiles the shared tree and hosts the test suite is no longer published: it was never part of a deployment, and one unsuffixed name covering two lines' bytecode is the same collision.
Flanderzz
marked this pull request as draft
September 8, 2026 07:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flink's streaming planner is not a stable API, so targeting more than one
release line normally means duplicating the planner integration. Measuring the
actual divergence first showed it is far smaller than that: across everything
StreamFusion links against, the 2.1 and 2.2 planners disagree on a handful of
members, and most of those are renames that erase to the same bytecode. What
genuinely differs is narrow — a watermark push-down argument added in 2.2, two
changelog-normalize predicates that only exist in 2.2, a state-backend accessor
absent from 2.1, and one helper whose Scala and Java collection return types are
indistinguishable after erasure and so only surface when compiling.
Concentrate that divergence in a single class compiled once per line and
selected by a build profile, with the values it hands back carried in
line-neutral holders. The rest of the tree compiles unchanged for either target,
which keeps the seam reviewable and stops version drift from leaking into the
matchers and execution nodes.
The profile also pins the dependencies that track the Flink line rather than our
own release cadence: the Kafka connector and the Delta connector publish a build
per line, and Flink's Protobuf format generates against a different Protobuf
runtime major on each, which fails at runtime inside Flink's own deserializer
rather than at build time.