[FLINK-40131][docs] Document LATERAL SNAPSHOT join by fhueske · Pull Request #28737 · apache/flink

fhueske · 2026-07-14T08:29:47Z

What is the purpose of the change

Add documentation for FLIP-579: LATERAL SNAPSHOT join

Brief change log

Add docs for FLIP-579: LATERAL SNAPSHOT join

Verifying this change

docs only change

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? n/a

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Code 2.1.200 (Opus 4.8)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

flinkbot · 2026-07-14T08:35:39Z

CI report:

bdde889 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

davidradl · 2026-07-15T08:31:41Z

+
+{{< label Streaming >}}
+
+A `LATERAL SNAPSHOT` join is a *stream enrichment* join that augments an append-only table (the *probe* side) with the current state of an updating table (the *build* side).


have we defined what the probe side is - if so we should include a link to the definition). If not we should define the names for each side before referencing this.

davidradl · 2026-07-15T08:34:16Z

+
+A `LATERAL SNAPSHOT` join is a *stream enrichment* join that augments an append-only table (the *probe* side) with the current state of an updating table (the *build* side).
+Unlike an [event-time temporal join](#event-time-temporal-join), it does not correlate each probe-side row with a specific historical version of the build side.
+Instead, every probe-side row is joined with the build-side state that is current at the time the row is processed, similar to a [processing-time temporal join](#processing-time-temporal-join), but with well-defined behavior at query start-up.


I am curious what the other joins not having well-defined behavior means. It would be worth mentioning this here to help show the value of this join.

This refers to the note about the processing-time temporal table join:

The reason why the FOR SYSTEM_TIME AS OF syntax used in temporal join with latest version of any table/view is not support is only the semantic consideration, because the join processing for left stream doesn’t wait for the complete snapshot of temporal table, this may mislead users in production environment.

I'll expand the point a bit.

davidradl · 2026-07-15T08:43:36Z

+        [ load_completed_time        => <timestamp_ltz>, ]
+        [ load_completed_idle_timeout => <interval>, ]
+        [ state_ttl                  => <interval> ])) AS s
+ON probe_table.key = s.key


it would be worth expanding the example so we can see where the .keys are defined for each side.

these are just random columns that both tables are joined on. Columns references by equality join predicates are often called join keys. They don't have to be defined as primary or foreign keys, but it's quite common that they are.
I'll rename the columns to make this less confusing.

davidradl · 2026-07-15T08:45:59Z

+
+The `LATERAL SNAPSHOT` join is designed for scenarios where the other temporal joins are a poor fit:
+
+- The build side updates **infrequently**. An event-time temporal join relies on continuous build-side watermarks to emit results, so a build side that rarely advances its watermark stalls the join and lets probe-side state accumulate. A `LATERAL SNAPSHOT` join does not stall when the build side goes idle.


when you say infrequently. It would be worth defining how the frequency is defined

Fair point, I'll rephrase that. It really depends on the specific use case (how much latency is acceptable, how much data is being processed, etc.).

davidradl · 2026-07-15T08:46:56Z

+
+- The build side updates **infrequently**. An event-time temporal join relies on continuous build-side watermarks to emit results, so a build side that rarely advances its watermark stalls the join and lets probe-side state accumulate. A `LATERAL SNAPSHOT` join does not stall when the build side goes idle.
+- The application has **low-latency** requirements that are incompatible with the watermark-induced delay of an event-time temporal join.
+- The build side has **no primary key**. Event-time and processing-time temporal joins require the build-side primary key to appear in the equi-join condition; a `LATERAL SNAPSHOT` join does not.


does this mean the probe side has a primary key?

No, this statement doesn't say anything about the probe side. It just compares the build side requirement of the lateral snapshot join with the requirements of both temporal joins.
There is no PK requirement for the probe side (but there isn't one for the temporal joins either).

davidradl · 2026-07-15T08:51:56Z

+ON orders.currency = rates.currency;
+```
+
+The probe side must be an append-only table, and its time attributes are preserved in the output. The build side may be an updating table and must declare a watermark. The result is append-only. A build-side rowtime attribute that is projected into the output is materialized as a regular `TIMESTAMP` and is no longer a time attribute. Probe-side watermarks are forwarded downstream in join phase; build-side watermarks are consumed internally and are not propagated.


I suggest a picture (sequence diagram or the like) showing the time line including

the build side and probe side

watermarks (the internal build one and the probe one)

how the select is triggered including the load phase

maybe an example input and output

davidradl · 2026-07-15T08:53:16Z

+ON probe_table.key = s.key
+```
+
+The `SNAPSHOT` function accepts the following arguments:


shouldn't the SNAPSHOT function be documented in the SQL functions section

The SNAPSHOT function can only be used in a LATERAL context and not as a stand-alone table function.
But we can add a new section "Table Functions" to https://nightlies.apache.org/flink/flink-docs-release-2.3/docs/sql/functions/built-in-functions/ and list it there with a pointer to the this section in the join docs.

davidradl · 2026-07-15T08:53:58Z

+| Argument | Type | Required | Description |
+| --- | --- | --- | --- |
+| `input` | TABLE | yes | The build-side table. It may use any [changelog mode]({{< ref "docs/sql/reference/queries/changelog" >}}) (inserts, updates, and deletes) and must declare a [watermark]({{< ref "docs/concepts/sql-table-concepts/time_attributes" >}}#event-time). |
+| `load_completed_condition` | STRING | no | Determines when the initial load phase completes. One of `'compile_time'` (default) or `'user_time'`. With `'compile_time'`, the load phase completes once the build-side watermark reaches the wall-clock time at which the query was compiled. With `'user_time'`, it completes once the build-side watermark reaches the explicit `load_completed_time`. |


I think a table here would be good describing each option and why you would use them . I think more description as to what exactly compile time and user time means would help.

does the load phase happen more than once?What other phases are there?

Do you mean a nested table in the arguments table?
Not sure how well this is supported by the docs framework.
TBH, I also don't know what else to add to the description of both conditions. compile_time is just sets the load_completed_time to the current system time, while user_time requires users to explicitly set it.

But I noticed that the table discusses about LOAD phase before it had been discussed.
So, I'll move the paragraph about the internals of the join up above the syntax.

[FLINK-40131][docs] Document LATERAL SNAPSHOT join

bdde889

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

davidradl reviewed Jul 15, 2026

View reviewed changes

fhueske mentioned this pull request Jul 16, 2026

[FLINK-40158][table-planner] Support LATERAL SNAPSHOT join in batch mode #28763

Open

1 task


		{{< label Streaming >}}

		A `LATERAL SNAPSHOT` join is a stream enrichment join that augments an append-only table (the probe side) with the current state of an updating table (the build side).


		The `LATERAL SNAPSHOT` join is designed for scenarios where the other temporal joins are a poor fit:

		- The build side updates infrequently. An event-time temporal join relies on continuous build-side watermarks to emit results, so a build side that rarely advances its watermark stalls the join and lets probe-side state accumulate. A `LATERAL SNAPSHOT` join does not stall when the build side goes idle.

Uh oh!

Conversation

fhueske commented Jul 14, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidradl Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidradl Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

flinkbot commented Jul 14, 2026 •

edited

Loading

davidradl Jul 15, 2026 •

edited

Loading

davidradl Jul 15, 2026 •

edited

Loading