-
Notifications
You must be signed in to change notification settings - Fork 14k
[FLINK-40131][docs] Document LATERAL SNAPSHOT join #28737
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -295,6 +295,101 @@ WHERE | |
| - SQL 中可以定义 temporal table DDL,但不能定义 temporal table 函数; | ||
| - temporal table DDL 和 temporal table function 都支持 temporal join 版本表,但只有 temporal table function 可以 temporal join 任何表/视图的最新版本(即"处理时间 Temporal Join")。 | ||
|
|
||
| LATERAL SNAPSHOT Join | ||
| -------------- | ||
|
|
||
| {{< label Streaming >}} {{< label Batch >}} | ||
|
|
||
| A `LATERAL SNAPSHOT` join is a *stream enrichment* join that augments an append-only table with the current state of an updating table. | ||
| As in the [temporal joins](#temporal-joins), the enriched (left) input is called the *probe side* and the enriching (right) input is called the *build side*. | ||
| Every probe-side row is joined with the build-side state that is current at the time the row is processed. | ||
|
|
||
| This is the same mode of operation as the [processing-time temporal join](#processing-time-temporal-join), which also joins each probe-side row against the latest build-side state. | ||
| However, the processing-time temporal join has been disabled for Flink SQL due to a fundamental limitation. | ||
| The processing-time temporal join starts joining immediately when the query is started: early probe-side rows are joined against whatever build-side rows happen to have been loaded so far. | ||
| Right after query start-up, the build side is typically still incomplete, so early probe-side rows may be enriched with missing or stale data, and the result depends on the order in which the two inputs are read. | ||
| A `LATERAL SNAPSHOT` join addresses this problem by first loading the build side up to a well-defined point in time (the *load phase*) before it starts joining (the *join phase*). | ||
|
|
||
| **When to use it** | ||
|
|
||
| The `LATERAL SNAPSHOT` join is designed for enrichment scenarios where the other temporal joins are a poor fit: | ||
|
|
||
| - **The build side does not receive continuous updates.** An event-time temporal join only emits a joined row once the combined watermark of both inputs has passed the event time of the probe-side row. A build side that does not continuously produce records, and therefore does not advance its watermark, stalls the join and lets probe-side state accumulate. A `LATERAL SNAPSHOT` join keeps making progress even when the build side is idle. | ||
| - **Low latency is required.** An event-time temporal join holds back probe-side rows until the watermark catches up, which adds latency. After the initial load phase, the `LATERAL SNAPSHOT` join immediately joins a probe-side row when it arrives. | ||
| - **The build side has no primary key.** Event-time and processing-time temporal joins require the build side to have a primary key that appears in the equi-join condition. A `LATERAL SNAPSHOT` join has no such requirement; neither the probe side nor the build side needs a primary key. | ||
|
|
||
| **How the join works** | ||
|
|
||
| During the *load phase*, the operator accumulates the build-side changes into state until the load-completion condition is met, without emitting any results yet. This lets the build side load its initial state so that early probe-side rows are not joined against an incomplete build side. Probe-side rows that arrive during the load phase are buffered. The load phase completes when one of the following occurs: | ||
|
|
||
| - the build-side watermark reaches a configured `load_completed_time`. This time is either explicitly set by the user (`load_completed_condition => 'user_time'`) or automatically set to the wall-clock time when the query is compiled (`load_completed_condition => 'compile_time'`), or | ||
| - as a fallback, the `load_completed_idle_timeout` elapses in processing time without the build-side watermark advancing (which handles build sides that become idle during start-up). | ||
|
|
||
| When the load phase completes, the operator transitions to the *join phase*: all buffered probe-side rows are joined against the current build-side state and emitted. | ||
| From then on, each probe-side row is joined and emitted immediately against the build-side state that is current at that moment. | ||
| Build-side updates continue to be applied to the state and become visible to subsequent probe-side rows. | ||
|
|
||
| **Syntax** | ||
|
|
||
| The build side is wrapped in the `SNAPSHOT` table function inside a `LATERAL TABLE` clause. The outer (probe-side) table must be an append-only table. | ||
| Both `INNER JOIN` and `LEFT [OUTER] JOIN` are supported. The join requires at least one conjunctive equality predicate; additional non-equi predicates are allowed in the `ON` clause. | ||
|
|
||
| ```sql | ||
| SELECT [column_list] | ||
| FROM probe_table | ||
| [LEFT] JOIN LATERAL TABLE( | ||
| SNAPSHOT( | ||
| input => TABLE build_table, | ||
| [ load_completed_condition => <'compile_time' | 'user_time'>, ] | ||
| [ load_completed_time => <timestamp_ltz>, ] | ||
| [ load_completed_idle_timeout => <interval>, ] | ||
| [ state_ttl => <interval> ])) AS s | ||
| ON probe_table.col = s.col | ||
| ``` | ||
|
|
||
| The `SNAPSHOT` function accepts the following arguments: | ||
|
|
||
| | Argument | Type | Required | Description | | ||
| | --- | --- | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `input` | TABLE | yes | The build-side table. It may use any changelog mode (inserts, updates, and deletes). In streaming mode it must declare a [watermark]({{< ref "docs/concepts/sql-table-concepts/time_attributes" >}}#event-time). | | ||
| | `load_completed_condition` | STRING | no | Determines when the initial load phase completes. One of `'compile_time'` (default) or `'user_time'`. With `'compile_time'`, the load phase completes once the build-side watermark reaches the wall-clock time at which the query was compiled. With `'user_time'`, it completes once the build-side watermark reaches the explicit `load_completed_time`. | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think a table here would be good describing each option and why you would use them . I think more description as to what exactly compile time and user time means would help. does the load phase happen more than once?What other phases are there?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you mean a nested table in the arguments table? But I noticed that the table discusses about LOAD phase before it had been discussed. |
||
| | `load_completed_time` | TIMESTAMP_LTZ(3) | no | The build-side event time that completes the load phase. Required when `load_completed_condition` is `'user_time'` and must not be set otherwise. | | ||
| | `load_completed_idle_timeout` | INTERVAL | no | A processing-time fallback to complete the load phase. The transition to the join phase happens when the build-side watermark does not advance for more than the configured interval. | | ||
| | `state_ttl` | INTERVAL | no | Retention time for build-side state. Join keys that are not accessed within this duration become eligible for eviction. Only applied during the join phase. Defaults to the pipeline's [state TTL]({{< ref "docs/dev/table/config" >}}#table-exec-state-ttl). | | ||
|
|
||
| `load_completed_condition`, `load_completed_time`, `load_completed_idle_timeout`, and `state_ttl` only affect streaming execution and are ignored in batch mode (see **Batch mode** below). | ||
|
|
||
| **Example** | ||
|
|
||
| Reusing the `orders` and `currency_rates` tables from the [event-time temporal join](#event-time-temporal-join) example, the following query enriches each order with the conversion rate that is current when the order is processed. In contrast to the event-time temporal join, the join keeps making progress even if the rates are updated only rarely, e.g., once a day. | ||
|
|
||
| ```sql | ||
| SELECT | ||
| order_id, | ||
| price, | ||
| orders.currency, | ||
| conversion_rate, | ||
| order_time | ||
| FROM orders | ||
| JOIN LATERAL TABLE( | ||
| SNAPSHOT(input => TABLE currency_rates)) AS rates | ||
| ON orders.currency = rates.currency; | ||
| ``` | ||
|
|
||
| **Result and state characteristics** | ||
|
|
||
| The result is append-only and preserves the probe-side time attributes. A build-side rowtime attribute that is projected into the output is materialized as a regular `TIMESTAMP` and is no longer a time attribute. Probe-side watermarks are forwarded downstream during the join phase; build-side watermarks are consumed internally and are not propagated. | ||
|
|
||
| Because rows are joined against the build-side state that is current at processing time, the result is **not deterministic**. A given probe-side row may be joined with different build-side versions across different runs, depending on the relative timing of the two inputs. Probe and build-side inputs can be configured with watermark alignment to keep the two inputs roughly aligned on event time, so that a probe-side row tends to be joined with build-side changes of a similar event time. This is a best-effort alignment and does not make the result deterministic. | ||
|
|
||
| The build-side state grows with the number of distinct build-side keys, and during the load phase the buffered probe-side rows add to the state footprint until the operator transitions to the join phase. Use `state_ttl` to bound the build-side state for keys that are no longer updated or joined. You can reduce the amount of data that is buffered and processed during the load phase by configuring scan start offsets on the build and probe-side inputs, for example with a `scan.startup.*` [dynamic table option hint]({{< ref "docs/sql/reference/queries/hints" >}}#dynamic-table-options). | ||
|
|
||
| **Batch mode** | ||
|
|
||
| In batch mode, a `LATERAL SNAPSHOT` join is executed as a regular (`INNER` or `LEFT`) join between the probe side and the complete build side. Batch execution reads the entire build side before joining, so there is no load phase and no incremental state build-up. The streaming-specific arguments (`load_completed_condition`, `load_completed_time`, `load_completed_idle_timeout`, and `state_ttl`) are accepted but have no effect, and the build side does not need to declare a watermark. | ||
|
|
||
| Because every probe-side row is joined against the final, complete build side, the batch result is **deterministic**. | ||
|
|
||
| Lookup Join | ||
| -------------- | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shouldn't the SNAPSHOT function be documented in the SQL functions section
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The SNAPSHOT function can only be used in a LATERAL context and not as a stand-alone table function.
But we can add a new section "Table Functions" to https://nightlies.apache.org/flink/flink-docs-release-2.3/docs/sql/functions/built-in-functions/ and list it there with a pointer to the this section in the join docs.