[GSoC 2026] Kafka Streams runner: user documentation, marked experimental - #39627
Conversation
…ntal Adds the runner's documentation page, linked from the runners menu: what the runner is and why someone would choose it, how to start the job server and submit a pipeline, every pipeline option with its default, the internal topics it creates, and what is and is not supported. The unsupported list is specific rather than a general disclaimer, since these are core parts of the Beam model rather than nice-to-haves: side inputs, stateful ParDo and user timers, merging windows, custom WindowFns, splittable DoFn, TestStream, reading a source in parallel, the bundle time bound, finalizeCheckpoint, and committed metrics. Each says what it means for a user. KafkaStreamsRunner's javadoc now opens by saying the runner is experimental, naming the main gaps and warning that behaviour and options may change, so the caveat is visible from the code as well as the website. Adds :runners:kafka-streams:runJobServer, because the documentation needs a command that works and the job server could previously only be started by invoking its driver class directly.
je-ik
left a comment
je-ik
left a comment
There was a problem hiding this comment.
Can we also update capability matrix?
website/www/site/data/capability_matrix.yaml
| --runner=PortableRunner \ | ||
| --jobEndpoint=localhost:8099 \ | ||
| --bootstrapServers=localhost:9092 \ | ||
| --applicationId=my-beam-pipeline |
There was a problem hiding this comment.
We probably should provide a "wrappers" as flink does that will run their own jobserver automatically.
There was a problem hiding this comment.
Good catch, and it turned out we already have one — I'd just documented the wrong path. KafkaStreamsRunner.run() starts a KafkaStreamsJobServerDriver on a dynamic port when jobEndpoint is empty, and stops it when the pipeline finishes, so from Java there is nothing to start by hand.
The page now leads with --runner=KafkaStreamsRunner for that, and keeps the manual job server as what you would use from another SDK or against a shared deployment. My original text only showed the manual route, which made the runner look more awkward to use than it is.
There was a problem hiding this comment.
Flink adds a similar wrapper for python (and maybe go as well?), you can check it for inspiration.
Documents that the runner starts its own job server. Selecting KafkaStreamsRunner with no jobEndpoint set brings one up on a dynamic port and shuts it down with the pipeline, so nothing has to be started by hand from Java; the manual job server is now what another SDK or a shared deployment uses. Links a tracking issue against each unimplemented feature, so the list says where to follow up rather than only what is absent. Adds the runner to the capability matrix, with an entry in each capability rather than only the column. The partial ones say what the limit is: the source API is read by a single reader, metrics are attempted only, and event-time triggering covers the default trigger.
|
Also Added, in capability_matrix.yaml. The runner is now a column and has an entry in all 38 capabilities rather than just the header, so it doesn't show up as blanks. I kept the partial ones honest rather than optimistic: Source API is "Partially" because we read with a single reader, Metrics is "Partially" for attempted-only, and event-time triggers is "Partially" since only the default trigger is exercised. Everything else is a straight Yes or No against what the ValidatesRunner suite actually covers. The diff is additions only — no existing runner's entries were touched. |
|
Assigning reviewers: R: @damccorm for label website. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
2 participants
Summary
Part of #18479.
Adds the runner's user documentation and marks it experimental, which was the remaining documentation deliverable. Reviewing #39611 the point came up that the features which are core but not yet implemented must be written down; this is where they are.
The documentation
website/www/site/content/en/documentation/runners/kafkastreams.md, linked from the runners menu alongside the others. It covers what the runner is and why someone would pick it, how to start the job server and submit a pipeline, every pipeline option with its default, the internal topics it creates, and two lists that matter more than the rest: what is supported and what is not.The unsupported list is deliberately specific rather than a general disclaimer, and says what each gap means for a user:
ParDoand user timers; merging windows, so no session windows; customWindowFns; splittableDoFn;TestStreammaxBundleTimeMsis accepted but has no effect, because closing a bundle from a wall-clock punctuator duplicated output against a real broker and the cause is not yet understoodfinalizeCheckpointis not called, so a source relying on finalization to acknowledge data will not see itMarking it experimental
KafkaStreamsRunner's javadoc now opens by saying the runner is experimental, naming the main gaps and warning that behaviour and options may change, with links to the documentation and the tracking issue. The documentation says the same in its own section, so someone arriving from either direction learns it before they invest in it.A task to start the job server
The documentation needs a command that works, and there wasn't one — the job server could only be started by invoking
KafkaStreamsJobServerDriverdirectly.:runners:kafka-streams:runJobServernow does it, with arguments passed through-PjobServerArgs. Verified by running it: the job service comes up onlocalhost:8099and artifact staging onlocalhost:8098, which is what the documentation tells the reader to expect.Testing
The runner code change is javadoc only; the gate is unchanged and still green.