Skip to content

[AURON #2062] AuronKafkaSourceFunction support generating watermarks#2089

Open
Tartarus0zm wants to merge 1 commit intoapache:masterfrom
Tartarus0zm:flink-kafka-support-watermark
Open

[AURON #2062] AuronKafkaSourceFunction support generating watermarks#2089
Tartarus0zm wants to merge 1 commit intoapache:masterfrom
Tartarus0zm:flink-kafka-support-watermark

Conversation

@Tartarus0zm
Copy link
Contributor

Which issue does this PR close?

Closes #2062

Rationale for this change

  • auron kafka source support flink watermark

What changes are included in this PR?

  • modify AuronKafkaSourceFunction, Assign tasks to their corresponding Kafka partitions and generate watermark
  • modify kafka_scan_exec.rs, Consume the list of kafka partitions passed from the Java side
  • add KafkaTopicPartitionAssigner , copy from flink
  • add SourceContextWatermarkOutputAdapter , copy from flink

Are there any user-facing changes?

  • No

How was this patch tested?

  • There is currently no Kafka environment integration, so automated testing is not possible.

@Tartarus0zm
Copy link
Contributor Author

@richox CI is green, PTAL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AuronKafkaSourceFunction support generating watermarks

2 participants