fix(kafka): keep group consumer polling while Finish waits on deferred acks - #115
Merged
Divyanshu Tiwari (divyanshu-tiwari) merged 2 commits intoSep 8, 2026
Merged
Conversation
Divyanshu Tiwari (divyanshu-tiwari)
requested a review
from a team
as a code owner
September 8, 2026 07:24
Divyanshu Tiwari (divyanshu-tiwari)
force-pushed
the
fix/kafka-heartbeat-during-finish
branch
from
September 8, 2026 07:30
5db1ba7 to
9e4f29a
Compare
…d acks Group consumers stop polling once max_records, end_after, or retry_limit fires. Finish then waits for downstream tasks to settle, which for slow HTTP or sink drains exceeds max.poll.interval.ms (5m) and causes librdkafka to leave the group and fail the final commit with Broker: Unknown member.
Divyanshu Tiwari (divyanshu-tiwari)
force-pushed
the
fix/kafka-heartbeat-during-finish
branch
from
September 8, 2026 08:24
9e4f29a to
03440bf
Compare
Mahesh Kamble (ma-gk)
approved these changes
Sep 8, 2026
Divyanshu Tiwari (divyanshu-tiwari)
deleted the
fix/kafka-heartbeat-during-finish
branch
September 8, 2026 08:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
max_records,end_after, orretry_limitfires.Finishthen waits for downstream tasks to settle before committing stored offsets.max.poll.interval.ms(default 5 minutes), librdkafka deems the consumer dead, leaves the consumer group (MAXPOLL ... leaving group), and the subsequent offset commit fails withwarning: failed to commit offsets for topic ...: Broker: Unknown member.max_records: 900followed by 10-25m HTTP drain across 50 workers).pauseAll()assigned partitions on read loop exit so subsequent polls during shutdown do not fetch new messages.Finish(), usewaitWhile(k.tracker.Wait, k.heartbeatPoll)to keep polling librdkafka every 1s while waiting for deferred acks to settle.MAXPOLL/Unknown memberfailure mode in Kafka task docs.Test plan
waitWhilepolling mechanics under race detector with unit test (PASS)go test -v -race ./internal/pkg/pipeline/task/kafka/... ./internal/pkg/pipeline/ack/...MAXPOLLleaving group and successfully commits offsets inFinishon test deploy