Skip to content

fix: replace batcher dependency to avoid shutdown deadlock in logmq - #1160

Merged
alexluong merged 3 commits into
mainfrom
fix/logmq-batcher-shutdown-deadlock
Oct 24, 2026
Merged

alexluong merged 3 commits into
mainfrom
fix/logmq-batcher-shutdown-deadlock

Conversation

@alexluong

@alexluong alexluong commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

The log consumer's Shutdown can hang until the process is killed. In mikestefanello/batcher v0.1.0, Shutdown's final flush restarts the ticker as it returns, and Shutdown then takes the processing lock and keeps it. A tick in that window sends the ticker goroutine to wait on that lock, so Shutdown's unbuffered send to that goroutine is never received. Pending messages are already flushed by then, so nothing is lost; only shutdown stalls.

It is rare with the default 10 s batch delay. logmq tests use much shorter delays and hit it under CPU load: TestCharacterization_TwoDestinationsInterleaved, TestDisabledSignals_FailedEmitFailure_Nacks and others fail now and then with bp.Shutdown did not return within 30s. Running the logmq tests in parallel with CPU burners, 13 of 80 runs hung before this change and none of 40 after.

  1. internal/batcher replaces the dependency with a generic Batcher[T]. It flushes at MaxItems, or MaxWait after a batch's first item, and runs at most MaxInFlight flushes at once. Flushes run outside its lock, so Close has no handoff to wait on: it flushes what is pending and waits for running flushes. The dependency is removed from go.mod.
  2. logmq runs on it with one flush at a time. An Add that fills a batch waits for the previous write to finish, then starts its own. Before, every Add waited while a batch was written; now the next batch can fill during a write.

Other improvements:

  • The log message handler nacks a message the batcher no longer accepts (after shutdown) instead of leaving it unacked.
  • TestDelivery_ExhaustedRetries_LoserAcksWhileWinnerInFlight: wait for both attempt.failed sends before asserting; the winner's send runs concurrently and could land after the check.

🤖 Generated with Claude Code

@alexluong
alexluong merged commit c52a004 into main Oct 24, 2026
1 check passed
@alexluong
alexluong deleted the fix/logmq-batcher-shutdown-deadlock branch October 24, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants