Skip to content

test: run deliverymq retry tests on a fake clock - #1163

Open
alexluong wants to merge 2 commits into
test/fake-clock-redisfrom
test/fake-clock-deliverymq
Open

alexluong wants to merge 2 commits into
test/fake-clock-redisfrom
test/fake-clock-deliverymq

Conversation

@alexluong

@alexluong alexluong commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Run the deliverymq retry tests on a fake clock. Based on #1162.

deliverymq was the slowest package in the short suite: its tests ran one after another, each waiting out rsmq visibility timeouts and context deadlines. Everything now runs inside a synctest bubble (in-memory deliverymq, the retry scheduler on testutil.NewSynctestRedis, Monitor, mempubsub), with one shared teardown, stopRetryStack. The tests also run in parallel (t.Parallel); none of them touch package-level state.

The package now runs in under 1 s.

The conversion surfaced three test bugs:

  1. TestRetryScheduler_EventFetchSuccess was flaky: it could assert after the 2nd publish but before the 2nd log entry was written (also a data race on the entries).
  2. TestDeliveryMQRetry_RetryMaxCount published a task without Attempt (0); real tasks start at 1, so it made 4 attempts. The old check asserted as soon as it saw 3 and never waited for the 4th. Now Attempt: 1; setting it back to 0 fails with 4 vs 3.
  3. mockEventGetter.records was written and read without its mutex.

Timing choices:

  • WithRetryNotReadyDelay(time.Second): the 10 s default outlasted the test windows. It only shortens the wait; the tested behavior is the same.
  • RetryMaxCount and EligibleForRetryTrue use a 2 s backoff instead of 50 ms. Under ~1 s a retry is due at once and can run before the current attempt is logged. The lookup accepts attempts up to 1 s older than the retry's attempt time (retryLookupSkew), so it finds the previous attempt and publishes the current attempt number a second time. That's a race in the retry path, left for a follow-up; 2 s keeps these tests off it and costs nothing on the fake clock.
  • "No further attempts" checks advance the fake clock through the full window before asserting. EventNotFound and EventFetchError also assert the lookup ran.
  • The mempubsub consumer receives on context.Background() and stops through Shutdown: stopping it through its context leaks gocloud's receive goroutine, which fails the bubble.

The package was stress-tested with repeated runs, plain and with -race.

Not in this PR: the sub-second backoff race above.

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant