Skip to content

fix(throttling-manager): Correct the backoff for concurrent 429s and zero Retry-After - #2159

Open
Mantisus wants to merge 1 commit into
apify:masterfrom
Mantisus:fix-backoff-429
Open

fix(throttling-manager): Correct the backoff for concurrent 429s and zero Retry-After#2159
Mantisus wants to merge 1 commit into
apify:masterfrom
Mantisus:fix-backoff-429

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • Only the first 429 of a burst now advances the exponent, and it decays once the domain has left us alone for a full extra backoff window. Previously every concurrent 429 stepped it, so eight in-flight requests pushed the delay from 2s to past max_delay in one go, and any handled request reset the counter, including one that had just failed for good. throttled_until split into separate backoff_until and crawl_delay_until clocks, since the crawl delay is armed on every dispatch and would otherwise pass for an active backoff.

Issues

Testing

  • Added new unit tests

@Mantisus Mantisus self-assigned this Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ThrottlingRequestManager computes the wrong backoff for concurrent 429s and for Retry-After: 0

2 participants