A retry policy question.

[matecsaj]

When a URL request fails, is the retry count for it always decremented?

I can think of a few cases where the URL is not the problem:

• A bad proxy.
• Target server overload.
• Transient resource exhaustion on the local machine.

Could you clarify how Crawlee handles these situations?

[janbuchar]

• A bad proxy.

We try to detect these. If we conclude that the proxy is busted or that we got blocked because of suspicious activity, we raise a SessionError, which triggers a session rotation. There is a default limit of 10 rotations per request.

• Target server overload.
• Transient resource exhaustion on the local machine.

It is difficult to detect these reliably, so they will most likely be considered to be "normal" request errors and use the standard retry mechanism