documentation - how to restart failed crawl jobs

[yqiang]

It would be great if there was more documentation on how to deal with large crawl jobs that fail. For example, in my current crawl job, the job failed because the proxy I was using failed (e.g., ran out of credits). Now the request has reached its maximum retry count, and won't be processed again. What's the correct way to update the request so that it'll get picked up for processing again?

[Mantisus]

I just want to make sure I understand your case correctly.

You are interested in a solution for the case. How can I use CRAWLEE_PURGE_ON_START=0 to restore the queue so that links that were not processed because they ran out of repetition limits are taken back to work?

[yqiang]

Right - perhaps it's just me but this seems like it would be a common failure mode.

[Mantisus]

I don't think we have simple mechanisms for this.

However, we should probably expand the queue interaction capabilities to give users the ability to handle the case you described. And yes, we definitely need more examples in the documentation :)

I would handle the case you described during runtime using crawler.error_handler. When any errors occur, the error_handler decides whether the request should return to the queue or be processed differently. If many proxy-related errors are registered, then the crawler stops. After that, you can restart it with the flag CRAWLEE_PURGE_ON_START=0 and a new proxy pool, so that it continues working.

[janbuchar]

You can also use the BasicCrawler.failed_request_handler which gets invoked with after the maximum number of retries for a request has been reached and re-enqueue the failed request or store it somewhere so that you can restart it later.

[yqiang]

Great, I'll give this a try next time I run into this issue.