feat(crawlers): Add extract_form_requests helper for HTTP crawlers with HTML parsers

[Mantisus]

Description

• Adds extract_form_requests to ParselCrawlingContext and BeautifulSoupCrawlingContext. It reads a form from the page, fills in the given values and returns requests that submit it the way a browser does, keeping hidden fields such as CSRF tokens. By default, one form is submitted: the one sharing the most field names with fields. The selector, click and all_forms arguments pick the forms and the submit button.

Testing

• Added new unit tests

[Pijukatel]

Currently, this is supported as user-specific code. What motivated adding it as a dedicated helper in the library?

[Mantisus]

I believe it'll be quite useful for users, since it removes a fair amount of boilerplate. A good example is a form on an ASP.NET site, which may have a single user field next to several hidden fields that need specific handling, like __VIEWSTATE, __EVENTVALIDATION and the clicked submit button. It should also make migrating from Scrapy easier, since Scrapy has similar functionality.

[vdusek]

LGTM