Preferred way to use a search bar

[afuetterer]

Hi,

let's say a PlaywrightCrawler should use request a url which contains a search bar. The initial request should open the url.
Instead of scraping/parsing the initial page to find links to add the request queue, is there a way to have additional requests use the search bar instead? Playwright can fill the input field, but how to achieve this, if the url is generated dynamically and not part of the base url. A list of search queries is present on disk.

Is there a way to achieve this, or is this absolutely the wrong thing to try to achieve with crawlee?

Thanks.

[janbuchar]

Do I understand it right that you have a list of search queries and you want to submit each to a search box on a website and then scrape the search results?

[afuetterer]

Yes, that is correct. I have a base URL, which redirects me to another (dynamic) URL. There is a search bar.

I have a fixed list of search queries in advance and want to scrape the results.

Is this in scope of crawlee? Maybe via manually adding requests to the request queue?

[janbuchar]

Could you share the website that you're working with? It is probably possible to achieve what you want, but now I can only guess 🙂

[afuetterer]

I am sorry. It is a search engine, that is not available without licensing. So I cannot share the URL. Maybe you could explain using https://crawlee.dev/search? You just need to pretend that the search URL is the result of a redirect and contains a dynamically generated token.

My question: Where and when would I ideally store and access the list/queue of search queries? At the crawler level?

In my opinion, once the search bar is visible, I could create additional requests using the search bar, filling it via playwright, and handing them of to a search results handler? But I am unsure.

[janbuchar]

First off, I'd definitely advise you to use request labels to differentiate the request types, but you may already be doing that.

Regarding handling the search bar, there are multiple high level approaches. I'd probably make a request for each of your search terms like this:

start_requests = [
  Request.from_url(url=f"https://secret-search-engine.com#{term}", label="search", keep_url_fragment=True) 
  for term in search_terms
]

await crawler.run(start_requests)

In your handler for the search label, you can extract the fragment from context.request.url, enter it into the search bar and handle the results in any way you see fit.

[afuetterer]

Thanks a lot, I will try that.