Repository navigation
docs(crawl): say that a crawl whose patterns match nothing completes with 0 results - #52
Merged
Merged
Conversation
…ategies a crawl ran with
`outputFormat` takes `markdown` as well as `html`: the crawl returns each
kept URL's page converted to Markdown, read with `content()` or
`download()` as HTML already was.
A crawl reports `discovery`, the strategies it finds URLs with on each
page it walks, and a create can ask for them. `links` follows the links
in the page's HTML, and is what a crawl runs with when the create leaves
it out. The field stays optional, so the type parses an older answer too.
`content(..., { download: true })` asks for the page as a file to save
rather than one to show. The text is the same; the server sends a file
name built from the content id.
`CrawlErrorCode` gains `api_key_cap_reached`, which a crawl ends with
when the key that created it reaches one of its credit caps.
The lock names the API description this surface was synced to, by its blob id, so a later drift check compares against it instead of reporting the surface as never synced. Two operations in that description are not served to callers and no surface exposes them; the lock lists them as pending, so a change to either does not read as drift here.
… set The Crawl API now ends a crawl that runs out of credits, or whose API key hits a credit cap, as `stopped` with `error.code` `insufficient_credits` or `api_key_cap_reached` and no `stop_reason`, instead of `failed`. The type comments, the `wait` comment and the README said `error` belongs to a `failed` crawl and `stopped` to `stop()`. They now describe both cases, the README example handles the credit stop, and a unit test pins that `wait` resolves such a crawl with its error and no stop reason.
…it opens The Crawl API now treats include and exclude patterns as a filter on what a crawl returns. The crawl still opens on-site pages that fail them, within `depth` and `maxPages`, to follow their links to pages that match, and never returns those pages. Every page it opens counts toward `maxPages` and is billed. The JSDoc for `maxPages`, `includePatterns` and `excludePatterns`, and the README, now say so, and that a crawl whose patterns match nothing ends `completed` with no results.
…eted Acceptance check 19 asserted that an include pattern matching nothing ends the crawl `failed` with `no_items_found`. The Crawl API now ends such a crawl `completed` with 0 results and no `error`, so the check asserts that, and reads the crawl's results to confirm none came back. Check 21 still reads the defaults from the same crawl.
…esults is not a failure A URL that fails includePatterns or excludePatterns is neither returned nor opened, so the docs no longer say the crawl opens such pages or bills them. A crawl whose pages held nothing new to follow ends completed with no results and no error. seed_unreachable covers an empty start page, and no_items_found means Extract found no data on the start page.
zenrows-dev-pool
Bot
changed the title
…duct names no_items_found now reads as the crawl finding no data it could use on the start page, with the advice to start from a listing or product page.
…hat match nothing The check expects a completed crawl with 0 results, so the report no longer calls it a failed crawl.
StefanoGuerrini
approved these changes
Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before: The SDK did not say what happens when an include pattern matches nothing. Acceptance check 19 expected such a crawl to end
failedwithno_items_found. The SDK did not say whatseed_unreachableandno_items_foundmean.After: A crawl whose pages held nothing new to follow ends
completedwith no results and noerror. That covers a page with no link, only links to other sites, or only links the patterns reject. The README says to check the patterns against the URLs the start page links to.CrawlErrorCodesays thatseed_unreachablecovers a start page that could not be reached, was blocked, or came back empty. It says thatno_items_foundmeans the crawl read the start page but found no data it could use, so it had nothing to follow. The patterns still limit what a crawl opens as well as what it returns, somaxPagesandexcludePatternskeep their wording frommain.How:
README.md: one sentence on the 0-result outcome, next to how a failed crawl carrieserror.src/crawl/client.ts: theincludePatternsdoc says a crawl whose patterns reject every link endscompleted.src/crawl/types.ts: theCrawlErrorCodedoc gives the narrowed meaning ofseed_unreachableandno_items_found.acceptance/crawl: row 19, its check and its report title expectcompleted,items_found0, no results and noerror.acceptance/crawl/contract.lockrecords the contract this surface follows.maininto the branch.Tests:
pnpm check(biome): 41 files checked, no fixes needed.tsc --noEmitontsconfig.jsonandtsconfig.test.json: no errors.pnpm test -- --run: 27 files passed, 1 skipped; 372 tests passed, 6 skipped.bash acceptance/crawl/run.sh --no-key: PASS 8, SKIP 21 (the live checks need a key), 0 findings.