Skip to content

refactor(transformer): add Document-accepting storeChildUrls overload - #177

Merged
marevol merged 1 commit into
masterfrom
feat/html-transformer-shared-dom
Jul 8, 2026
Merged

refactor(transformer): add Document-accepting storeChildUrls overload#177
marevol merged 1 commit into
masterfrom
feat/html-transformer-shared-dom

Conversation

@marevol

@marevol marevol commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Refactor HtmlTransformer.storeChildUrls to expose a Document-accepting overload, so a caller that has already parsed the response body into a DOM (e.g. an XPath-based content extractor) can reuse that Document for child-URL extraction instead of parsing the same page a second time.

Changes

  • Add protected void storeChildUrls(ResponseData, ResultData, Document) containing the existing child-URL extraction logic (base-href resolution, child-URL rules, getChildUrlSet() merge, self-URL removal).
  • The existing storeChildUrls(ResponseData, ResultData) keeps identical behavior: it parses the body exactly as before and delegates to the new overload.

Behavior for all current callers (HtmlTransformer, FileTransformer, XpathTransformer) is unchanged:

  • The 2-arg method still parses (no BOM handling / setEncoding change) and its try/catch is preserved.
  • The new overload declares MalformedURLException, which the 2-arg method's existing catch (Exception) wraps into CrawlerSystemException("Could not store data.", e) exactly as before.
  • The overridable helpers (getChildUrlRules, getUrlFromTagAttribute, getBaseHref, convertChildUrlList, addChildUrlFromTagAttribute) are still invoked via dynamic dispatch, so subclass customizations continue to apply.

Testing

  • New test asserting the 3-arg overload produces the same child URLs as the 2-arg method for the same page, plus a sanity check that the 2-arg method still works end-to-end.
  • Full fess-crawler module test suite passes (1980 tests, 0 failures); mvn formatter:format / license:format / javadoc:jar all clean.

Motivation

This enables a follow-up change in Fess (FessXpathTransformer) to parse each crawled page's DOM once instead of twice per page.

Extract the child-URL extraction logic from storeChildUrls(ResponseData,
ResultData) into a new storeChildUrls(ResponseData, ResultData, Document)
overload so a caller that has already parsed the response body into a DOM
(e.g. an XPath-based content extractor) can reuse that Document instead of
forcing a second parse of the same page. The existing two-argument method
keeps identical behavior: it parses the body and delegates to the new
overload.
@marevol
marevol merged commit 17a640e into master Jul 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant