perf(http): enforce max content length before fully downloading the body

[marevol]

Summary

Both HTTP clients (Hc4HttpClient, Hc5HttpClient) previously copied the entire response body into memory / a temp file and only then compared it against the configured max content length — so an oversized or hostile response was fully downloaded and spilled to disk before being discarded. This enforces the limit before / during the download while preserving the exact accept/reject outcome, and fixes a concurrency issue in the robots.txt dedup.

Changes

Max content length (efficiency / DoS mitigation)

• Content-Length precheck: when the response declares a Content-Length that exceeds the effective max, reject with MaxLengthExceededException without opening the body — mirroring the existing robots.txt precheck.
• Bounded stream: wrap the entity stream in a commons-io BoundedInputStream capped at maxLength + 1, so a chunked / unknown-length oversized body is aborted mid-copy instead of fully buffered. The existing post-copy contentLength > maxLength check is unchanged and remains the authoritative gate, so the accept/reject boundary is provably identical to the old full-download behavior (a body of exactly maxLength is accepted; maxLength + 1 or larger is rejected).
• No limit (no ContentLengthHelper, or Long.MAX_VALUE): the stream is used unwrapped and nothing is prechecked/capped.
• Robustness: a malformed/hostile Content-Length header is treated as unknown (guarded parse) rather than failing the URL. When the Content-Type is not yet known (header absent), the precheck/bound use the overall maximum configured limit (ContentLengthHelper.getMaxLength()) rather than just the default, so the optimization can never truncate a body that the authoritative per-type check (run after sniffing) would accept. Under Fess's default config (default 10 MB, only text/html=2.5 MB ≤ default) this is a no-op.

robots.txt dedup (thread-safety)

• CrawlerContext.robotsTxtUrlSet is now Collections.synchronizedSet(new LruHashSet<>(10000)) (LRU bound/eviction unchanged), and the clients' non-atomic contains()-then-add() is collapsed into a single atomic add() (LruHashSet.add returns true only when newly added). This removes a race where two threads could both fetch the same host's robots.txt and could concurrently modify the backing map.

Testing

• Precheck rejects an over-declared Content-Length without awaiting the body; a chunked oversized body is capped mid-stream; within-limit and unlimited/no-helper cases are unaffected.
• Malformed Content-Length header no longer fails the URL (within-limit body succeeds; oversized body is still capped/rejected by the bound + post-copy check).
• Unknown Content-Type with a per-type limit larger than the default is not truncated-and-accepted.
• ContentLengthHelper.getMaxLength() returns the overall maximum; concurrent robots.txt add() yields a single winner.
• Full fess-crawler module suite passes (a pre-existing 1 ms-timeout race test is intermittently flaky under parallel load and never touches the modified code); mvn formatter:format / license:format / javadoc:jar clean.