Repository navigation
fix(http): identify downloads and verify conversion transport - #2830
Open
Jasper (jipeng6036-del) wants to merge 3 commits into
Open
Jasper (jipeng6036-del) wants to merge 3 commits into
Jasper (jipeng6036-del) wants to merge 3 commits into
Conversation
…microsoft#2820) (cherry picked from commit 27e7316)
…icrosoft#2820) (cherry picked from commit 52b92e9)
Use a real local HTTP server to cover normal downloads, redirects, built-in HTML conversion, Unicode content, and caller-supplied session headers. Retain the two original identification commits from sushan5140 microsoft#2821: 27e7316 and 52b92e9. Co-authored-by: Susan5140s <167346857+sushan5140@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and fix
Fixes #2820.
The default Requests User-Agent can be rejected before Wikipedia's converter receives a response. Identify sessions created by MarkItDown as
MarkItDown/<version> (https://github.com/microsoft/markitdown), retaining the existing Markdown-first Accept header. Caller-supplied sessions keep their own headers. This follows Wikimedia's descriptive User-Agent policy, without impersonating a browser.Based on sushan5140's unmerged #2821. Both original commits are retained separately, with their original author/date and
cherry-picked-fromsource SHAs:27e7316a2e07d705993b29bfce47162b1ffbf75f52b92e9ab7d9b25a8fbb4544c2d8e5cc987fed03The additional commit adds four real HTTP transport regressions through the public
MarkItDown.convert()entry: direct and redirected downloads, default identification, custom session headers, built-in HTML conversion, and Unicode content. The tests use a local HTTP server, with no mocked Requests transport or external network dependency. They extend the source PR's prepared-request header tests. Original authorship and a realCo-authored-bytrailer are preserved.Older related #1445 uses a bare client name, and #1849 uses a browser identity. This version retains #2821's version/contact identification and verifies the complete download/conversion path.
Verification
Baseline:
7482c092d9c0074a89803c9fdb63a969e7fe17fc. Candidate tested/final tree:a68ee954afeeeae358ff34fcc5a66d8d719ec5ae.hatch test --python 3.12: 1133 passed, 48 skipped.pre-commit run --all-files: passed (Black 26.10.1). The original header assertion's formatting is included in the additional commit.MarkItDown().convert("https://en.wikipedia.org/wiki/Python_(programming_language)"): baseline raises HTTP 403; the built and installed candidate wheel succeeds with titlePython (programming language)and 219,771 Markdown characters. The installedmarkitdown <url> -o <file>CLI also exits 0 and writes nonempty Markdown. No model/API service was used.The live Wikipedia result is an observation from this environment, not a guarantee that every site's access policy accepts this identity. Hosted CI and its remaining Python/OS matrix are checked after submission and are separate from these local results.
AI assistance: Codex was used to implement and review the additional transport coverage, inspect the reused patch, and run the verification above.