perf: add optional Rust kernels for native PDF pipelines

[myhloli]

Problem and behavior

Native PDF parsing and shared text/table evidence repeatedly compute character geometry, line relationships, script roles and dense-table candidates in Python. This PR adds optional private Rust kernels and a PDFium batch bridge while retaining public Python interfaces, complete Python fallback and output semantics. DOCVORTEX_COMPUTE_BACKEND=auto|python|rust selects the backend.

Round five, completed at 81c86f4596422e15623f2609616631ef6f1119bd, uses private protocol 6 and adds:

• Ordered overlap queries and conservative baseline geometry filtering, preserving the original source-character branch and final Python decisions.
• Batched stable mutual-nearest script matching; Python retains Unicode handling and recursive text/object materialization.
• Table-row bounds reuse and annotation aggregation that returns original coordinate sources. Bounded immutable caches provide independent mutable containers to each candidate. Enumeration, scoring and merge order are unchanged.
• Exact fallback for integer, nonfinite and unsupported geometry; no lossy integer-to-float conversion in the new paths.
• Explicit recursion arguments to avoid retaining consumed lines in a self-referencing closure. Empty text pages skip character-address packing; skipped empty work is counted separately from real native bridge calls. Production GC policy is unchanged.

Final performance

Frozen round-four a281218 with its matching extension versus the final implementation. macOS arm64, CPython 3.14.4, pypdfium2 5.13.0. Four variants use independent processes, one warmup and five samples, alternating order by document; process-tree RSS is sampled separately. Formal timing never overlaps local tests, builds or profiling.

Public parse	Round 4 Rust	Round 5 Rust	Time reduction
49.-	2.8249 s	2.0069 s	29.0%
caibao1	2.5552 s	2.5731 s	-0.7%
demo1	1.5553 s	1.5540 s	0.1%
demo2	1.1031 s	1.1052 s	-0.2%

Dense shared text: 6.4204 → 0.6670 s (89.6% lower). Both core targets pass: public parsing ≥20% lower and shared text ≥60% lower. These shared measurements use frozen layout inputs and do not represent complete model inference.

Absolute-reference gap retained: caibao1 is 0.0071 s above the prior 2.566 s reference; it is approximately 0.7% slower than the concurrent frozen Rust baseline and passes the 5% regression gate. demo1/demo2 retain their original absolute references. This does not claim every absolute-time reference was met.

All 32 public and 32 shared document checks pass complete output equality and the unchanged >5% sustained time/RSS regression gates. Shared coverage was expanded beyond the four headline PDFs. Seven shared documents triggered prescribed reverse-order checks; all pass final adjudication. Maximum current/reference Rust tree-RSS ratio: public 1.0134, shared 1.0039.

Validation

• 32 PDFs / 299 pages, four-way complete ModelJson/MiddleJson, text evidence, membership/source indices, diagnostics and asset digests; 186 table regions retain structure, HTML, confidence and diagnostics. No goldens regenerated.
• 515,343 raw characters and font-sharing relationships equal; empty-page skip accounting independently verified.
• Full local Rust: 4,617 passed, 1 skipped. Python: 4,290 passed, 328 skipped. Rust workspace tests/core contracts, all-target Clippy and formatting passed.
• Unmodified MinerU dev@cabe6e34: 137 tests per backend, plus four-document medium/high four-way replay equality and normal process exits. The existing opt-in adapter connects six legacy raster mocks to the encoded-crop boundary without changing business code or original assertions; real rendering is independently tested.
• 42 pages: overlay pixels, HTML, Markdown and assets equal. Four original/final page spot-checks cover dense and rotated tables, annotations, formulas/scripts and charts. Independent Poppler rendering also matches the frozen dense-page overlay.
• Installed native wheel matches 393 source/binary files. pypdfium2 5.10.1 and 5.13.0 each pass 205 installed kernel/FFI tests plus real dense-PDF parity smoke; ABI, CLI, pure-Python wheel and Rust-source/lockfile sdist verified.
• Final main CI: 39/39 jobs passed, final wheel CI: 7/7 jobs passed. Includes Python 3.10–3.14 and installed wheels on five platforms. Intel macOS 3.14 retains the existing Magika/ORT limitation: independent ABI/kernel checks on 3.14 and complete parsing on supported dependency versions.

Evidence and remaining hotspots

Independent profiling reduces Python overlap-connection calls 21,480,737 → 242,048, baseline decisions 872,946 → 15,282, and dense-parser bbox unions 1,613,658 → 136,968. Candidate materialization remains 7,216 calls and expansion 7,217: no candidate truncation. Remaining work is candidate expansion, rule-interval partition/compatibility checks and annotation/marker processing. Profiler totals are separate from formal timing.

Local detailed report: output/pdf/rust-round5-20260925/REPORT.md; final evidence is under its accepted-*, validation, installation and visual files. Rejected batching and earlier GC/empty-page measurements remain separately identified. The external PDF is not uploaded; SHA256 1bbd28a4f0733ab3dbd7e651afed11338e8baf535ad92413d96b108eab5a76ed.

PR remains draft; no merge or release performed.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-25T18:52:01.587238Z	5c1289b	Draft marked ready
🔒 Security Review	✅ Completed 2026-09-25T18:53:23.131185Z	5c1289b	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.