fix: replace OCR truncation with quality checks to prevent data loss - #40
Merged
Conversation
- Remove 50K char truncation in pdf_handler.py and ocr_utils.py - Add is_bad_ocr() quality check on OCR output to catch garbled text - Restore TEXT_KEYS ordering in find_text() to prefer 'md' over 'text' - Log warnings for oversized output (>50K chars) without data loss - Fix test_force_flags.py mock for new quality check behavior - Fix lint errors: unused imports, unsorted imports, unused function
SamD
force-pushed
the
fix/ocr-oversized-output-quality-check
branch
from
June 8, 2026 17:17
57f4c46 to
8503283
Compare
SamD
commented
Jun 8, 2026
| mock_preprocess.return_value = MagicMock() | ||
| mock_convert.return_value = [MagicMock()] | ||
| mock_send_ocr.return_value = ("ocr text", None, None, "docling", None, None) | ||
| mock_is_bad_ocr.return_value = True |
Contributor
|
APIError: Insufficient balance. Manage your billing here: https://opencode.ai/workspace/wrk_01KM1TM9SZGWE0MPG87G3MS3JK/billing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The previous OCR fallback code had a bug where one page returned over 20 million characters, which is impossible for a single PDF page. The root cause was uncommitted changes on top of revert
ed51e00that introduced 4 bugs:pdf_handler.py:54— Garbled-text check (chars/word > 1000) was useless. A 20M char page with normal-looking words (e.g., 3M words) has a ratio of ~6.7, passing right through.ocr_utils.py:278—TEXT_KEYSreordered to prefer"text"over"md", picking the wrong/larger field from docling-serve responses.ocr_utils.py:360-367— Local Doclingrun_ocr()had no size cap at all.ocr_utils.py:315-317— Oversized text was discarded entirely (lost whole page) instead of being handled gracefully.Solution
Replace hard truncation with quality checks to ensure zero data loss:
pdf_handler.pyandocr_utils.py(both remote and local paths)is_bad_ocr()quality check on OCR output inpdf_handler.pyto catch genuinely garbled text (gibberish, repetition, abnormal word lengths)TEXT_KEYSordering infind_text()to prefer"md"over"text"(shorter, cleaner fields first)test_force_flags.pyfor new quality check behaviorAdditional Fixes
test_chat_session_service.pyandtest_ingestion_recovery.pyclearChat()function inindex.astroTesting
ruff checkpassesImpact
find_text()now prefers markdown fields over raw text dumps