Extract structured JSON (policy number, endorsement number, insurer) from
Turkish insurance PDFs. Single fast engine (pypdf, layout-preserving text),
plus a long-running --serve mode designed for desktop app integration
(watch a folder, report results as JSON on stdout).
Grab the latest policy-extract-windows-x64.exe from
Releases:
Verify (optional, PowerShell):
$exe = "policy-extract-windows-x64.exe"
$expected = (Get-Content "$exe.sha256" -Raw).Split()[0]
$actual = (Get-FileHash $exe -Algorithm SHA256).Hash.ToLower()
if ($expected.ToLower() -ne $actual) { throw "sha256 mismatch" }
.\$exe --versionpython -m venv .venv
.venv/Scripts/Activate
pip install -e .Requires Python >= 3.10. Runtime deps: pypdf, fonttools (needed by pypdf
for CFF/Type1-font PDFs). No other dependencies — intentionally stdlib-only
beyond those two so the PyInstaller exe stays small.
# single file: JSON to stdout
python -m policy_extract.cli sample.pdf
# single file, explicit flag (handy in tests)
python -m policy_extract.cli --file "C:\policies\sample.pdf"
# folder: scan first 7 pages (default), one JSON object per line (JSONL)
python -m policy_extract.cli ./policies
# change / remove the page limit
python -m policy_extract.cli ./policies --max-pages 5
python -m policy_extract.cli ./policies --max-pages 0
# custom .metadata path
python -m policy_extract.cli ./policies -o ./policies/.metadata
# ignore the cache, recompute everything
python -m policy_extract.cli ./policies --no-cache
# write to file, drop tables (full_text is off by default)
python -m policy_extract.cli sample.pdf -o out.json --no-tables
# opt in to full_text
python -m policy_extract.cli sample.pdf -o out.json --with-text{
"source_file": "sample.pdf",
"police_no": "06857993",
"police_no_source": "table",
"zeyil_no": null,
"zeyil_no_source": null,
"company": "allianz",
"company_confidence": "high",
"company_scores": {"allianz": 28},
"full_text": "...",
"tables": [...]
}police_no: Allianz composite numbers keep only the last segment (0001-0310-05746303→05746303, including leading zeros). HDI numbers drop an optional letter/digit suffix (175101008353 - T3→175101008353); numbers without it stay unchanged.police_no_source:inline= fromPolicy No: 0000text,table= from a table headedPolicy No.zeyil_no(endorsement no): fromEndorsement No/Zeyil No.Yenileme NoandEk / Yenileme Noare ignored; renewal numbers are not endorsement numbers. CombinedPoliçe No / Yenileme Nofields are used only to extract the policy number.0,00,0/0mean "base policy", so they returnnull.company: scored dictionary match, never a single-word search — domain (+10), legal title (+5), brand (+2, +1 for generic words likegunes/turkiye), header bonus. Below-threshold or tied scores returnnull+unknown.company_scoresexplains the decision.
One engine: fast path (pypdf) — layout-preserving text extraction of a digital PDF, policy number / insurer found in text. Takes seconds, downloads no extra models.
Folder scans keep all results in .metadata. Each line carries the file's
sha256; on rescan, unchanged complete records are reused as-is
(--no-cache disables this). Records missing police_no or company
(also written to .not-found-metadata) are retried every scan, so they
heal automatically as the extractor improves.
Extraction, the serve mode and the update flow are covered. Every test file also
runs standalone (python tests/<file>.py); with pytest installed the whole suite
runs at once:
pytest -q| File | Scope |
|---|---|
tests/test_extractor.py |
extraction, company scoring, serve/batch flow, optional real-data run |
tests/test_github_update.py |
GitHub Releases source, version single-source |
tests/test_extractor_edge_cases.py |
label/size/geometry edge cases, synthetic text catalogue |
tests/test_company_detection.py |
scoring threshold, ties, header window, per-signal caps |
tests/test_synthetic_pdfs.py |
end-to-end on generated PDFs + CLI single/batch/serve |
tests/test_service_edge_cases.py |
queue, command protocol, folder lock, parent watch, metadata |
tests/test_cli_encoding.py |
UTF-8 desktop pipes with legacy Windows encodings, Turkish paths, retry commands; optional frozen exe checks via POLICY_EXTRACT_TEST_EXE |
tests/test_updater_edge_cases.py |
version compare, API/download/install against a local server |
tests/synthetic_data.py |
synthetic data generators (minimal PDF writer + catalogues) |
No real policy PDFs are required: tests/synthetic_data.py writes valid PDFs
with the stdlib (pypdf reads them as ordinary PDFs) and feeds the text/table
catalogues used by the edge-case tests.
Point at your own policy folder; tests only read it, never write
.metadata/.lock there (serve tests copy samples to temp):
$env:POLICY_EXTRACT_TESTDATA_DIR = "C:\policies"
$env:POLICY_EXTRACT_TESTDATA_MAX = "20" # default 20, stride sampling
.venv\Scripts\python.exe tests/test_extractor.pyWhat it checks: no file crashes the extractor, record shape is consistent
(police_no→police_no_source, company→confidence/scores), two reads give
the same result, two serve passes keep line counts stable. Without the env var
these two tests SKIP, the rest of the suite is unaffected.
Long-running loop instead of one-shot scanning:
# Watch a folder, queue existing PDFs, pick up new ones as they arrive.
# Events go to stdout as JSONL, logs to stderr.
python -m policy_extract.cli ./policies --serve
# Tuned: poll every 1s, exit when the parent dies
python -m policy_extract.cli ./policies --serve --poll-interval 1 --parent-pid 1234
# Version (the host app compares it against the release API)
python -m policy_extract.cli --versionService behavior:
- On start, queues every PDF in the folder, processes them in order.
- New/changed PDFs arriving mid-run are queued automatically — nothing is lost while 70k files are being processed.
.metadatais written incrementally: only changed records are appended; repeats (cache hits or identical not-found retries) don't touch the file. Every 200 computations and on every shutdown the file is deduplicated (compact); on start, stale duplicate lines are cleaned once.--no-cachewipes old records first (same rule as one-shot batch).- One serve per folder: a
.policy-extract.lockis taken on start, a second serve is rejected withfatal+ exit 2 (double writers would corrupt metadata). A stale lock from a dead process is taken over automatically.--no-lockskips this (special setups only). - Ctrl+C / close signals shut down gracefully (closing compact +
bye); hard kills are healed by the startup cleanup. - Half-written files are not processed: files with mtime fresher than
10s wait until their size is stable
stable-checkstimes in a row (--stable-checks 3 --stable-interval-ms 500); older files go straight to sha+cache. That keeps a 690-file cached rescan at seconds instead of minutes (--stable-grace-secstunes it). - Shutdown: writing
{"cmd":"shutdown"}to stdin, closing the stdin pipe (host app exited), or death of the--parent-pidPID finishes the current file, compacts.metadata, then exits. - stdout is quiet by default: only
hello, command replies (status,paused, ...) andbyeflow. File records are read from.metadata, counters from a{"cmd":"status"}reply (polling model). The old per-file event stream is available via--stream-events(debug/tests). - stderr is fully silent: not a single byte (
--verboseexcepted).
All CLI standard streams (stdin, stdout, stderr) use UTF-8, including
the Windows exe when launched with redirected pipes. The host must encode
commands and decode output as UTF-8. This preserves Turkish paths and filenames
regardless of the Windows locale or PYTHONIOENCODING setting.
Quiet by default (hello + command replies + bye); starred rows only flow
with --stream-events.
| type | content |
|---|---|
hello |
version, pid, folder, total_queued, parent_pid |
progress * |
file, done, total_hint, queued |
file_done * |
record (police_no/company...), done |
file_cached * |
cached record, done |
file_error * |
file, error, done (record is also in .metadata) |
file_queued * |
file, reason (initial/created/modified), queued |
watch_error |
error when the folder is unreadable (watching continues) |
idle * |
queue empty, watching (done, computed, cached, failures) |
status |
reply to status (counters + paused + version) |
paused / resumed / rescanned |
command acknowledgements |
parent_gone |
parent died, shutting down |
bye |
closing counters + persist_error (set when metadata unwritable) |
Polling model: the host sends periodic {"cmd":"status"}
(queued == 0 means caught up) and reads finished records from .metadata.
Pass --stream-events for live per-file events (debug).
A single bad file never kills the service; a safety net keeps the loop alive.
| situation | signal | host action |
|---|---|---|
| Broken/unreadable PDF | file_error (stream) + .metadata {"file":..,"error":..} + status.failures++ |
None needed; optionally show a "broken" badge |
| Fields not found (null) | .not-found-metadata + automatic retry next scan |
None |
| Folder unreadable (USB unplugged etc.) | watch_error, watching continues, self-heals on return |
None; show "folder unreachable" if prolonged |
| Folder missing at startup | fatal + exit code 2, process ends |
Show error, try restarting the process |
| Unexpected internal error | file_error, service survives |
None |
| Process died (exit code != 0 or pipe closed) | no bye, exitCode |
Restart the process (serve is idempotent: resumes where it left off) |
{"cmd": "status"}
{"cmd": "pause"}
{"cmd": "resume"}
{"cmd": "rescan"}
{"cmd": "shutdown"}void test() async {
final proc = await Process.start('policy-extract.exe', [
folder, '--serve', '--parent-pid', pid.toString(),
]);
proc.stdout.transform(utf8.decoder).transform(const LineSplitter()).listen((line) {
final event = jsonDecode(line) as Map<String, dynamic>;
switch (event['type']) {
case 'hello':
/* version check, ready */
break;
case 'status':
/* counters: done/queued/paused */
break;
case 'bye':
/* service closed */
break;
}
});
// Polling: periodic status for counters, finished records from .metadata.
Timer.periodic(const Duration(seconds: 1), (_) {
proc.stdin.writeln(jsonEncode({'cmd': 'status'}));
});
// On close:
proc.stdin.writeln(jsonEncode({'cmd': 'shutdown'}));
// When the app exits the stdin pipe closes -> the exe exits by itself.
// Extra safety: pass --parent-pid so the exe exits when the parent dies.
// For live per-file events (debug): add '--stream-events'.
}The host owns check-and-swap; the heavy lifting (check/fetch/install)
is done by updater.py. No extra dependencies (stdlib urllib).
Each v* tag publishes policy-extract-windows-x64.exe plus a
policy-extract-windows-x64.exe.sha256 sidecar. The host asks the exe:
# a) Is there an update? (stdout: single-line JSON)
policy-extract.exe --check-update --github-repo infumia/policy-extract
# -> {"current_version":"0.4.8","latest_version":"0.5.0",
# "download_url":"https://github.com/.../*.exe",
# "sha256":"...","update_available":true}
# b) If so: stop the serve process (stdin {"cmd":"shutdown"} + wait for exit),
# then download (stdout: fetch_progress ... + fetch_done JSONL)
policy-extract.exe --fetch-update --github-repo infumia/policy-extract --dest %AppData%\App\staging\policy-extract.exe
# sha mismatch -> exit 1 + staging deleted, old exe untouched.
# c) Install (old exe backed up to .bak, staging moved to target)
policy-extract.exe --install-update --staged %AppData%\App\staging\policy-extract.exe --target %AppData%\App\policy-extract.exe
# -> {"type":"install_done","target":"...","backup":"....bak"}
# d) Start serve with the new exe, verify with --version.On the host side, when update_available == true: shutdown → fetch (drive a
progress bar from per-line fetch_progress) → install → restart --serve
with the new exe. If the target is not writable (serve still running etc.)
install exits 1 with an explanatory error; the .bak file is the rollback.
A GitHub Releases API URL also works directly:
policy-extract.exe --check-update --update-info-url https://api.github.com/repos/infumia/policy-extract/releases/latestGET {update-info-url} -> application/json{
"version": "0.4.0",
"download_url": "https://cdn.example.com/extractor/policy-extract-0.4.0.exe",
"sha256": "hex...",
"notes": "optional release notes"
}version + download_url are required, sha256 is strongly recommended
(without it the download is installed unverified; --sha256 can pass it
manually). Same --check-update / --fetch-update flow with
--update-info-url, or a direct download with
--fetch-update --url <direct-link> --sha256 <hex> --dest ....
Local build:
powershell -ExecutionPolicy Bypass -File ./build_exe.ps1
# Output: dist/policy-extract.exe
dist/policy-extract.exe --version
dist/policy-extract.exe ./policies --serveRelease build (CI): push a tag, GitHub Actions builds the exe on Windows,
renames it to policy-extract-windows-x64.exe, writes the .sha256
sidecar, and attaches both to the GitHub Release. See
.github/workflows/release.yml.
Single source of truth: SERVICE_VERSION in src/policy_extract/version.py.
pyproject.toml's version must match it, and release tags are v +
that version (e.g. v0.5.0). CI fails the release if they diverge.
MIT — see LICENSE.