Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/tickets/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,9 @@ Open bugs and improvements for figmark, numbered `T-NNN`.
| [T-075](T-075-truncated-descriptions-shared-as-complete.md) | **Closed** | Low | A truncated figure description is stored in the shared cross-request cache as if complete |
| [T-076](T-076-cache-operational-envelope-unenforced.md) | **Closed** | Medium | The cache's operational envelope is unenforced — no schema version, disk use beyond the cap, world-readable directory, undocumented scaling assumptions |
| [T-077](T-077-doc-drift-guard.md) | **Closed** | Medium | Hand-maintained docs drift from code — no guard catches a stale index, module map, or metadata |
| [T-078](T-078-vector-bar-charts-dropped.md) | Open | High | Vector bar and combo charts are silently dropped by the diagram detector (line charts only) |
| [T-078](T-078-vector-bar-charts-dropped.md) | **Closed** | High | Vector bar and combo charts are silently dropped by the diagram detector — resolved by T-080, verified live on FSR 28/28 |
| [T-079](T-079-raster-charts-get-alt-text-prompt.md) | Open | Medium | Charts embedded as raster images are described with the alt-text prompt, losing most of their information |
| [T-080](T-080-capture-regions-stop-classifying.md) | Open | High | figmark misses figures because the code tries to guess which regions are charts |
| [T-080](T-080-capture-regions-stop-classifying.md) | **Closed** | High | figmark misses figures because the code tries to guess which regions are charts — one-box-per-page, coverage 58/79→100 % |
| [T-081](T-081-structured-describe-output.md) | Open | High | The skip/keep decision rides on a free-text "[SKIP]" string the model can fail to emit |
| [T-082](T-082-description-quality-judge.md) | Open | Medium | No way to measure figure-description quality, only whether a figure was captured |
| [T-083](T-083-image-cap-should-be-config-model-validated.md) | Open | Medium | The image payload cap (1500 px / 500 KB) is a hardcoded guess, not validated against the model |
Expand Down
30 changes: 29 additions & 1 deletion docs/tickets/T-078-vector-bar-charts-dropped.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# T-078: Vector bar and combo charts are silently dropped by the diagram detector (line charts only)

**Status:** Open
**Status:** Closed — resolved by T-080's one-box-per-page extraction; resolution below.
**Priority:** High — figure interpretation is figmark's differentiator, and this
silently loses a whole class of charts. A dropped bar chart leaves only its
caption text; the data is gone, with no warning — the exact "silent degradation"
Expand Down Expand Up @@ -81,3 +81,31 @@ rescuing it.
not mis-detected as bar regions) — measured, not assumed.
- A dropped chart is never silent: at minimum a loud warning when a captioned
chart has no captured region.

## Resolution (2026-07-22, via T-080)

The root cause was the chart-classification gates themselves, and T-080 removed
them entirely: the detector no longer judges what geometry "looks like a chart"
— it unions each page's vector content into one box per band and lets the
vision model decide. There is no line-vs-bar distinction left to fail.

**Verified live** (Riksbank FSR 2025:1, gemma-4-31B via Berget, 2026-07-22):
all three repro charts are captured *and described correctly, in Swedish*:

| Chart | Page | Outcome |
|---|---|---|
| Diagram 10 — Svensk varuexport (stacked bar) | 27 | captured; export split per varugrupp described |
| Diagram 20 — Likviditetskvoter (grouped bar) | 37 | captured; LCR/NSFR bars described |
| Diagram 24 — Nettofondflöden (bar + line, dual axis) | 41 | captured; both axes and the time span described |

Coverage on the full document: **28/28 captions (100 %)**, no misses; cost
0.019 EUR for the full conversion. The controlled contrast is closed: bar
charts are now handled identically however the publisher embedded them.

**On the "loud gap warning" criterion (option 3):** not implemented, by
decision. The failure mode it guarded against — a classification gate silently
rejecting a chart — no longer exists, coverage is measured at 100 % across six
corpus documents (`scripts/coverage_bench/BASELINE.md`), and implementing the
warning would require caption parsing, which T-080 deliberately removed from
the extraction path. The coverage bench is the ongoing guard: run it when the
extraction changes.
42 changes: 39 additions & 3 deletions docs/tickets/T-080-capture-regions-stop-classifying.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# T-080: figmark misses figures because the code tries to guess which regions are charts

**Status:** Openrefactor validated on branch `feat/simpler-extraction-t080`
(commit saved), **blocked on T-081** before merge.
**Status:** Closedone-box-per-page shipped (with T-081's structured skip in
place). Resolution below.
**Priority:** High — figure interpretation is the product's differentiator, and
the coverage bench shows we silently drop ~1 in 4 figures.
the coverage bench showed we silently dropped ~1 in 4 figures.

## Symptom

Expand Down Expand Up @@ -82,3 +82,39 @@ That is why this is blocked on **T-081** (structured describe) before merge.
- `diagrams.py` is *simpler* (fewer constants, no `solid`/classification code),
not merely detuned.
- Cost increase measured and documented.

## Resolution (2026-07-22)

Shipped as **one box per page-band** (Option 3), with the open split question
resolved by bench:

- **Banding:** a page's filtered drawings group into maximal vertically
overlapping runs; adjacent bands merge unless a **body paragraph** sits in
the gap. Captions/source lines between two charts do not split — they belong
to the figure and land inside the rendered box.
- **Paragraph vs furniture** (the open question): word counts alone do NOT
separate them — BIS panel-title rows run 15–44 words. Bench-validated rule:
≥ 20 words AND ≥ 2 lines AND sentence punctuation (`.`/`,` — tick-label runs
like "RO CL MX …" never have it; junk leakage 107→0 words on BIS).
- **Paragraph guard (new, found by the bench):** a one-box band can
legitimately cover flowing text (column layouts, bridged bands) — measured
29/156/538 swallowed paragraph-words on BoC/BoJ/BIS even with per-band
boxes. `text_block_in_region` therefore never claims a body paragraph: mild
duplication beats silent deletion. Rescued prose: 831/2016/9350 words on
BoC/BoJ/BIS, junk 0.
- **Page furniture:** a drawing spanning ≥ 90 % of a page dimension (margin
rules, borders) is filtered — one 5 px margin bar otherwise bridged every
band on a table page into a full-page box.
- **Tables:** drawings the table path owns are excluded *before* banding. A
ruled table `find_tables` misses (the report's p70) is now captured for the
model instead of dropped — it addresses the T-050 symptom instead of
flattening to loose text.

**Geometric caption coverage** (extraction-side, `scripts/coverage_bench`
ground truth): BoC **58→100 %**, BoJ **79→100 %**, BIS/Fed/Riksbank PPR/
Riksbank FSR **100 %** — nothing missed on six docs. Region counts bounded:
35/54/81/64/44/28 — aggregate **301→284 vs main over seven docs** (BIS halves
as panels merge; BoC/Fed roughly double, exactly where main dropped figures).
T-078's three dropped FSR bar/combo charts (p27/37/41) all capture.

Live before→after numbers recorded in `scripts/coverage_bench/BASELINE.md`.
29 changes: 29 additions & 0 deletions scripts/coverage_bench/BASELINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,32 @@ produced 41 placements that dedupe to 6; one repeated header is correctly
skipped). The docx's open questions are *quality* (two logos over-described), not
coverage — so it belongs to the LLM-judge track, but it now has a real coverage
number instead of N/A.

## After T-080 — one-box-per-page extraction (2026-07-22, gemma-4-31B via Berget)

Same yardstick, same documents, live end-to-end runs with the T-080 band
extraction + T-081 structured skip:

| Document | Mode | Ground truth | Covered | Coverage | Captured | Skipped by model |
|---|---|---|---|---|---|---|
| boc-mpr-202410.pdf | caption | 26 captions | 26 | **100 %** (was 58 %) | 32 figs | 3 (cover/decoration) |
| boj-outlook-2410.pdf | caption | 58 captions | 58 | **100 %** (was 79 %) | 55 figs | 4 |
| riksbank-fsr-202505.pdf | caption | 28 captions | 28 | **100 %** (3 charts dropped before, T-078) | 38 figs | — |
| **Aggregate (caption)** | | **112** | **112** | **100 %** | | |

**Missed figures: none.**

**Cost** (per full conversion, live): BoC 37 calls / 46.8k tokens ≈ 0.016 EUR,
BoJ 61 calls / 86.1k tokens ≈ 0.030 EUR, FSR 40 calls / 56.7k ≈ 0.019 EUR. Aggregate regions across seven corpus
docs are 301→284 vs main (BIS AR halves as multi-panel figures merge into one
box; BoC/Fed roughly double — exactly where main dropped 1 in 4 figures), so
capture-100 % costs about the same as the old gated detector overall.

**Skip behaviour (T-081):** over-captured non-figures (cover decorations,
title-page rules) return `is_figure=false` and are cleanly skipped — 3/35 on
BoC, 4/59 on BoJ, no junk in the Markdown.

Geometric (extraction-side) coverage is also 100 % on bis-ar-2024,
fed-mpr-202407 and riksbank-ppr-202503. The three FSR bar/combo charts T-078
reported dropped (pages 27/37/41) are captured *and described correctly* in
the live FSR run — T-078 closes with this baseline.
Loading
Loading