Filter repeating running heads and feet before segmentation#616
Draft
de-code wants to merge 5 commits into
Draft
Filter repeating running heads and feet before segmentation#616de-code wants to merge 5 commits into
de-code wants to merge 5 commits into
Conversation
related to eLifePathways/ScienceBeam2.0#61 Add a pre-segmentation noise filter that detects layout blocks repeating at the top or bottom of pages across a document (running heads, running feet) using position and cross-page text repetition. Detected blocks are excluded from the segmentation model input and preserved in the output XML as <note type="running-head"> / <note type="running-foot"> elements for auditability. Enabled by default via noise_filter_enabled in config.yml.
de-code
changed the title
ScienceBeam Parser EvaluationOverall (59 docs across 6 corpora)grobid 0.9.0-crf: 60 docs | sciencebeam-parser:main-83dec818-20260527.2241: 59 docs | sciencebeam-parser:pr-616-00ef481e-20260601.1137: 59 docs
biorxiv (9 docs)grobid 0.9.0-crf: 10 docs | sciencebeam-parser:main-83dec818-20260527.2241: 9 docs | sciencebeam-parser:pr-616-00ef481e-20260601.1137: 9 docs
ore (10 docs)grobid 0.9.0-crf: 10 docs | sciencebeam-parser:main-83dec818-20260527.2241: 10 docs | sciencebeam-parser:pr-616-00ef481e-20260601.1137: 10 docs
pkp (10 docs)grobid 0.9.0-crf: 10 docs | sciencebeam-parser:main-83dec818-20260527.2241: 10 docs | sciencebeam-parser:pr-616-00ef481e-20260601.1137: 10 docs
scielo_br (10 docs)grobid 0.9.0-crf: 10 docs | sciencebeam-parser:main-83dec818-20260527.2241: 10 docs | sciencebeam-parser:pr-616-00ef481e-20260601.1137: 10 docs
scielo_mx (10 docs)grobid 0.9.0-crf: 10 docs | sciencebeam-parser:main-83dec818-20260527.2241: 10 docs | sciencebeam-parser:pr-616-00ef481e-20260601.1137: 10 docs
scielo_preprints-jats (10 docs)grobid 0.9.0-crf: 10 docs | sciencebeam-parser:main-83dec818-20260527.2241: 10 docs | sciencebeam-parser:pr-616-00ef481e-20260601.1137: 10 docs
|
Collaborator
Author
|
For |
Rewrite the noise block classifier to avoid hardcoded 0.2/0.8 page fractions. Instead, the top/bottom quartile of each page's own block y-distribution defines the noise zone, so the threshold adapts to each document's layout. Two additional guards prevent false positives: - Stddev check: position must be stable across occurrences - Height check: occurrences whose block height exceeds 2× the group median are not filtered (catches a large title on page 1 that also repeats as a small footer on pages 2+, as seen in scielo_br)
Collaborator
Author
|
Not having looked at the validation dataset. But there are two cases for the training dataset where the PDF doesn't match the JATS document:
Both are In those two cases, it is now extracting title as English + Portuguese, whereas before it was English only. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
related to https://github.com/eLifePathways/ScienceBeam2.0/issues/61
Add a pre-segmentation noise filter that detects layout blocks whose text repeats at the top or bottom of pages across a document. Detected blocks are excluded from the segmentation model input and preserved in the output XML as / elements for auditability.
Enabled by default via noise_filter_enabled in config.yml.