A pipeline of tools for matching library catalog (MARC) records to U.S. copyright registration and renewal data — the Library of Congress's Catalog of Copyright Entries, encoded by NYPL — to help discover public domain works.
pd-matcher matches library catalog (MARC) records to copyright registration and renewal data from the Library of Congress, as encoded by the New York Public Library — the Catalog of Copyright Entries (CCE).
The CCE and a MARC data describe the same underlying works but were never designed to join: standard numbers are rare on the CCE side, and titles, authors, publishers, and years drift between the two through OCR garbles, abbreviation conventions, and transcription errors. With no reliable shared identifiers, matching them is fuzzy work.
This tool attempts to do that matching at scale. It links each in-scope MARC record to its CCE registration and any renewal, and outputs a MARC ↔ CCE linkage with the registration and renewal facts attached along with any other standard and OCLC number it can find, plus a confidence score. It is a linkage producer, not a public-domain determiner: it surfaces the linkage and the facts, and a human or downstream consumer applies the copyright reasoning. Every published row should be human-verified.
Producing the linkage at scale needs a matcher good enough to trust on fuzzy pairs, and that matcher needs a human-verified ground-truth set to learn from and be measured against. The matcher and the ground-truth vault are the means to the linkage, not ends in themselves. They live in a two-CLI pipeline:
pd-matcher— the matching engine. It proposes candidate(MARC record, CCE registration, optional CCE renewal)triples with per-field scores and a confidence. The default combiner is a weighted mean over per-field scorers (no extra dependencies; optionally Platt-calibrated). A LightGBM learned combiner is also built and selectable with--scorer learned: train it withpd-matcher train-scorer(needs the optionalmlextra), and it emits a calibrated match probability directly. It outperforms the weighted mean on held-out separation — see docs/LEARNED_MATCHER.md.pd-groundtruth— the labeling subsystem. It turns proposals into a labeled corpus: it samples candidates across the score range, serves a local review UI, and persists every human verdict todata/training/label_vault.jsonl— the vault, source of truth.
The matcher's role is candidate surfacing, not direct publishing. Every published row is human-verified, so the matcher's mistakes only cost labeling throughput, not output quality. You point the pipeline at a MARC dump from your own institution; the matcher fits at any catalog that publishes MARCXML.
The published training data lives in a separate repo, jpstroop/cce-marc-linkage, pinned here as the data/training/ submodule. It holds exactly two files:
label_vault.jsonl— the vault itself: every adjudicated verdict (match / no_match / unsure) with the labeler's notes. The labeling UI writes here, so it is always current. This is the source of truth and the training labels for the learned matcher.marc.xml— MARCXML of every MARC the vault references (regenerated bydump-vault-marcs), so the pairs can be re-scored without the full candidate pool.
There is no separate reshaped/filtered table: a frozen matches list is only valid for one catalog, and the vault is the training table. See docs/USER_GUIDE.md for the publish flow.
This project does not decide public-domain status. Consumers apply whatever copyright reasoning they want — Cornell's decision matrix, the URAA restoration rules, country-of-origin analysis — to the verified linkage.
Pre-1.0. The pipeline runs end to end against the NYPL-transcribed CCE; it is developed and tested against one library's MARC dump so far, but nothing about it is institution-specific. Top-1 linkage precision/recall against the labeled vault is high (~99%), but that metric is in-sample and saturated; the operative measure now is pair-level separation — whether a score threshold can auto-decide a pair without a human. On a (deliberately hard) held-out sample the learned combiner reaches ROC-AUC ≈ 0.95 against the weighted mean's ≈ 0.94. The labeled vault (~2,000 verified pairs) is the bottleneck on training-set size and triage thresholds. The labeling subsystem is single-user and local; multi-user labeling behind OAuth is tracked at GitHub #34.
Requires Python 3.14+ (standard CPython — not the free-threaded t build) and PDM. asdf is recommended for managing the interpreter pin.
git clone --recurse-submodules <repo-url>
cd public_domain
pdm install
pdm run pre-commit installData comes in via git submodules. The labeled training bundle (the vault + the MARC it references) under data/training/ is pulled normally — --recurse-submodules gets it. The NYPL-transcribed CCE reference corpus under data/nypl-reg/ and data/nypl-ren/ (~1.5 GB) is lazy: --recurse-submodules does not pull it, so a casual clone stays small. You fetch it on demand only when you need to build the CCE index — see docs/USER_GUIDE.md for the command. If you forgot --recurse-submodules, git submodule update --init pulls data/training (but still not the lazy NYPL submodules).
The LightGBM combiner (--scorer learned, pd-matcher train-scorer) is an optional extra; plain pdm install skips it. The wheel is precompiled, but it needs an OpenMP runtime present when LightGBM is imported.
macOS — install the OpenMP runtime, then the extra:
brew install libomp
pdm install -G mlOn Apple Silicon, if import lightgbm still can't find libomp, point it at Homebrew's copy:
export DYLD_LIBRARY_PATH="$(brew --prefix libomp)/lib:$DYLD_LIBRARY_PATH"Linux — OpenMP (libgomp) usually ships with the gcc runtime; install it only if it's missing:
sudo apt-get install -y libgomp1 # Debian/Ubuntu (RHEL/Fedora: sudo dnf install libgomp)
pdm install -G mlVerify the extra loads:
pdm run python -c "import lightgbm; print(lightgbm.__version__)"(pdm install -G all pulls the acquire and review extras alongside ml.)
| If you want to… | Read |
|---|---|
| Run the pipeline (operator or labeler) | docs/USER_GUIDE.md — mental model, daily flows, troubleshooting |
| Label pairs | docs/LABELING_WORKFLOW.md (operational) + docs/LABELING_GUIDE.md (decision rules) |
| Understand the matching algorithm | docs/DESIGN.md |
| Understand candidate retrieval vs scoring | docs/MATCHING_ARCHITECTURE.md |
| Understand what a match says about copyright status | docs/COPYRIGHT_SCENARIOS.md |
| Train and use the learned (LightGBM) matcher | docs/LEARNED_MATCHER.md |
| Ship a code change | docs/PHASE_WORKFLOW.md |
| Look up a term | docs/GLOSSARY.md |
| Read past experiments | docs/studies/ (committed studies) and docs/findings/ (durable diagnostic records) |
| See active work | GitHub issues |
CLI flag reference is in --help on each command:
pdm run pd-matcher --help
pdm run pd-groundtruth --help- U.S. Copyright Office — Circular 23, "Copyright Office Records": https://www.copyright.gov/circs/circ23.pdf. Authoritative breakdown of which records exist for which years. Confirms that December 31, 1977 is the last day of registrations under the 1909 Copyright Act — the boundary the pipeline's coverage reflects.
- Internet Archive — copyright records collection: https://archive.org/details/copyrightrecords. OCR / scans of the same CCE volumes NYPL transcribed; useful as a human-readable cross-reference.
- Cornell University Library — "Copyright Term and the Public Domain in the United States": https://guides.library.cornell.edu/copyright/publicdomain. Reference matrix for downstream consumers applying copyright reasoning to the linkage dataset; this project does not encode it.
The pd-matcher source code is licensed under the GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later); see LICENSE. Distributed or network-deployed modifications must be released under the same license.
This license covers the code only, not the bundled data:
- CCE registration and renewal data are pulled in as NYPL-transcribed submodules; NYPL's transcriptions carry their own licenses.
- The underlying Catalog of Copyright Entries is a work of the U.S. Copyright Office and is in the public domain in the United States.
- Any MARC catalog you match against, and the ground-truth pairings you produce, are your own data under your own terms.