UK dashboard parity: migrate UK dataset production fully to populace, plus UK-independent dashboard fixes

[juaristi22]

The UK side of the dashboard (/populace with the UK toggle) renders visibly worse than the US side: a blank calibration map when any geography option is selected, empty Geography/Level columns and filters, a per-family fit table with one target per row, "—" KPI cards, and an empty loss-trajectory section. We investigated where the UK data actually comes from and why its shape differs from the US. Root cause splits in two: the UK release artifacts are produced by a bespoke, uncommitted pipeline whose logging structure doesn't match the US, and, independently, a handful of dashboard bugs that bite the UK regardless of logging.

Primary fix: migrate UK dataset production fully to populace

Who publishes what today (investigated 2026-07-03 against PolicyEngine/populace@2c93936 and PolicyEngine/policyengine-uk-data@6a0acf9):

• policyengine/populace-uk-private on HF is published by PolicyEngine/populace — publish_release() in packages/populace-data/src/populace/data/release.py via the populace-publish-release CLI / tools/publish_release.sh, run manually (no publishing CI). The UK dataset is registered in packages/populace-data/src/populace/data/registry.py#L98-L109.
• policyengine-uk-data is legacy for publishing (its CI only uploads enhanced-FRS artifacts to policyengine-uk-data-private; see populace#144 / populace#41), but it still supplies the UK target surface: the ons/..., slc/..., obr/... target names come from policyengine_uk_data/targets/, pinned as target_registry.version = "policyengine-uk-data-dd68c73".
• The UK diagnostics writer is not committed anywhere. The US has a canonical writer (packages/populace-calibrate/src/populace/calibrate/diagnostics.py, now schema v4, contract-enforced at publish time by contract.py). The UK's calibration_diagnostics.json / build_manifest.json were written by a local harness on the build machine: org-wide code search finds zero hits for its distinctive strings (pre_calibration, uk_surface_candidate, calibrated_households, release_surface, …), and the SHAs embedded in the UK release ids (dd68c73, 4aa4b14, 72aeefc) resolve to no commit on any PolicyEngine remote — the builds are unauditable and would fail today's publish contract.

Because of that, the UK artifact is structurally leaner than the US one, and the dashboard has nothing to read for whole sections:

Field	US (populace.calibrate.diagnostics)	UK (bespoke local writer)
Per-target metadata (ledger_* fields, source_measure_id, base_variables)	Full dict per row	null on every row
Per-target registry (family, se, notes)	Present	null
n_records, n_nonzero	Top-level ints	Absent
initial_loss, final_loss, fraction_within_10pct	Top-level scalars	Absent
loss_trajectory	Per-epoch numeric list	Two checkpoints: [{step: "pre_calibration", relative_loss}, …]
options	Solver options (method/epochs/learning_rate/…)	Build metadata (calibrated_households, candidate_surface, reference_surface, max_abs_relative_error_gate, year)
build_manifest.gates	calibration, target_compilation, target_profile_coverage, …	Only exported_nonzero, parity, release_surface
Target registry identity	Content-addressed populace TargetRegistry	Repo-pin string to an unpushed commit

Proposed migration (so the UK publishes the same structure as the US and the dashboard needs no UK special-casing):

1. Port the UK target surface from policyengine_uk_data/targets/ into a populace.calibrate.TargetRegistry (declarative TargetSpecs with value/se/citation/family/metadata) — populace#147 already tracks this. Per-row metadata/registry and a content-addressed registry version then come for free.
2. Run the UK solve through populace.calibrate.solve.calibrate (or wrap the existing solver to return a CalibrationResult) and write diagnostics with write_calibration_diagnostics(...) — yielding numeric per-epoch loss_trajectory, n_records/n_nonzero, initial_loss/final_loss/fraction_within_10pct, real solver options, and skipped, with zero UK-specific code.
3. Move the surface-comparison inputs (candidate_surface/reference_surface/max_abs_relative_error_gate) out of options into the diagnostics build block and build_manifest.gates (the committed export_surface/target_surface gates from populace#39), alongside a target_compilation gate.
4. Commit and push a UK build driver (a tools/build_uk_release.py sibling of build_us_fiscal_refresh_release.py) and build from pushed commits, keeping publishing on populace-publish-release --repo-id policyengine/populace-uk-private so the schema-v4 contract is enforced for the UK exactly as for the US.

Also important: dashboard bugs independent of logging

These bite the UK today regardless of the artifact structure, and are needed for full dashboard operability. We're fixing these dashboard-side now (the migration above is the eventual upstream fix):

1. Calibration-map geography filter hardcodes US levels (National / State / Congressional district). UK rows have no matching level, so selecting any option blanks the map; the selection also went stale across country switches. Fix open: Make the calibration-map geography filter data-driven (fixes blank UK map) #69 (data-driven levels from the release, select hidden when a release has no levels).
2. Target-name parsing only understands US naming (FIPS/state patterns in frontend/lib/populace/latest-artifact.ts). UK names like ons/west_midlands_age_80_89 and ons/household_land_value/LONDON fall through with empty geography/level/breakdown and no facets — blank Geography columns, no Geography/Level filters, a per-family fit table of singletons (138 "families" for 149 targets), and a fragmented treemap (region and age band stay baked into the variable name). Needs a UK-aware parser (region + age-band extraction, national/region levels).
3. Loss-trajectory shape mismatch: the loader casts loss_trajectory to number[]; the UK's object-shaped checkpoints render as an empty "Loss trajectory" section instead of a before/after readout.
4. "Calibration details" panel renders empty for UK: the options grid only knows US solver keys, and the "Declared/compiled/dropped" line shows em-dashes when gates.target_compilation is absent.
5. KPI cards show "—" for derivable stats: "Within 10% of target" can be computed from the target rows (the targets page already does; UK is at ~98%), and "Records kept" could fall back to options.calibrated_households.
6. /api/populace/target-investigation ignores country — investigating a UK target queries the US dataset.
7. Cosmetic: the sidebar labels the UK dataset policyengine/populace-uk while the data is read from policyengine/populace-uk-private.

Items 2–7 are dashboard-side and don't depend on the migration; items 3–5 become moot once the UK artifact matches the US schema, but graceful degradation is still worth having for older releases.

🤖 Generated with Claude Code