Skip to content

Add UK raw-source and local-geography parity#156

Draft
MaxGhenis wants to merge 4 commits into
mainfrom
codex/uk-assigned-local-20260619
Draft

Add UK raw-source and local-geography parity#156
MaxGhenis wants to merge 4 commits into
mainfrom
codex/uk-assigned-local-20260619

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR moves the UK replacement build closer to full Populace ownership by adding a raw-source parity contract and the long local-geography path needed to replace the legacy matrix workflow.

  • Add a packaged UK source-stage manifest covering FRS, WAS, LCFS, ETB, NHS, SPI, FRS-only fill, capital gains, salary sacrifice, SLC student loans, national/local calibration, rail public-service calibration, road fuel/energy scaling, and row-wise OA/LA/constituency geography.
  • Move UK stage order, donor citations, and structural-stage grouping out of Python lists and into source_stages.json; Python now derives UK_STAGE_NAMES, UK_DONORS, and UK_STRUCTURAL_SOURCE_STAGES from the manifest.
  • Declare UK national and local calibration as Ledger-backed stages: read ledger_consumer_facts, compile uk_national_calibration / uk_local_geography target profiles, then calibrate weights against household metric tables.
  • Add assigned long local-geography matrix/solver/runner support so local weights export as (area_type, area_code, household_id, weight) rows rather than preserving the dense area-by-household matrix artifact.
  • Extend UK SPI support surfaces and tests for the SPI support channel and FRS-only fill variables.

Validation

  • uv run ruff check .
  • uv run --package populace-build --group dev python -m pytest packages/populace-build/tests/test_uk_source_manifest.py -q
  • uv run --package populace-build --group dev python -m pytest packages/populace-build/tests -q
  • python3 -m json.tool packages/populace-build/src/populace/build/uk/source_stages.json >/dev/null
  • git diff --check
  • Manifest source URL scan: all unique URLs returned HTTP 200, including https://github.com/PolicyEngine/arch-data
  • Incumbent/eFRS reference scan over touched UK/docs/test surfaces: no matches
  • Review cycle: independent read-only subagent review loop ended with no actionable findings remaining

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Catch-up update after merging current origin/main (b451bc7, through #221):

  • Kept the new spec-only country-package architecture. populace.build.uk is now a JSON resource package only; UK runtime imports live under populace.build.uk_runtime.
  • Registered uk/source_stages.json in uk/country_package.json and removed the old manifest-owned compile_ledger_targets / calibrate_weights stages. Ledger target-profile selection now stays in the shared UK local-target runtime; target values remain explicit build inputs.
  • Ported the recent UK-data / open-PR source-surface changes without depending on policyengine-uk-data:
    • employment_sector and sic_industry_division from raw FRS.
    • bus_fare_spending from LCFS detailed bus/coach fare codes c73212, c73213, c73214, plus DfT bus fare/subsidy amount anchors.
    • cash_isa and stocks_and_shares_isa from WAS, with stocks-and-shares ISA folded into corporate_wealth for back-compatibility.
  • Updated the UK source-manifest tests to assert the current exported surface, rewritten-output writers, absence of incumbent UK-data dependencies, and rejection of the old weight-calibration source operations.

Local verification:

  • uv run --package populace-build --group dev pytest packages/populace-build/tests -> 623 passed, 1 skipped.
  • uv run pytest packages/populace-build/tests packages/populace-calibrate/tests packages/populace-data/tests packages/populace-frame/tests -> 990 passed, 7 skipped, 2 PyTorch sparse warnings.
  • uv run --package populace-build --group dev ruff check packages/populace-build/tests/test_uk_source_manifest.py -> passed.
  • uv run --package populace-build --group dev ruff format --check packages/populace-build/tests/test_uk_source_manifest.py -> passed.
  • git diff --check -> clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant