Skip to content

Publish US releases as a single national H5#237

Merged
MaxGhenis merged 1 commit into
mainfrom
codex/national-only-populace-release-20260701
Jul 1, 2026
Merged

Publish US releases as a single national H5#237
MaxGhenis merged 1 commit into
mainfrom
codex/national-only-populace-release-20260701

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Summary

  • stop writing one-H5-per-state and one-H5-per-congressional-district artifacts from the US fiscal release builder
  • keep the release artifact surface to a single national populace_us_2024.h5 plus diagnostics, manifests, reform validation, demographics, and source coverage
  • reject US release manifests that try to publish states/ or districts/ artifact keys
  • remove the now-obsolete regional artifact writer CLI and tests

Calibration context

This follows the sparse-default 57k full-target-surface work. The completed full-surface selected-H5 run used 32,637 fiscal targets and 57,234 nonzero households, with:

  • initial loss: 0.045060887932777405
  • final loss: 0.04423309330088607
  • fraction within 10%: 0.7624168888071821
  • release gates: no failures in calibration_diagnostics.json

A national-only release rerun is in progress from the same 57,240-household selected H5 and reconstructed warm-start weights. It intentionally does not write state or district files; API and policyengine.py consumers should use the single national file and region filters.

Verification

  • git diff --check
  • uv run ruff check tools/build_us_fiscal_refresh_release.py packages/populace-data/src/populace/data/contract.py packages/populace-build/tests/test_us_fiscal_refresh_builder.py packages/populace-data/tests/test_contract.py
  • uv run ruff format --check tools/build_us_fiscal_refresh_release.py packages/populace-data/src/populace/data/contract.py packages/populace-build/tests/test_us_fiscal_refresh_builder.py packages/populace-data/tests/test_contract.py
  • uv run pytest packages/populace-build/tests/test_us_fiscal_refresh_builder.py packages/populace-data/tests/test_contract.py

Consumer spot checks on current upstreams:

  • policyengine-api origin/master: default US dataset handling already leaves default/national resolution to certified metadata/filters; focused tests passed.
  • policyengine.py origin/main: US state and congressional district regions already use dataset_path=None plus row filters on the national Populace file.

@MaxGhenis MaxGhenis merged commit 673da1c into main Jul 1, 2026
4 checks passed
@MaxGhenis MaxGhenis deleted the codex/national-only-populace-release-20260701 branch July 1, 2026 13:29
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

National-only release follow-up from the local run currently validating:\n\n- Output contract: single root artifact only so far (populace_us_2024.h5 plus calibration NPZ/diagnostics); no states/ or districts/ artifacts are being written.\n- Calibration diagnostics from out/sparse-default-release-20260701/refit-selected-current-surface-b788afc-national-only-20260701/releases/populace-us-2024-sparse-l0-refit-57k-b788afc-national-only-20260701/calibration_diagnostics.json:\n - initial_loss: 0.04423392191529274\n - final_loss: 0.04393288697745081\n - fraction_within_10pct: 0.7672886601096914\n - n_nonzero: 57196\n - gate_failures: null\n- Full target surface:\n - n_targets: 32637\n - n_records: 57240\n - constraint matrix: 32637 x 57240, 6186732 nonzero entries\n - surface sha256: bc3e2dc57edf7795da6cab776554806bb426ffd449377e36af7348b141f5b2f1\n - names sha256: be5a1fd8879e12df6e3c90884cf5cf8f1170f9b06f29d74c32720ae3bbcfe50b\n - values sha256: 1725eed9db538c89afe628317aa6d5616f60459477db3a2f49c27dfcba12922e\n\nPublication is still pending the slow post-calibration reform-validation/manifests phase. I am not treating this as live until reform_validation.json, demographics.json, us_source_coverage.json, build_manifest.json, and release_manifest.json exist and the publish contract passes.

This was referenced Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant