You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
stop writing one-H5-per-state and one-H5-per-congressional-district artifacts from the US fiscal release builder
keep the release artifact surface to a single national populace_us_2024.h5 plus diagnostics, manifests, reform validation, demographics, and source coverage
reject US release manifests that try to publish states/ or districts/ artifact keys
remove the now-obsolete regional artifact writer CLI and tests
Calibration context
This follows the sparse-default 57k full-target-surface work. The completed full-surface selected-H5 run used 32,637 fiscal targets and 57,234 nonzero households, with:
initial loss: 0.045060887932777405
final loss: 0.04423309330088607
fraction within 10%: 0.7624168888071821
release gates: no failures in calibration_diagnostics.json
A national-only release rerun is in progress from the same 57,240-household selected H5 and reconstructed warm-start weights. It intentionally does not write state or district files; API and policyengine.py consumers should use the single national file and region filters.
Verification
git diff --check
uv run ruff check tools/build_us_fiscal_refresh_release.py packages/populace-data/src/populace/data/contract.py packages/populace-build/tests/test_us_fiscal_refresh_builder.py packages/populace-data/tests/test_contract.py
uv run ruff format --check tools/build_us_fiscal_refresh_release.py packages/populace-data/src/populace/data/contract.py packages/populace-build/tests/test_us_fiscal_refresh_builder.py packages/populace-data/tests/test_contract.py
uv run pytest packages/populace-build/tests/test_us_fiscal_refresh_builder.py packages/populace-data/tests/test_contract.py
Consumer spot checks on current upstreams:
policyengine-apiorigin/master: default US dataset handling already leaves default/national resolution to certified metadata/filters; focused tests passed.
policyengine.pyorigin/main: US state and congressional district regions already use dataset_path=None plus row filters on the national Populace file.
National-only release follow-up from the local run currently validating:\n\n- Output contract: single root artifact only so far (populace_us_2024.h5 plus calibration NPZ/diagnostics); no states/ or districts/ artifacts are being written.\n- Calibration diagnostics from out/sparse-default-release-20260701/refit-selected-current-surface-b788afc-national-only-20260701/releases/populace-us-2024-sparse-l0-refit-57k-b788afc-national-only-20260701/calibration_diagnostics.json:\n - initial_loss: 0.04423392191529274\n - final_loss: 0.04393288697745081\n - fraction_within_10pct: 0.7672886601096914\n - n_nonzero: 57196\n - gate_failures: null\n- Full target surface:\n - n_targets: 32637\n - n_records: 57240\n - constraint matrix: 32637 x 57240, 6186732 nonzero entries\n - surface sha256: bc3e2dc57edf7795da6cab776554806bb426ffd449377e36af7348b141f5b2f1\n - names sha256: be5a1fd8879e12df6e3c90884cf5cf8f1170f9b06f29d74c32720ae3bbcfe50b\n - values sha256: 1725eed9db538c89afe628317aa6d5616f60459477db3a2f49c27dfcba12922e\n\nPublication is still pending the slow post-calibration reform-validation/manifests phase. I am not treating this as live until reform_validation.json, demographics.json, us_source_coverage.json, build_manifest.json, and release_manifest.json exist and the publish contract passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
populace_us_2024.h5plus diagnostics, manifests, reform validation, demographics, and source coveragestates/ordistricts/artifact keysCalibration context
This follows the sparse-default 57k full-target-surface work. The completed full-surface selected-H5 run used 32,637 fiscal targets and 57,234 nonzero households, with:
0.0450608879327774050.044233093300886070.7624168888071821calibration_diagnostics.jsonA national-only release rerun is in progress from the same 57,240-household selected H5 and reconstructed warm-start weights. It intentionally does not write state or district files; API and
policyengine.pyconsumers should use the single national file and region filters.Verification
git diff --checkuv run ruff check tools/build_us_fiscal_refresh_release.py packages/populace-data/src/populace/data/contract.py packages/populace-build/tests/test_us_fiscal_refresh_builder.py packages/populace-data/tests/test_contract.pyuv run ruff format --check tools/build_us_fiscal_refresh_release.py packages/populace-data/src/populace/data/contract.py packages/populace-build/tests/test_us_fiscal_refresh_builder.py packages/populace-data/tests/test_contract.pyuv run pytest packages/populace-build/tests/test_us_fiscal_refresh_builder.py packages/populace-data/tests/test_contract.pyConsumer spot checks on current upstreams:
policyengine-apiorigin/master: default US dataset handling already leaves default/national resolution to certified metadata/filters; focused tests passed.policyengine.pyorigin/main: US state and congressional district regions already usedataset_path=Noneplus row filters on the national Populace file.