feat(analysis): step 7 dispatch — workflow input + EF time-series dispatcher

[MoLi7]

cc:
Closes:

What changed? Why?

Step 7 / Phase 1 — infra to dispatch the EF time-series across (scenario, approach, year) cells without YAML proliferation, with year-aligned GHG inventory data and Sheets-API-quota-safe serialization.

Workflow + config plumbing

Two new optional workflow inputs on generate_diagnostics.yml, mirrored as CLI options on bedrock/utils/validation/generate_diagnostics.py:

• model_base_year — overrides cfg.model_base_year. Drives the A-matrix scaling target year + inflation target year.
• usa_ghg_data_year — overrides cfg.usa_ghg_data_year. Drives the GHG (E) inventory year + gross-output x-vector year.

Both flow through set_global_usa_config's diagnostics_cli_overrides path. USAConfig.model_base_year Literal expands to {2019…2024} (was {2022, 2023, 2024}); usa_ghg_data_year Literal expands to {2019…2024} (was {2023, 2024}). Both keys added to DIAGNOSTICS_CLI_OVERRIDE_KEYS.

GHG FBS year templatization

load_E_from_flowsa() in bedrock/transform/allocation/derived.py previously hard-coded 'GHG_national_Cornerstone_2023' / 'GHG_national_CEDA_2023'. Templatizes the new_ghg_method and CEDA fallback branches with cfg.usa_ghg_data_year — those FBS YAMLs and GCS parquets exist for 2019–2023. Variant FBSes (*_coa_allocation, *_electricity, etc.) only exist for 2023, so the function raises a clear ValueError if any update_*_method flag is set with year ≠ 2023, instead of failing later with an opaque "FBS not found".

Workflow-level serialization

Add concurrency: { group: generate_diagnostics, cancel-in-progress: false } to generate_diagnostics.yml. Belt-and-suspenders against the Sheets API write quota (60/min/user) — guarantees only one generate_diagnostics job is writing to Sheets at a time, regardless of dispatch source.

Dispatch script

bedrock/analysis/a_matrix_time_series/dispatch_ef_time_series.py. Per (scenario, approach, year) cell:

1. Creates a Sheet in the v0.3 Diagnostics Drive folder with deterministic title [{run_date}, {model_year}, {baseline} based, {approach_label}, {scenario}] EFs diagnostics.
2. Triggers generate_diagnostics via gh workflow run with config_name, model_base_year, usa_ghg_data_year, sheet_id, use_useeio_baseline. Both year overrides get the same value per cell.
3. Appends a row to output/results/ef_run_index.csv (audit trail).

Idempotent — already-recorded cells are skipped, so re-running picks up only unfilled cells. CEDA-only baseline as the starting cut.

Two scenarios with explicit names:

• isolate_a_matrix — vary only A-matrix scaling, hold everything else to v0 defaults. Reuses the four Step 6 candidate YAMLs. Currently parked.
• bundle_v0_2 (default) — single config 2025_usa_cornerstone_full_model representing the full v0.2 release-candidate stack. 5 cells (1 approach × 5 years).

Throttle modes (--throttle):

• poll (default) — block until prior workflow runs clear via gh run list.
• sleep:N — fixed N-second sleep between triggers.
• none — fire immediately (only safe with bumped Sheets quota).

Re-dispatch from CSV (--re-dispatch-from-csv): re-trigger workflows for cells already in ef_run_index.csv — used to recover from rate-limit batch failures, re-uses existing Sheets.

Compile-script extension

compile_ef_diagnostics.py now tolerates the optional scenario/year columns. When populated, per-pair tab names get a {scenario}_{year}_ prefix and summary rows stamp the dimensions. Step 6's existing CSV (no scenario/year) still works — backfilled to empty strings on load.

Testing

• Dry-run dispatches the expected 5 cells for bundle_v0_2 × 2019–2023 with the agreed title format.
• compile_ef_diagnostics.py still compiles Step 6's existing 7-row index unchanged.
• black --check, ruff check ., mypy bedrock (385 files) all clean.
• Live dispatch run on this branch (post-merge of feat(analysis): step 6 phase 2/3 — EF diagnostics compile + plot scripts #379, with the new GHG-year wiring) currently in flight to validate end-to-end.

Pre-merge note

Live dispatch needs the workflow on the target ref. Either land this PR to main first and run with --git-ref main, or run with --git-ref mo__step7-ef-time-series-dispatch to dispatch against the branch (current state).

[MoLi7]

• Add LMDI signal-noise diagnosis for A-matrix physical residual #405
• feat(analysis): N time-series stability comparison across A-matrix methods #381
• feat(analysis): step 7 dispatch — workflow input + EF time-series dispatcher #380 👈 (View in Graphite)
• feat(analysis): step 6 phase 2/3 — EF diagnostics compile + plot scripts #379
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[WesIngwersen]

I'm getting a read error where its reading in year data from the tab for scenarios where its getting a float and expecting an int

Pulling tabs for scenario=bundle_v0_2, approach=commodity_price_index, year=2019.0, baseline=ceda
Traceback (most recent call last):
  File "c:\Users\ingwersw\bedrock\bedrock\analysis\a_matrix_time_series\compile_ef_diagnostics.py", line 295, in <module>
    main()
  File "c:\Users\ingwersw\bedrock\bedrock\analysis\a_matrix_time_series\compile_ef_diagnostics.py", line 231, in main
    joined, source_year=int(year), ref_year=REFERENCE_DOLLAR_YEAR
                        ^^^^^^^^^
ValueError: invalid literal for int() with base 10: '2019.0'
(bedrock) 

[WesIngwersen]

I'm getting a read error where its reading in year data from the tab for scenarios where its getting a float and expecting an int

Pulling tabs for scenario=bundle_v0_2, approach=commodity_price_index, year=2019.0, baseline=ceda
Traceback (most recent call last):
  File "c:\Users\ingwersw\bedrock\bedrock\analysis\a_matrix_time_series\compile_ef_diagnostics.py", line 295, in <module>
    main()
  File "c:\Users\ingwersw\bedrock\bedrock\analysis\a_matrix_time_series\compile_ef_diagnostics.py", line 231, in main
    joined, source_year=int(year), ref_year=REFERENCE_DOLLAR_YEAR
                        ^^^^^^^^^
ValueError: invalid literal for int() with base 10: '2019.0'
(bedrock) 

Addressed with 1a0b35c