Add new q deriviation and compare q's against references

[WesIngwersen]

cc: @MoLi7
Closes: #457

Summary

Adds a new q-estimation pathway — scaling the 2017 Cornerstone Make matrix to match BEA annual gross industry output — and an analysis script that compares it against the two existing pathways at both Cornerstone detail and BEA summary level.

NOTE This PR does not implement the new q-estimation pathway into any model builds - it just creates the pathway and compares it with existing pathways.

New functions in derived_cornerstone.py

• scale_cornerstone_V_with_authoritative_x(): inflates the base-2017 V matrix to model_base_year (optionally applying commodity PI inflation), then row-scales each industry so that V's row sums exactly match derive_cornerstone_x_after_redefinition(model_base_year) — the BEA annual gross industry output time series. The result is a Make matrix consistent with authoritative BEA output totals.
• derive_q_from_scaled_cornerstone_V_from_authoritative_x(): thin wrapper that calls the above and returns commodity q = V.sum(axis=0).
• derive_cornerstone_x_after_redefinition gains an optional year parameter (defaults to usa_ghg_data_year when omitted, preserving existing behavior) so callers can request a specific year without re-reading config.

New analysis script: bedrock/analysis/q/compare_alt_q.py

Compares three q-estimation pathways at the 405-sector Cornerstone detail level and aggregated to BEA summary level:

1. V_scaled — derive_q_from_scaled_cornerstone_V_from_authoritative_x() (this branch)
2. Aq_scaled — replicates scale_a_matrix_with_summary_tables=True + adjust_summary_A_and_q_dollar_year=True by calling adjust_summary_q_dollar_year() and inflate_cornerstone_q_or_y_with_commodity_pi() directly, independent of the active config
3. inflated — inflate_cornerstone_q_or_y_with_commodity_pi() applied to the base scaled q without the dollar-year rebase step

Reference benchmarks: derive_cornerstone_x_after_redefinition(model_year) (authoritative BEA x) and derive_summary_q_usa(model_year) (summary Make table q).

Outputs three matplotlib plots to bedrock/analysis/q/output/plots/:

• detail_scatter_{year}.png — 1×4 log-log scatter at Cornerstone detail level; panels 1–3 show each pathway vs authoritative x; panel 4 is a 2017 reference showing compute_q(V) vs compute_x(V) from the unmodified Make matrix
• summary_levels_{year}.png — ranked line plot of all five series at BEA summary level
• summary_pct_deviations_{year}.png — grouped bar chart of % deviation from each reference benchmark

Tests

test_scale_V_from_authoritative_x.py verifies that scale_cornerstone_V_with_authoritative_x() row sums match the authoritative x vector to within 1e-6 relative tolerance.

Analysis results

Aggregate totals at summary level ($M):
x_authoritative 48,424,233,164,693
q_make_table 48,366,332,000,000
V_scaled 48,424,233,164,693
Aq_scaled 48,307,281,448,622
inflated 42,699,140,375,411

Overall pct deviation from Make Table q:
V_scaled +0.120%
Aq_scaled -0.122%
inflated -11.717%

Overall pct deviation from x_authoritative:
V_scaled -0.000%
Aq_scaled -0.242%
inflated -11.823%

[WesIngwersen]

I'm fixing the type error(s)

[bl-young]

I'm curious about the choice of usa_ghg_data_year instead of model_year

[WesIngwersen]

That is an old choice to make it work for the useeio model... not related to this PR except a param was added to be able to override it. It does need to be relooked at maybe in an issue outside this PR.

[WesIngwersen]

@bl-young I added the text of the analysis run which includes additional summaries but note one aggregation error occuring in
WARNING __main__: aggregate_q_to_summary: 1 sector(s) not in cs_to_summary and will be dropped (total $M 8078560285): ['331314']

[WesIngwersen]

@bl-young I added the text of the analysis run which includes additional summaries but note one aggregation error occuring in WARNING __main__: aggregate_q_to_summary: 1 sector(s) not in cs_to_summary and will be dropped (total $M 8078560285): ['331314']

I'm working on this...

[WesIngwersen]

@bl-young I added the text of the analysis run which includes additional summaries but note one aggregation error occuring in WARNING __main__: aggregate_q_to_summary: 1 sector(s) not in cs_to_summary and will be dropped (total $M 8078560285): ['331314']

I'm working on this...

331314 industry output is getting lost in the industry output mapping to commodity output for the comparison and the totaling.

[jvendries]

Bug fix: derived.py
load_E_from_flowsa referenced methodname (a local string extracted from a data column) as the condition for a GWP override instead of the config flag usa.use_ghg_national_2023_m2. This caused a ValueError when methodname was unassigned for certain data slices.

I think this was fixed in #454

[WesIngwersen]

Reference

Claude did this. I guess the fix is identical character-by-character so no merge conflict is showing up, so easier just to leave it it in code again but I removed it from the description of the changes in this PR.

[WesIngwersen]

@bl-young I added the text of the analysis run which includes additional summaries but note one aggregation error occuring in WARNING __main__: aggregate_q_to_summary: 1 sector(s) not in cs_to_summary and will be dropped (total $M 8078560285): ['331314']

I'm working on this...

331314 industry output is getting lost in the industry output mapping to commodity output for the comparison and the totaling.

Fixed to assign it temporarily to 331313 only in the scope of this script. Updated results. Now scaled V approach q sum = authoritative x sum.

[jvendries]

@WesIngwersen thanks for doing this analysis and sharing the results, it makes the differences between approaches much easier to see.

One piece still missing for a fully informed comparison would be EF diagnostics (D/N) for the V_scaled and Aq_scaled pathways (since the latter is not quite the same as the approach for EFs we've already seen if I'm not mistaken). While aggregate q totals for V_scaled and Aq_scaled are close (~0.1–0.2%), I don't have a good sense of how detail-sector EFs might differ between them. Since V_scaled isn't yet part of a model build I imagine this would have to be a follow-on analysis.

Beyond EF impacts, each approach seems to have a distinct main benefit:

• V_scaled: Would allow for greater internal consistency. x, q, and A could be derived from the same scaled Make matrix, rather than B using authoritative x while q/A follow a separate scaling path.
• Aq_scaled: Continuity with the summary-table scaling framework we are currently using (even the calculation has been changed a bit via adjust_summary_q_dollar_year)
• inflation: Simplest to explain and communicate though with the largest aggregate error by a large margin if used on its own.

If EF diagnostics show relatively little difference between V_scaled and Aq_scaled at detail sector level, prioritizing among these benefits might be a reasonable way to choose an approach.

[WesIngwersen]

@WesIngwersen thanks for doing this analysis and sharing the results, it makes the differences between approaches much easier to see.

One piece still missing for a fully informed comparison would be EF diagnostics (D/N) for the V_scaled and Aq_scaled pathways (since the latter is not quite the same as the approach for EFs we've already seen if I'm not mistaken). While aggregate q totals for V_scaled and Aq_scaled are close (~0.1–0.2%), I don't have a good sense of how detail-sector EFs might differ between them. Since V_scaled isn't yet part of a model build I imagine this would have to be a follow-on analysis.

Beyond EF impacts, each approach seems to have a distinct main benefit:

• V_scaled: Would allow for greater internal consistency. x, q, and A could be derived from the same scaled Make matrix, rather than B using authoritative x while q/A follow a separate scaling path.
• Aq_scaled: Continuity with the summary-table scaling framework we are currently using (even the calculation has been changed a bit via adjust_summary_q_dollar_year)
• inflation: Simplest to explain and communicate though with the largest aggregate error by a large margin if used on its own.

If EF diagnostics show relatively little difference between V_scaled and Aq_scaled at detail sector level, prioritizing among these benefits might be a reasonable way to choose an approach.

@jvendries I will do a full N comparison in a future PR.