Skip to content

Add sector mapping CSV, sector_label/use rollout, and validation#3459

Draft
louispt1 wants to merge 8 commits into
emissionsfrom
emissions-mapping
Draft

Add sector mapping CSV, sector_label/use rollout, and validation#3459
louispt1 wants to merge 8 commits into
emissionsfrom
emissions-mapping

Conversation

@louispt1

@louispt1 louispt1 commented Jul 8, 2026

Copy link
Copy Markdown
Member

Context

  • config/sector_mapping.csv: the canonical mapping from ETM sector labels to external classification schemes (IPCC CRT codes, klimaattafels, national sectors).
  • sector_label attribute added across energy and molecule node files, joined by (sector_label, use); use also added/aligned on molecule nodes. Stale per-node emissions_* groups removed now that mapping supersedes them.
  • spec/sector_mapping_spec.rb: validates every labelled node's (sector_label, use) pair exists in the mapping, the join key (sector_label == slug(sector)_slug(subsector)) hasn't drifted, emissions-store coverage per mapping pair × GHG, scheme names don't collide with emissions-store sector keys, and a static scan of gqueries/inputs for literal SECTOR/MSECTOR/scheme-form EMISSIONS calls against unknown schemes or values.
  • Bumps Atlas to pick up Atlas::SectorMapping.

Note: The spec is failing currently, presumably due to a mismatch with the mock sector_mapping file.

Related

atlas
etengine
etsource

Checklist

  • I have tested these changes
  • I have updated documentation as needed
  • I have tagged the relevant people for review

@kndehaan

kndehaan commented Jul 9, 2026

Copy link
Copy Markdown
Member

Nice work! The mapping seems to generally work well.

I updated the sector_mapping.csv to represent the correct one. An important change here is the removal of the "klimaattafel" mapping, as we want to first implement the ipcc crt code mapping; the klimaattafel mapping will be added in a later increment.

I think now some parts of the ETSource spec don't work as they should anymore. Could you have a look and update this? I think it's best to use the ipcc_crt_code_agg instead. (Sorry for forgetting to tell you that Klimaattafel will be added in a later increment). Perhaps etengine and atlas need to be checked as well?

@kndehaan

kndehaan commented Jul 9, 2026

Copy link
Copy Markdown
Member

I found some issues when a value of the columns contains white spaces:
For example, SECTOR(emissions_subsector, "Steel") works well, but for Electricity and heat production it gives this error:

image

Or another error example:
image

So it's something about handling the blank spaces that needs some fixing. @louispt1 could you look into this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants