Skip to content

Fail immediately and emit a relevant error if column mappings are missing from config.json#595

Open
kmoscoe wants to merge 9 commits into
datacommonsorg:masterfrom
kmoscoe:errors
Open

Fail immediately and emit a relevant error if column mappings are missing from config.json#595
kmoscoe wants to merge 9 commits into
datacommonsorg:masterfrom
kmoscoe:errors

Conversation

@kmoscoe

@kmoscoe kmoscoe commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Again, this is a bit of a stopgap, but the current logic makes no sense, i.e. defaulting the expected columns to variable, entity, date, and value, and then later raising an error saying that "entity" etc. are missing from the column headings. Since this is a major breaking change, it really needs a proper enforcement check and a meaningful error.

Let me know if I should add some tests for this. (It might not be a bad idea to add some DCP config.json tests in general since it has changed so much from CDC.)

@kmoscoe kmoscoe requested a review from gmechali June 26, 2026 03:19
@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity

Metric Results
Complexity 0

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a check to fail immediately with a ValueError if column mappings are missing from the configuration during observations import in DCP_BRIDGE mode, and updates the corresponding tests. The review feedback suggests improving the error message by including the path of the file missing the mappings, and removing redundant columnMappings configurations for MCF files in the tests since MCF files do not have columns.

Comment thread simple/stats/runner.py Outdated
Comment thread simple/tests/stats/runner_test.py Outdated
Comment thread simple/tests/stats/runner_test.py Outdated
@kmoscoe kmoscoe removed the request for review from gmechali June 26, 2026 03:27
@kmoscoe kmoscoe requested review from gmechali and removed request for gmechali June 26, 2026 03:35
@kmoscoe kmoscoe requested a review from gmechali June 26, 2026 04:09

@gmechali gmechali left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ll approve this but we will be refactoring all this shortly for the proper long term behavior here. So no need to do more of these stop gaps. Thanks Kara

@kmoscoe

kmoscoe commented Jun 26, 2026 via email

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants