DB056 – Finalise Ingredient Data Quality

[barbie-jan]

Ticket

• DB056 – Finalise Ingredient Data Quality

Description

Reviewed the release dataset's ingredient fields against the actual cleaning code and found a real, release-critical bug: cleanProductData.py's ingredient cleaning checked for the post-rename camelCase column names (ingredientsText, ingredients) before the rename step actually happens, so clean_ingredients_text()/clean_ingredients_list() and a per-record safety-net pass silently never ran against the real data.

Confirmed with real evidence, not a hypothetical: the committed sample shows the Tuna product's empty ingredients_text landing in the cleaned output as "ingredientsText": "" instead of null — exactly the kind of inconsistent ingredient data this ticket asks to fix.

Fix: two small column-name corrections in database/clean_data/cleanProductData.py — no change to the cleaning logic itself, just fixing when it runs, so it actually operates on the real (still snake_case) column names at that point in the pipeline instead of silently creating a duplicate, blank shadow column.

Verified safe:

• detect_allergens() already has a defensive ingredientsText or ingredients_text fallback — needs no change, and now transparently receives cleaned instead of raw text.
• missing_field_handler.py already treats ingredients_text as a critical field alias — missing ingredient data is correctly flagged, not a pipeline failure.
• normalize_string/normalize_list are None-safe by design.

Documented, not fixed (out of ingredient scope): the same wrong-column-name bug class exists for productQuantity/servingQuantity — flagged as a follow-up ticket rather than opportunistically fixed here.

Full detail, before/after evidence, and the acceptance-criteria checklist are in Documents/Database/2026 Trimester 1/DB056-Ingredient-Data-Quality-Finalisation.md.

Files changed

• database/clean_data/cleanProductData.py (bug fix — two column-name corrections)
• test/test_ingredient_cleaning_db056.py (new — 3 regression tests)
• Documents/Database/2026 Trimester 1/DB056-Ingredient-Data-Quality-Finalisation.md (new)

Checks

• All requirements of the ticket have been implemented, or I have commented on any exclusions
• Unit tests have been added or updated for any backend changes (if applicable) — test/test_ingredient_cleaning_db056.py, 3 new tests
• I have reviewed the Files Changed tab and verified it only contains relevant changes (comment if unsure about any)
• This PR has been reviewed and approved

Screenshots

N/A — backend data-cleaning fix, no frontend/UI involved. Test output requested below as evidence instead.

How to verify

pytest test/test_ingredient_cleaning_db056.py -v
pytest test/test_clean_product_data.py test/test_validate_cleaned_dataset.py -v

The first command should show all 3 new tests passing, in particular confirming the Tuna product's ingredientsText is now null rather than "". The second confirms no regressions in existing cleaning/validation tests.