BE034

contracts/REASON_CODE_CATALOGUE_V1.md

@@ -0,0 +1,72 @@
+# Recommendation Reason-Code Catalogue (v1.0.0)
+
+**Contract Version:** 1.0.0  
+**Scope:** Machine-readable reason codes returned in `substitutions[].reasonCodes` for the Food Remedy API.
+
+---
+
+## 1. Overview
+
+Reason codes provide standardized, language-agnostic machine tags explaining why a candidate product was recommended as a substitute. Mobile clients use these codes to render localized badge icons, filter suggestions, or generate formatted UI text.
+
+---
+
+## 2. Standard Reason Codes
+
+### 2.1 Category Alignment Codes
+
+| Code | Description |
+| :--- | :--- |
+| `MATCH_CATEGORY_EXACT` | Candidate product shares the exact primary category as the target product. |
+| `MATCH_CATEGORY_SUBSTRING` | Candidate product belongs to a related sub-category or category hierarchy match. |
+
+---
+
+### 2.2 Safety & Dietary Alignment Codes
+
+| Code | Description |
+| :--- | :--- |
+| `SAFE_ALLERGEN_FREE` | Candidate product is verified free from all allergens flagged in user profile/overrides. |
+| `DIET_ALIGNED_VEGAN` | Candidate product is verified compliant with Vegan dietary restrictions. |
+| `DIET_ALIGNED_VEGETARIAN` | Candidate product is verified compliant with Vegetarian dietary restrictions. |
+| `DIET_ALIGNED_GLUTEN_FREE` | Candidate product is verified Gluten-Free. |
+
+---
+
+### 2.3 Nutritional Improvement Codes
+
+| Code | Description |
+| :--- | :--- |
+| `BETTER_NUTRI_SCORE` | Candidate product has a superior Nutri-Score grade (e.g. Grade A vs Grade C). |
+| `LOWER_SUGAR` | Candidate product contains significantly lower sugar per 100g. |
+| `LOWER_SODIUM` | Candidate product contains significantly lower sodium/salt per 100g. |
+| `LOWER_SATURATED_FAT` | Candidate product contains lower saturated fat per 100g. |
+| `HIGHER_FIBER` | Candidate product contains higher dietary fiber per 100g. |
+| `HIGHER_PROTEIN` | Candidate product contains higher protein per 100g. |
+
+---
+
+## 3. Example Usage in API Response Payload
+
+```json
+{
+  "barcode": "9300601234567",
+  "productName": "Oatly Barista Oat Milk 1L",
+  "brand": "Oatly",
+  "nutriscoreGrade": "a",
+  "safetyRating": "green",
+  "confidenceScore": 0.92,
+  "reasonCodes": [
+    "MATCH_CATEGORY_EXACT",
+    "SAFE_ALLERGEN_FREE",
+    "DIET_ALIGNED_VEGAN",
+    "BETTER_NUTRI_SCORE"
+  ],
+  "reasons": [
+    "Exact category match (Plant-based milk)",
+    "Verified free from milk allergens",
+    "Fully aligned with Vegan diet",
+    "Nutri-Score A (Better than Grade C)"
+  ]
+}
+```

contracts/RECOMMENDATION_ELIGIBILITY_POLICY_V1.md

@@ -0,0 +1,67 @@
+# Recommendation Substitution Eligibility & Missing-Data Policy (v1.0.0)
+
+**Document Version:** 1.0.0  
+**Effective Date:** September 2026  
+**Applies To:** Backend API, Mobile Client, Scan Service Consumers  
+
+---
+
+## 1. Executive Summary
+
+This policy governs the eligibility of food product candidates for substitution recommendations in the **Food Remedy API**. The primary directive of this system is **fail-safe consumer protection**: **under no circumstances may an unsafe or unverified product candidate be presented to a user as a safe substitute.**
+
+---
+
+## 2. Universal Eligibility Rules
+
+### 2.1 Hard Exclusion Rules (Unsafe Candidates)
+A candidate product is **hard-excluded** (dropped completely from substitution results) if any of the following apply:
+1. **Direct Allergen Match**: The candidate contains any allergen listed in the user's active profile or request-level `avoidAllergens` overrides.
+2. **Traces / Cross-Contamination Warning**: The candidate lists traces of an allergen flagged as severe in the user profile.
+3. **Violated Dietary Restriction**: The user specifies a mandatory diet (e.g., `vegan`, `vegetarian`) and the candidate fails compliance (e.g. contains animal derivatives).
+
+### 2.2 Fail-Safe Rule for Unknown or Insufficient Data
+If evidence regarding a product's safety is missing, unparsed, or incomplete:
+1. **Unknown Allergen Status**: If candidate product has `allergens = null` or `ingredients = []`, it **CANNOT** be classified as `"green"` (safe).
+2. **Downgrade to Grey / Exclusion**:
+   - If the user profile has active allergen restrictions, candidates with unknown allergen status are **hard-excluded**.
+   - If the user has no severe allergen restrictions, the candidate is restricted to `"grey"` (Acceptable with Caution) and penalized in score.
+3. **Missing Category Data**: If the original scanned product or candidate product lacks category tags, category similarity matching cannot be verified. Status is set to `INSUFFICIENT_PRODUCT_DATA`.
+
+---
+
+## 3. Privacy & Profile Non-Leakage
+
+To prevent leaking sensitive user health data (e.g., specific medical allergies or dietary conditions) via network response logs:
+- Profile matching is evaluated securely on the backend server.
+- The API response payload **MUST NOT** echo back the user's full nutritional profile or sensitive allergen preferences.
+- Only non-sensitive, machine-readable reason codes (e.g., `SAFE_ALLERGEN_FREE`, `DIET_ALIGNED_VEGAN`) are returned.
+
+---
+
+## 4. Empty-State Taxonomy
+
+When an API call yields zero substitution candidates, the response MUST distinguish between data gaps and true absence of alternatives via `emptyStateReason`:
+
+| Reason Code | Trigger Condition |
+| :--- | :--- |
+| `INSUFFICIENT_PRODUCT_DATA` | Scanned target product lacks necessary categories, nutrients, or ingredient tags to find substitutes. |
+| `NO_SAFE_ALTERNATIVES_IN_CATEGORY` | Target product category is valid, but no higher-scoring alternatives exist in catalogue. |
+| `STRICT_ALLERGEN_EXCLUSION_ALL_CANDIDATES` | Candidate alternatives exist, but 100% were excluded due to strict allergen safety rules. |
+| `PRODUCT_NOT_FOUND` | Scanned barcode does not exist in product database. |
+
+---
+
+## 5. Sanitized Error Envelopes
+
+All client validation errors (e.g., invalid barcodes, out-of-range limits) return standardized HTTP status codes and sanitized JSON envelopes:
+
+```json
+{
+  "error": {
+    "code": "INVALID_BARCODE",
+    "message": "Barcode must be a valid numeric string between 8 and 14 digits.",
+    "details": null
+  }
+}
+```

contracts/recommendation_substitutions_v1.schema.json

@@ -0,0 +1,146 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "https://food-remedy.dev/schemas/recommendation-substitutions/v1",
+  "title": "RecommendationSubstitutionsV1",
+  "description": "Versioned Product Substitution Recommendation Contract v1.0.0. Shared between Backend, Scan, and Mobile Consumers.",
+  "definitions": {
+    "SubstitutionRequest": {
+      "type": "object",
+      "required": ["barcode"],
+      "properties": {
+        "barcode": {
+          "type": "string",
+          "pattern": "^[0-9]{8,14}$",
+          "description": "Scanned product GTIN / barcode (8 to 14 digits)"
+        },
+        "overrides": {
+          "type": "object",
+          "properties": {
+            "avoidAllergens": {
+              "type": "array",
+              "items": { "type": "string" },
+              "default": [],
+              "description": "List of allergen codes/names to strictly avoid"
+            },
+            "dietaryRestrictions": {
+              "type": "array",
+              "items": { "type": "string" },
+              "default": [],
+              "description": "List of dietary restrictions to strictly enforce (e.g. 'vegan', 'vegetarian')"
+            }
+          },
+          "additionalProperties": false,
+          "description": "Optional request-level profile preference overrides"
+        },
+        "limit": {
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 20,
+          "default": 5,
+          "description": "Maximum number of substitute recommendations to return"
+        }
+      },
+      "additionalProperties": false
+    },
+    "SubstitutionItem": {
+      "type": "object",
+      "required": ["barcode", "productName", "safetyRating", "confidenceScore", "reasonCodes", "reasons"],
+      "properties": {
+        "barcode": { "type": "string", "description": "Candidate product barcode" },
+        "productName": { "type": "string", "description": "Candidate product display name" },
+        "brand": { "type": ["string", "null"], "description": "Candidate brand name" },
+        "nutriscoreGrade": {
+          "type": ["string", "null"],
+          "enum": ["a", "b", "c", "d", "e", "unknown", null],
+          "description": "Nutri-Score grade"
+        },
+        "safetyRating": {
+          "type": "string",
+          "enum": ["green", "grey"],
+          "description": "Evaluated safety rating. Unsafe ('red') candidates are hard-excluded."
+        },
+        "confidenceScore": {
+          "type": "number",
+          "minimum": 0.0,
+          "maximum": 1.0,
+          "description": "Recommendation confidence score between 0.0 and 1.0"
+        },
+        "reasonCodes": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Machine-readable reason codes from standardized catalogue"
+        },
+        "reasons": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Human-readable justification statements"
+        }
+      },
+      "additionalProperties": false
+    },
+    "SubstitutionResponse": {
+      "type": "object",
+      "required": ["version", "status", "targetProduct", "substitutions", "emptyStateReason"],
+      "properties": {
+        "version": {
+          "type": "string",
+          "const": "1.0.0",
+          "description": "Contract schema version"
+        },
+        "status": {
+          "type": "string",
+          "enum": ["success", "no_eligible_candidates", "insufficient_data"],
+          "description": "High-level execution status"
+        },
+        "targetProduct": {
+          "type": "object",
+          "required": ["barcode", "productName"],
+          "properties": {
+            "barcode": { "type": "string" },
+            "productName": { "type": "string" },
+            "category": { "type": ["string", "null"] },
+            "allergens": { "type": "array", "items": { "type": "string" } },
+            "nutriscoreGrade": { "type": ["string", "null"] }
+          },
+          "additionalProperties": true
+        },
+        "substitutions": {
+          "type": "array",
+          "items": { "$ref": "#/definitions/SubstitutionItem" }
+        },
+        "emptyStateReason": {
+          "type": ["string", "null"],
+          "enum": [
+            null,
+            "INSUFFICIENT_PRODUCT_DATA",
+            "NO_SAFE_ALTERNATIVES_IN_CATEGORY",
+            "STRICT_ALLERGEN_EXCLUSION_ALL_CANDIDATES",
+            "PRODUCT_NOT_FOUND"
+          ],
+          "description": "Explicit empty state taxonomy when substitutions array is empty"
+        }
+      },
+      "additionalProperties": false
+    },
+    "ErrorEnvelope": {
+      "type": "object",
+      "required": ["error"],
+      "properties": {
+        "error": {
+          "type": "object",
+          "required": ["code", "message"],
+          "properties": {
+            "code": {
+              "type": "string",
+              "enum": ["INVALID_BARCODE", "INVALID_LIMIT", "MALFORMED_REQUEST_BODY", "INTERNAL_SERVER_ERROR"]
+            },
+            "message": { "type": "string" },
+            "details": { "type": ["object", "array", "null"] }
+          },
+          "additionalProperties": false
+        }
+      },
+      "additionalProperties": false
+    }
+  }
+}

database/clean_data/cleanProductData.py

@@ -833,6 +833,11 @@ def _apply_conflicts(row) -> pd.Series:
     if "tagsRemoved" in df.columns:
         df = df.drop(columns=["tagsRemoved"])
 
+    # Populate normalized search fields
+    from database.clean_data.normalization.SearchNormalisation import normalize_search_text
+    df["productNameSearch"] = df.get("productName", pd.Series("", index=df.index)).apply(normalize_search_text)
+    df["brandSearch"] = df.get("brand", pd.Series("", index=df.index)).apply(normalize_search_text)
+
     for _, record in df.iterrows():
         record_warnings = validate_record(record.to_dict())
         if record_warnings:

database/clean_data/normalization/SearchNormalisation.py

@@ -0,0 +1,67 @@
+"""
+Search Text Normalization Module for Food Remedy Catalogue.
+
+Defines deterministic normalization rules for searchable text fields
+(productNameSearch, brandSearch) to support case-insensitive prefix queries in Firestore.
+"""
+
+import re
+import unicodedata
+from typing import Any, Dict, Optional
+
+
+def normalize_search_text(text: Any) -> str:
+    """
+    Normalize text for deterministic search matching.
+
+    Rules applied:
+    1. Handle missing, None, or non-string values safely (return empty string "").
+    2. Convert to string and normalize Unicode to NFC form.
+    3. Standardize smart/curly apostrophes and quotes to standard ASCII single quote ('').
+    4. Convert text to lowercase.
+    5. Strip leading and trailing whitespace.
+    6. Collapse repeated internal whitespace sequences into a single space.
+
+    Original text retains punctuation and hyphens in normalized form.
+    """
+    if text is None:
+        return ""
+
+    if not isinstance(text, str):
+        # Handle non-string types gracefully
+        text = str(text)
+
+    # Unicode NFC normalization
+    normalized = unicodedata.normalize("NFC", text)
+
+    # Standardize curly quotes and apostrophes to standard single quote
+    normalized = re.sub(r"[’‘`]", "'", normalized)
+
+    # Lowercase
+    normalized = normalized.lower()
+
+    # Collapse repeated internal whitespace and strip leading/trailing whitespace
+    normalized = re.sub(r"\s+", " ", normalized).strip()
+
+    return normalized
+
+
+def add_search_fields_to_product(product: Dict[str, Any]) -> Dict[str, Any]:
+    """
+    Enrich a product dictionary with productNameSearch and brandSearch fields.
+
+    Preserves original productName and brand/brands values untouched.
+    Modifies dictionary in-place and returns it.
+    """
+    if not isinstance(product, dict):
+        return product
+
+    # Retrieve source values
+    name_val = product.get("productName") or product.get("product_name") or ""
+    brand_val = product.get("brand") or product.get("brands") or ""
+
+    # Generate normalized search fields
+    product["productNameSearch"] = normalize_search_text(name_val)
+    product["brandSearch"] = normalize_search_text(brand_val)
+
+    return product