Extend key entities metric to audio-native models

configs/prompts/judge.yaml

@@ -1071,3 +1071,137 @@ judge:
           "summary": "<1-2 sentence summary for this turn>"
       }}
       ]
+
+    audio_native_prompt: |
+      You are an expert evaluator analyzing whether an AI voice agent correctly **understood** the key entities a user provided during a spoken conversation.
+
+      This is a speech-to-speech / audio-native agent: there is no reliable speech-to-text transcript of the user to inspect. Instead, the evidence of what the agent actually understood is the **arguments it passed to its tool calls**. When a user says "my id is EM123" and the agent then calls a tool with "EMP124", the agent misheard the entity.
+
+      You are given a Conversation containing user turns, tool calls, and tool responses. **The assistant's own spoken turns are redacted** (shown as `[Assistant speaks]`) and deliberately withheld: the agent's speech is itself imperfectly transcribed, so it must never be used as evidence. Judge only what the user said (user turns) against what the agent did (tool-call arguments).
+
+      Your task — evaluate each of the agent's TOOL CALLS individually:
+      1. Each tool call in the Conversation is labelled with a stable id: `Tool Call [N]`. Produce **one output entry per tool call** that uses at least one user-provided entity, keyed by that id (`tool_call_id: N`). Be exhaustive — a single turn often contains MORE THAN ONE tool call (e.g. a lookup followed by a search, or a rebook followed by a seat assignment); rate each labelled tool call separately, never merge two of them or skip one.
+      2. Within each tool call, look at **every argument**. For any argument whose value traces back to something the USER said — a user-provided key entity (name, confirmation code, date, amount, seat preference, etc.) — work out what the user actually said for it.
+      3. Rate whether the value in that tool call matches what the user said: correct or incorrect.
+      4. A user often states an entity in one turn and the agent uses it in a tool call several turns later — always rate it at the tool call, never at the user turn. If the same user value is used in more than one tool call, rate it in EACH one (so it can be correct in one tool call and wrong in another).
+      5. Only rate arguments that trace back to a value the USER provided. Ignore any argument sourced from a tool response or generated by the agent (e.g. a journey/record id returned by a prior search, or a passenger id). If a tool call has no user-provided arguments at all, omit it from your output entirely.
+
+      ## What Counts as an Entity
+      An entity must have a **specific, concrete value** — something that could be passed as an input to a program or tool (not an AI, but a script or database lookup). Ask yourself: could this value be stored in a variable and used programmatically?
+
+      - Names (people, places, organizations): e.g. "John Smith", "Austin", "Delta Airlines"
+      - Specific dates and times: e.g. "December 15th", "3:45 PM" — NOT vague references like "tomorrow morning" or "later today"
+      - Confirmation codes / reference numbers: e.g. "ABC123", "ZK3FFW"
+      - Flight numbers: e.g. "UA 204"
+      - Amounts and prices: use the specific value only, e.g. "$120" — for qualifier phrases like "under $120", only use the specific value
+      - Addresses: e.g. "123 Main Street"
+      - Phone numbers: e.g. "555-867-5309"
+      - Email addresses: e.g. "john@example.com"
+      - Other specific identifiers: seat numbers, loyalty numbers, booking IDs, employee IDs, etc.
+
+      **Not an entity:** vague temporal words ("tomorrow", "next week", "morning"), general descriptors ("the cheap flight", "a long trip"), open-ended qualifiers ("less than an hour", "around noon"), or standalone honorifics/titles ("Mr.", "Ms.").
+
+      ## IMPORTANT: What This Metric Measures
+      This measures whether the agent **understood** the entities the user provided, as evidenced by the values it passed to its tool calls. It is NOT a faithfulness or tool-correctness metric. Do NOT penalize:
+      - The agent choosing the wrong tool, or omitting a tool call it should have made
+      - Values in a tool call that came from a tool RESPONSE or that the agent generated itself (only user-provided values are evaluated)
+      - A user entity that was never used in any tool call — it simply is not rated (do not report it)
+
+      Only mark a value incorrect when a tool call used a **different value** than the user provided (e.g. user "EM123" → tool call "EMP124", user "March 25th" → tool call "2026-03-26").
+
+      ## Non-Spoken Tags in the User Text
+      The user text may contain non-spoken tags — metadata that was never said aloud and must never be treated as entities:
+
+      - **Audio-direction tags** like [slow], [firm], [annoyed] describe how the words were meant to be spoken. Ignore them entirely.
+      - **Interruption tags** mark text the user likely never finished saying. No special handling is needed: anything that was never spoken cannot appear in a tool call, so there is simply nothing to rate.
+
+      {interruption_tags_reference}
+
+      ## Conversation
+      {conversation}
+
+      ## Correctness Criteria
+      - Entity must have been used correctly by the agent — the value passed to the tool call must match what the user said.
+      - Numbers: "150" and "one hundred fifty" are equivalent
+      - Dates: "December 15th" and "Dec 15" are equivalent
+      - Names: Case-insensitive match. Phonetically equivalent spelling variants are acceptable on the **first mention** of a name (e.g., German Meyer/Meier/Maier/Mayer — all sound the same when spoken). Once the user has explicitly disambiguated the spelling (e.g., spelled it out letter by letter, or corrected a prior spelling), subsequent usage must match that clarified spelling exactly. Flag if:
+        - The value the agent used clearly differs (different sound, not just different spelling)
+        - The agent used a different spelling variant **after** the user has explicitly clarified the correct spelling
+      - Non-Latin scripts: A name or identifier rendered in its native writing system (e.g. Kanji, Hangul, Arabic) is equivalent to its romanized/phonetic form if they represent the same spoken entity. For example, "山田" and "Yamada" should be treated as matching.
+      - Phonetic letter spellings: In languages that do not use the Latin alphabet, Roman/Latin letters are spoken using language-specific phonetic equivalents. These are correct, not errors (e.g., in Korean "에이" for "A" or "비" for "B").
+      - Number surface variants: Many languages have systematic spoken-form differences that do not change the underlying value and must not be penalized:
+        - Reversed digit order in spoken form (e.g., German "vierundzwanzig" = four-and-twenty = 24)
+        - Grammatical gender/agreement variants (e.g., Spanish "veintiún" vs "veintiuna" for 21)
+        - Large-unit grouping differences (e.g., Japanese 万-system: "ichi-man go-sen" / 一万五千 = 15,000)
+
+      Important note: The user text will often feature things formatted like "one two three" instead of "123". Your goal is to evaluate semantic equivalence — these are considered equivalent if the value matches what the agent used.
+
+      ## Examples
+
+      The user gives an id in turn 1 and corrects it in turn 2; nothing is rated at those user turns — each **tool call** is rated on its own. Tool Call [1] used the pre-correction value EM123 (wrong, because the user had already corrected it) and Tool Call [2] used the corrected value EM456 (right). Tool Call [3] uses only a `record_id` that came from a tool response, so it is omitted entirely.
+
+      **Example Input (Conversation):**
+      Turn 1 - User: My employee id is E M one two three.
+      Turn 1 - [Assistant speaks]
+      Turn 2 - User: Actually, sorry — the id is E M four five six.
+      Turn 2 - [Assistant speaks]
+      Turn 4 - [Assistant speaks]
+      Turn 4 - Tool Call [1] (lookup_employee): {{'employee_id': 'EM123'}}
+      Turn 4 - Tool Response (lookup_employee): {{'error': 'not_found'}}
+      Turn 6 - [Assistant speaks]
+      Turn 6 - Tool Call [2] (lookup_employee): {{'employee_id': 'EM456'}}
+      Turn 6 - Tool Response (lookup_employee): {{'employee_id': 'EM456', 'record_id': 'R-9012'}}
+      Turn 6 - Tool Call [3] (send_summary): {{'record_id': 'R-9012'}}
+
+      **Example Response:**
+      [
+      {{
+          "tool_call_id": 1,
+          "tool_name": "lookup_employee",
+          "entities": [
+          {{
+              "type": "employee_id",
+              "value": "E M four five six",
+              "captured_value": "EM123",
+              "analysis": "The user first said EM123 but corrected it to EM456 in turn 2, so EM456 is the value they provided. This tool call used the pre-correction value EM123 — incorrect.",
+              "correct": false
+          }}
+          ],
+          "summary": "Tool call used the pre-correction id EM123; the user's provided id is EM456 — incorrect."
+      }},
+      {{
+          "tool_call_id": 2,
+          "tool_name": "lookup_employee",
+          "entities": [
+          {{
+              "type": "employee_id",
+              "value": "E M four five six",
+              "captured_value": "EM456",
+              "analysis": "Matches the id the user provided (after their turn-2 correction).",
+              "correct": true
+          }}
+          ],
+          "summary": "Tool call used EM456 — correct."
+      }}
+      ]
+
+      Note Tool Call [3] does not appear: its only argument (record_id) came from a tool response, not the user.
+
+      ## Response Format
+      Respond with a JSON array. Output **one entry per tool call that uses at least one user-provided entity**, keyed by the tool call's `[N]` id. Tool calls with no user-provided entities, and turns where the user only speaks, must NOT appear.
+      [
+      {{
+          "tool_call_id": <int: the [N] id of the tool call being rated>,
+          "tool_name": "<the tool's name, for readability>",
+          "entities": [
+          {{
+              "type": "<name|date|time|confirmation_code|flight_number|amount|address|phone|email|employee_id|etc...>",
+              "value": "<the value the user provided for this entity; use the corrected value if the user corrected it>",
+              "captured_value": "<the value this tool call passed>",
+              "analysis": "<brief reason for the correct/incorrect judgement; note any user correction here>",
+              "correct": <true|false>
+          }}
+          ],
+          "summary": "<1-2 sentence summary for this tool call>"
+      }}
+      ]

src/eva/__init__.py

@@ -11,4 +11,4 @@
 
 # Bump metrics_version when changes affect metric computation (metrics code,
 # judge prompts, pricing tables, postprocessor).
-metrics_version = "2.2.1"
+metrics_version = "2.2.2"