Skip to content

feat: EVALS for data-product-create skill. - #108

Draft
dmitrii-mikh-collibra wants to merge 1 commit into
mainfrom
evals-for-data-product-create-skill
Draft

feat: EVALS for data-product-create skill.#108
dmitrii-mikh-collibra wants to merge 1 commit into
mainfrom
evals-for-data-product-create-skill

Conversation

@dmitrii-mikh-collibra

Copy link
Copy Markdown
Contributor

🎯 What does this PR do?

Add Inspect-based eval harness for the data-product-create skill

What

Introduces an evals/ harness that measures whether the collibra/data-product-create skill actually produces the correct asset graph in a real Collibra environment. First eval is the data-product-create happy path; the layout is set up so each future skill gets its own task file.

How it works

The harness uses Inspect AI. A single task drives chip (built as an MCP stdio server) through the full skill flow, then scores the outcome from three independent angles:

  • end_state.py — ignores how the agent got there and asks the live environment whether the expected graph exists. Four equally-weighted checks (product exists, ports linked, tables grouped, contract governs a port) so partial results score proportionally.
  • trajectory.py — walks the transcript's tool calls to enforce procedural rules:
    • skill_loaded — the skill was discovered/loaded before any write (isolates triggering failures from execution failures).
    • write_order — every create_asset precedes any relation-adding edit, and the contract push comes after its port exists.
  • collibra.py — thin async DGC REST 2.0 client shared by the scorers and cleanup.

Other pieces:

  • tasks/data_product_create.py — the Inspect task. Injects a system message replicating chip's server-level "discover-and-load-skills" instruction (Inspect doesn't surface it), and supports EVAL_ENABLED_TOOLS to trim the tool surface on low API rate-limit tiers.
  • fixtures/data_product_create.yaml — the only per-environment file; holds the source table and expected outcome. The task fails fast with a clear message if unfilled.
  • scripts/cleanup.py — deletes eval-created assets between runs (dry-run by default, --apply to execute), since duplicate names break reruns.

✅ Checklist

  • My code follows the style guidelines of this project.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation (if needed).
  • My commit messages follow the Conventional Commits standard.

@svc-snyk-github-jira

svc-snyk-github-jira commented Jul 24, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues
Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants