feat(monitoring): add business_diagnostic v0.1 — scenario ledger and calibration (#338)

[mrmasa88]

Closes #338

Scope

v0.1 only — adjudicate and calibrate. Stateless, stdlib-only, offline; schemas shipped as documentation with explicit runtime validation; fail-closed contract errors. Projections against declared targets and a report skeleton are out of scope here and will come as separate Skill Upgrade issues after this merges.

Bundle (skills/monitoring/business_diagnostic/)

• manifest.yaml, skill.py, instructions.md, card.json, __init__.py
• schemas/ — framework, marks, observations, outcome, output (documentation; runtime uses explicit checks)
• kb/demo_scenarios.json — synthetic framework, timestamped and sourced
• fixtures/ — E2E adjudicate/calibrate, honest state, one fixture per error code
• test_skill.py — 36 offline tests: loader, manifest identity, E2E equality against fixtures, every error code from its fixture (registry set == fixture set), non-exhaustive update, negative observations, re-mark boundary, honest state, Brier trend ordering, identity without indicators, bit-identical repeat, no clock in source

Modeled on monitoring/kpi_gate (#318).

Repo ripples

docs/skills/business_diagnostic.md (catalog page) · docs/skills/README.md row · docs/usage/agent_loops.md · docs/usage/install_extras.md + pyproject.toml extras (via scripts/sync_extras.py) · tests/fixtures/card_ui_schema/monitoring__business_diagnostic.json · examples/business_diagnostic_demo.py (+ README, smoke test) · host-simulation stress discovery assert · CHANGELOG.md

Deviations from the issue text (small, deliberate)

1. as_of is a required input — the skill never reads the clock, so identical input returns identical output.
2. Contract violations use the same shape as kpi_gate (status: error + error{code, detail}); the contract_errors: [] array in the issue's example is not used.
3. A duplicate observation for the same indicator is a contract error (hosts de-duplicate); observations dated on/before marked_on or after adjudication_date are rejected — the ledger cannot double-count.
4. The error registry has 14 codes: an INVALID_REQUEST envelope code was added so that UNKNOWN_ACTION means exactly that. Validation order is fixed and documented.
5. Output carries three top-level booleans for when: gates in chains (fired_any, marks_stale, insufficient).

Tests

python3 -m pytest skills/monitoring/business_diagnostic tests/test_skill_issuer.py tests/test_registry_identity.py tests/test_card_ui_schema.py tests/test_registry_docs.py tests/test_extras_sync.py tests/test_examples_smoke.py -q → 91 passed
Full suite: 721 passed / 5 skipped. black --check and flake8 clean; scripts/sync_extras.py --check OK. Wheel smoke: run locally — 21 registry skills verified in the built wheel, monitoring/business_diagnostic loadable from a fresh base install, 4 unrelated skills deferred (optional extras not installed).

---

Type of Change

• New Skill — new registry bundle under skills/
• Skill Upgrade — changes to an existing skill under skills/
• Bug Fix — incorrect runtime or framework behavior
• Documentation — docs, README, CONTRIBUTING only
• Framework Feature — skillware/core/ loader, env, adapters
• CLI — skillware/cli.py, docs/usage/cli.md
• Examples — examples/*.py, agent loops, examples/README.md
• Packaging — PyPI wheel, pyproject.toml, MANIFEST.in
• RFC / meta — templates, labels, CI, or large design doc

Checklist (all PRs)

• Linked GitHub issue (Fixes #… or Refs #…)
• Scope matches the issue — no unrelated refactors
• python -m black --check . and flake8 pass locally (or CI-equivalent subset)
• pytest skills/ and pytest tests/ pass locally when relevant
• CHANGELOG.md updated under [Unreleased] when user-visible behavior changes
• examples/README.md updated if this PR adds, renames, or removes a runnable script
• Ran pytest tests/test_registry_docs.py when skills, examples index, or agent-loops matrix changed

New or updated skill

Bundle and metadata

• Skill at skills/<category>/<skill_name>/ (from templates/python_skill/ or equivalent)
• manifest.yaml: name (full ID), version, description, parameters, constitution, real issuer
• Optional: short_description, issuer.github, issuer.org, requirements, env_vars

Effect, Directive, Assurance

• Deterministic skill.py (Effect; no ad-hoc LLM-generated execution paths)
• instructions.md (Directive) explains when and how to use the skill
• card.json (Presentation) issuer matches manifest when present
• test_skill.py (Assurance) covers execution and schema expectations
• SkillLoader.load_skill("<category>/<skill_name>") succeeds (or deps documented)

Documentation and catalog

• docs/skills/<skill_name>.md and row in docs/skills/README.md
• Usage Examples for Gemini, Claude, OpenAI, DeepSeek, Ollama per skill usage template

Constitution and safety (skills only)

Compute only: no data fetching, no network, no clock, no side effects. Operator marks are never overwritten; model-implied values are labeled as what the declared weights imply, not forecasts. Undeclared ids are contract errors; what cannot be computed honestly returns insufficient_data with a reason code.

Related Issues

Closes #338 · Refs #318 (kpi_gate, the template for this bundle)

[rosspeili]

@mrmasa88 this is excessive and quite deep work, probably your best first PR so far. <3

---

Some things to consider before we can merge:

Offline demo, full v0.1 lifecycle

Expand examples/business_diagnostic_demo.py to match kpi_gate_demo.py depth (multi-scenario narrative, not adjudicate + calibrate + one error only):

1. Adjudicate, in-bundle e2e fixture, print deltas + fired + marks_stale + days_to_adjudication ([New Skill]: monitoring/business_diagnostic — deterministic scenario ledger and calibration (companion to kpi_gate) #338 numbers).
2. Calibrate, resolved outcome, print brier: 0.425.
3. Calibrate + brier_trend, load example_marks_history.json, print sorted trend.
4. Honest state, adjudication_date_passed fixture, print insufficient_data, null model_implied/delta.
5. Second fail-closed error, eg MARKS_NOT_NORMALIZED or DUPLICATE_OBSERVATION (kpi_gate demo runs two contract errors).

Minor ripples to be addressed in the same PR after the above is in:

• Update examples/README.md row to describe the expanded scenarios.
• Update tests/test_examples_smoke.py anchor strings for business_diagnostic_demo.py (keep brier:, add honest-state marker, eg. insufficient_data: or adjudication_date_passed).
• Run python examples/business_diagnostic_demo.py and pytest tests/test_examples_smoke.py -k business_diagnostic.

---

Catalog, two-action loop + all providers

docs/skills/business_diagnostic.md has Gemini / Claude / OpenAI / DeepSeek / Ollama blocks, but they only exercise adjudicate. This skill has two actions, the catalog must show the full loop.

• Add shared constants alongside the compact setup:
  • CALIBRATE_USER_MESSAGE (or equivalent) with action: "calibrate", outcome, optional marks_history, same framework/marks/as_of.
• Add ### Full loop (adjudicate → calibrate) before provider subsections: host supplies as_of every call, stateless (marks/observations/outcome travel in JSON), surface delta + fired next to operator_mark, never model_implied alone, marks_stale → prompt re-mark, insufficient_data → report reason, no backfill, chain booleans fired_any, marks_stale, insufficient.
• Every provider block (Gemini, Claude, OpenAI, DeepSeek, Ollama): show both tool calls in sequence, adjudicate after observation, then calibrate after resolution, with skill.execute(...) on each. Calibrate path must print calibration["brier"] (and brier_trend when history is present), not only scenarios.
• Keep snippets runnable per skill_usage_template.md: load_env_file(), bundle["class"](), adapter, bundle["instructions"], sample user message(s), execute on tool call.

---

Agent loops doc

• Add ### Business Diagnostic, adjudicate → calibrate loop to docs/usage/agent_loops.md (same pattern as UK CH pipeline subsection): agent-loop contract, both actions with copy-paste execute payloads, link to business_diagnostic_demo.py and catalog Usage Examples. Table row already points catalog → keep in sync.

---

Directive + catalog semantics to tighten honesty

Mirror in instructions.md and catalog Limitations / Arithmetic:

• observations on calibrate: validated if present, not used for Brier scoring (operator marks only).
• exhaustive: false: per-scenario independent updates, implied values may not sum to 1, one compact numeric example (tests already cover this, docs/snippets do not).
• Multi-class Brier: Σ_s (p_s − y_s)², one-hot winner, one line so forecasters don’t argue post-merge.
• Weight semantics: yes → +w, no → −w, unlisted scenario → 0, unobserved indicator → 0 (no imputation).

---

Tests already solid, but re-run after the above. Also remove PR from work in progress / draft whe you done, so we can merge. <3

[mrmasa88]

Thanks a lot! @rosspeili

Round 1 addressed in one commit (0ac8eb0) — thanks for the thorough pass.

• Demo: five scenes (adjudicate → calibrate → calibrate with brier_trend → honest state → two fail-closed errors: MARKS_NOT_NORMALIZED, DUPLICATE_OBSERVATION), fixtures only, offline. README row and smoke anchors (brier:, brier_trend:, insufficient_data:, adjudication_date_passed) updated; python examples/business_diagnostic_demo.py and pytest tests/test_examples_smoke.py -k business_diagnostic green.
• Catalog: ADJUDICATE_USER_MESSAGE / CALIBRATE_USER_MESSAGE, a Full loop (adjudicate → calibrate) section ahead of the providers, and all five provider blocks now run both tool calls in sequence and print calibration["brier"] / brier_trend. The shared constants now use the same four-scenario framework as the E2E fixture, so catalog, demo, agent-loops doc and fixtures all show identical numbers.
• Agent loops: Business Diagnostic, adjudicate → calibrate loop section with copy-paste payloads and links.
• Semantics, same wording in instructions.md and the catalog: observations on calibrate validated but unused; exhaustive: false independent updates with a numeric example (A 0.324 / B 0.332 / C 0.300 / D 0.100, sum 1.056 — same inputs as the existing test); multi-class Brier one-liner; weight semantics with no imputation.

No changes to skill.py, schemas, fixtures, or test_skill.py. Suite: 721 passed / 5 skipped; CI green on 3.10–3.12 plus wheel smoke. Draft removed.

Warm regards,

Masa

[rosspeili]

Thanks @mrmasa88, LGTM.

Before merge: merge/rebase main, your branch predates #345, so [Unreleased] → Changed is missing the Gemini 3.5 migration lines (they'll land on merge anyway I think cause CI is green, but please rebase so the PR diff is accurate). Also update docs/skills/business_diagnostic.md Gemini snippet from gemini-2.5-flash-lite → gemini-3.5-flash-lite to match repo standard, or anywhere else you used gemini-2.5 or any version before 3.5, 3.5-flash-lite, and 3.6-flash which are now used across the entire repo.

After these tiny fixes we can merge as is, and I will prepare a v2 issue you can pick and upgrade on your own pace. <3

[mrmasa88]

@rosspeili Thanks a lot!

Synced with main (merge commit c20ab1e, so [Unreleased] → Changed now carries the #345 Gemini 3.5 migration lines alongside this PR's Added line; the only conflict was two table rows in docs/usage/install_extras.md, resolved by keeping both sides — test_extras_sync and sync_extras.py --check pass) and aligned the Gemini snippet to gemini-3.5-flash-lite (3b87ab5) — no other pre-3.5 ids in the diff. Suite 792 passed / 5 skipped, CI green on 3.10–3.12 plus wheel smoke; nothing else changed. Ready to merge on your side. On v2: the next scope is already drafted from the #338 roadmap (readiness / diagnose against declared targets, chained after kpi_gate). I'll file it as a Skill Upgrade issue once this merges — anything you had in mind for your v2 note, just add it there and we'll fold it in.

Warm regards,

Masa

[rosspeili]

Thanks, round 1 + Gemini sync look great, and the skill bundle is solid.

#346 merged since your last sync, so the PR is conflicted again. Please merge/rebase main and:

1. agent_loops.md, keep your Business Diagnostic subsection, do not re-add the UK CH pipeline block (Refactor and update uk_companies_house_handler for v1.2.1 (#341)  #346 moved that to the catalog). Use main’s UK CH matrix row (includes claude_uk_companies_house_handler.py).
2. examples/README.md, keep business_diagnostic_demo.py and main’s updated UK CH / Claude example rows.
3. business_diagnostic.md, Claude snippet: claude-3-5-haiku-latest → claude-haiku-4-5-20251001 (repo standard after Refactor and update uk_companies_house_handler for v1.2.1 (#341)  #346).

After that we can merge. v2 issue whenever you’re ready, and we can review together. <3

[mrmasa88]

@rosspeili Hey!

Synced with main — merge commit e704734, then 1d5c4fe (head); #346 and #347 are in.

1. docs/usage/agent_loops.md — Business Diagnostic subsection kept; the UK CH block that Refactor and update uk_companies_house_handler for v1.2.1 (#341)  #346 moved to the catalog is not re-added, and main's UK CH matrix row is taken as-is.
2. examples/README.md — the business_diagnostic_demo.py row is kept alongside main's updated UK CH / Claude rows.
3. docs/skills/business_diagnostic.md — Claude snippet now uses claude-haiku-4-5-20251001 (1d5c4fe).

tests/test_skill_docs.py passes with the five-provider guard; full suite 795 passed; CI green (3.10 / 3.11 / 3.12, wheel-smoke, CodeQL). No changes to skill.py, schemas, fixtures or tests, and git diff upstream/main...HEAD touches no main-side lines.

One thing I noticed while resolving and left untouched here: after #347 the agent_loops.md matrix lists office/gmail_handler twice (near the top and at the end). Happy to send a one-line follow-up PR if useful.

I'll open the v0.2 Skill Upgrade issue as soon as this merges — happy to review it together.

Warm regards,

Masa