[New Skill]: monitoring/business_diagnostic — deterministic scenario ledger and calibration (companion to kpi_gate)

[mrmasa88]

Skill ID

monitoring/business_diagnostic

Category

monitoring

What should this skill do?

What it does. A deterministic scenario ledger and calibration skill that sits next to monitoring/kpi_gate. Given an operator-maintained, versioned framework — an exhaustive set of scenarios with an adjudication date, leading indicators with per-scenario weights, and the operator's probability marks — plus dated indicator observations, the skill returns the model-implied probability of each scenario under the declared weights, the delta from the operator's marks, which indicators fired, the next re-mark due date, days to adjudication, and — once the outcome is resolved — Brier scores per mark date. Where kpi_gate answers "did this period breach a rule?", this skill answers "across periods, which way has the balance of plausibility shifted, and how well calibrated were the operator's marks?"

Proposed short_description (brief mode, ~80 chars): Scenario ledger: declared-weight odds vs operator marks, re-mark dates, Brier.

The problem. Tracking several named scenarios with assigned probabilities is common practice — investment decisions, business planning, project risk — but the ledger usually lives in a spreadsheet or a table inside a document. Three things are then missing: (i) it is not traceable which observation moved which probability by how much; (ii) re-mark deadlines are not enforced; (iii) calibration (how right the marks turned out to be) is never computed after the fact. This skill mechanizes those three things without replacing any operator judgment.

Design split (same philosophy as kpi_gate).

• The framework is operator-owned, versioned, and strict-schema. Scenario ids and indicator ids are a closed set declared by the framework; the skill ships no scenarios or indicators of its own.
• The skill's closed error registry covers contract violations only: schema failures, undeclared weights, unknown indicators, marks that do not sum to 1, malformed dates.
• Operator marks are never overwritten. The output shows marks and model-implied values side by side, with deltas. "Model-implied" means "what the declared weights imply" — not a forecast and not a recommendation; instructions.md states this and asks hosts to present it that way.
• Unobserved indicators contribute 0; nothing is estimated. Missing required inputs return insufficient_data with reason codes (the honest third state).
• Renormalization happens only when exhaustive: true; otherwise each scenario updates independently.

Arithmetic (v0.1). For each scenario s: logit(p_s') = logit(p_s) + Σ_i w_{i,s} · o_i, where o_i ∈ {+1, 0, −1} (observed / unobserved / observed-negative) and the weights w are declared in the framework (default ±1.0). Given an outcome, Brier = Σ(p_s − y_s)² per mark date. Pure Python, standard library only, no network calls; identical input returns identical output.

Chaining and context (SkillContext / chains:).

• The skill never calls another skill; the host owns ordering (per docs/usage/skill_chaining.md).
• Stateless. SkillContext reuses skill instances across calls, so all ledger state — marks history, observations, outcome — travels in the input and comes back in the output. Nothing is kept on the instance.
• Position in chains. v0.1 is a terminal step: the host (or an upstream step) records indicator observations, adjudicate computes, the host presents. From v0.2 it is the natural step after kpi_gate — same policy / metrics vocabulary, findings and metrics mapped straight in:

chains:
  period_review:
    description: Gate this period's metrics, then update projections and the scenario ledger.
    when: A period snapshot has been recorded.
    steps:
      - id: gate
        skill: monitoring/kpi_gate
        input_from:
          metrics: host.metrics
          policy: host.policy
        map_out:
          findings: next.findings
      - skill: monitoring/work_diagnostic
        params:
          action: diagnose
        input_from:
          snapshots: host.snapshots
          policy: host.policy

• when: gates. Top-level booleans for chain conditions: fired_any (at least one indicator moved a scenario), marks_stale (re-mark overdue), insufficient (required input missing). calibration.* fields remain reachable by dot path.
• Relation to office/outreach_manager (proposed separately by the same issuer, filed alongside this one): the two skills are never adjacent steps in a chain. From v0.2 they connect only through data — outreach_manager.advance emits period aggregates in kpi_gate's metrics schema, which period_review above consumes.
• Brief mode. instructions.md is kept short (it is injected on prepare() / execute()): how to present deltas next to marks, when to prompt for re-marking, never to backfill insufficient_data.

Agent-loop contract. Hosts (a) surface deltas and fired indicators to the operator; (b) prompt for re-marking when marks_stale is true; (c) never backfill insufficient_data; (d) never present model-implied values on their own — always alongside the operator's marks.

Bundle. skill.py (Effect) · instructions.md (Directive, brief-mode sized) · schemas/ (JSON Schemas for framework, observations, and outcome — documentation; runtime uses explicit stdlib checks) · kb/demo_scenarios.json (synthetic example framework, timestamped and sourced) · test_skill.py (synthetic fixtures only) · card.json. The manifest declares named outputs: (scenarios, calibration, fired_any, marks_stale, insufficient, insufficient_data, contract_errors).

Roadmap (to be proposed as Skill Upgrades on the same ID). v0.2 — readiness / diagnose: a coverage scan of an operator-declared policy (N areas with targets, deadlines, and thresholds) and trend/projection from period snapshots using kpi_gate's metrics schema (default 90-day horizon; reach-by-deadline where a target has one) — this is where the period_review chain above becomes live. v0.3 — skeleton: an alert-first report skeleton assembled from those outputs (prose stays with the host). v0.1 is deliberately limited to the ledger and calibration to keep the first PR small.

Why monitoring. Business-side monitoring alongside kpi_gate (single-period gate) and token_limiter (resource gate); this skill adds the multi-period, calibrated layer.

Ideal Inputs & Outputs

Input (action: "adjudicate"):

{
  "action": "adjudicate",
  "framework": {
    "schema_version": 1,
    "framework_id": "demo_launch_2026",
    "adjudication_date": "2027-12-31",
    "exhaustive": true,
    "scenarios": [
      {"id": "A", "label": "wired launch",   "criterion": "..."},
      {"id": "B", "label": "branded launch", "criterion": "..."},
      {"id": "C", "label": "no launch",      "criterion": "..."},
      {"id": "D", "label": "residual",       "criterion": "..."}
    ],
    "indicators": [
      {"id": "I01", "label": "design document published",            "strengthens": {"A": 1.0, "B": -0.5}},
      {"id": "I07", "label": "timing synchronized with market window", "strengthens": {"B": 1.0, "A": -0.5}}
    ]
  },
  "marks": {"marked_on": "2026-07-18", "remark_every_days": 90,
            "values": {"A": 0.15, "B": 0.45, "C": 0.30, "D": 0.10}},
  "observations": [
    {"indicator": "I01", "observed": "yes", "on": "2026-08-20", "source": "ref-12"}
  ],
  "outcome": null
}

Output:

{
  "scenarios": [
    {"id": "A", "operator_mark": 0.15, "model_implied": 0.307, "delta":  0.157, "fired": ["I01"]},
    {"id": "B", "operator_mark": 0.45, "model_implied": 0.314, "delta": -0.136, "fired": ["I01"]},
    {"id": "C", "operator_mark": 0.30, "model_implied": 0.284, "delta": -0.016, "fired": []},
    {"id": "D", "operator_mark": 0.10, "model_implied": 0.095, "delta": -0.005, "fired": []}
  ],
  "calibration": {"next_remark_due": "2026-10-16", "days_to_adjudication": 485, "brier": null},
  "fired_any": true,
  "marks_stale": false,
  "insufficient": false,
  "insufficient_data": [],
  "contract_errors": []
}

action: "calibrate" takes outcome: {"resolved_on": "...", "scenario": "B"} and returns the Brier score for each mark date and its trend across re-marks.

Target runtime

Model agnostic (all supported adapters)

External APIs & env vars (if any)

None. No network calls; no environment variables.

[rosspeili]

Thanks @mrmasa88, this is a thoughtful, even tho confusing with so many terms and vocabulary, it is well-specified proposal and it fits the monitoring category as a multi-period companion to your kpi_gate skill. I would keep terms as generic as possible, but again, this will be under AO org, so up to you if you wanna make it more lean towards where you intended to use this skill <3

What lands well

• Same design split as kpi_gate: operator-owned versioned framework, closed contract errors, honest insufficient_data, stdlib-only deterministic core, etc.
• Stateless ledger (all state in I/O) is the right call for the new SkillContext and future chains.
• v0.1 scope (adjudicate + calibrate only) is appropriately bounded I guess, the arithmetic and example I/O are clear enough to implement and test offline.

Steering before implementation

1. Naming, work_diagnostic may read as generic workplace/IT diagnostics to newcomers. If you’re open to it, consider a more literal id (scenario_ledger / probability_ledger or something more concrete), otherwise lean hard on short_description and the catalog intro so the purpose is obvious in one line.
2. Scope discipline, Please keep the first PR to v0.1 only. v0.2 (readiness/diagnose, kpi_gate metrics wiring) and v0.3 (skeleton) are better as follow-up Skill Upgrade issues once the ledger core is merged.

Reference implementation, your previous monitoring/kpi_gate (#317 → #318) is the closest template for bundle layout I think, schemas-as-docs, fail-closed validation, and catalog/agent-loop ripples.

Happy to green-light v0.1 on this spec, but I'll defo need to see the first PR to see where this should be heading. <3

[mrmasa88]

Hey! @rosspeili

Thanks Ross — green light noted, and point taken on vocabulary: the PR's catalog page and instructions.md will define the two or three terms that matter once and drop the rest.

Naming: agreed that work_ reads wider than intended. I'll file the PR as monitoring/business_diagnostic and update this issue's title to match. "business" pins the domain; short_description will state literally what v0.1 does — "Scenario ledger for a business: declared-weight odds vs operator marks; Brier." — and the catalog intro will say in one line that v0.1 is the ledger, with projections against declared targets (v0.2) and an alert-first report skeleton (v0.3) to follow as Skill Upgrades. Nobody should expect more from the first PR than it delivers.

Scope: the first PR is v0.1 only — adjudicate + calibrate, stateless, stdlib-only, schemas as docs, fail-closed contract errors — built on the kpi_gate bundle as the template. Anything beyond the ledger comes as separate Skill Upgrade issues after this merges.

PR to follow; I'll link it here.

Warm regards,