Skip to content

Define and verify the canonical provider-preflight artifact #24

Description

@Punisheroot

Context

Live commands emit related but different preflight reports. Their passed
flags and caller-supplied counters are not independently trustworthy. Before a
native collector or execution gate can be shared, Needle needs one bounded
artifact schema and a pure verifier that performs no process launch.

Outcome

Define canonical expectation, observed-evidence, artifact, and
verification-result contracts for provider preflight. Add an offline verifier
that recomputes identity and rejects drift, unsafe bounds, or any observed
provider model turn. Native transport collection is a separate dependent issue.

Scope

  • Define versioned bounded schemas with unknown-field rejection.
  • Bind source snapshot/repository SHA, platform, Codex binary/version identity,
    compatibility fixture, model/reasoning/tier, pricing, execution limits,
    output-root identity/containment, estimated budget, cleanup, and observed
    transport evidence.
  • Define which values contribute to artifact identity and freshness/expiry.
  • Derive live-call and provider-turn counters from supplied observed protocol
    events; never trust summary counters or a passed field.
  • Add a pure verifier that compares the artifact with an independently supplied
    current expectation and emits a canonical verification digest.
  • Reject credentials, raw auth state, provider responses, sealed evaluator
    material, raw secret paths, unbounded diagnostics, and unknown fields.
  • Add fixtures for pass, malformed/unknown fields, source/binary/fixture drift,
    pricing/limit mismatch, unsafe root, counter mismatch, non-zero provider
    events, expiry, and cleanup failure.

Non-goals

  • Running Codex, collecting native transport events, starting a provider turn,
    recording approval, migrating commands, or authorizing spend.
  • Executing a benchmark/evaluator or treating verification as readiness
    evidence by itself.
  • Adding a native environment probe to credential-free CI.

Acceptance criteria

  • The verifier is a deterministic offline function over canonical artifact,
    observed events, and current expectation.
  • It recomputes digests, event-derived counters, compatibility, freshness,
    bounds, path identity, and cost inputs instead of trusting summaries.
  • Any provider model event, drift, expiry, unsafe path, malformed field,
    incompatible fixture, or cleanup failure rejects the artifact.
  • The artifact cannot encode credentials, auth contents, provider response,
    sealed material, or raw secret paths.
  • Verification produces a digest suitable for later approval but never
    changes approval state or launches a process.
  • Synthetic fixtures cover every acceptance and rejection branch.

Starting points

Validation

Run focused schema, canonicalization, verifier, event-counter, compatibility,
freshness, path, budget, and adversarial fixture tests. All validation is
process-free and provider-free.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: benchmarksBenchmark corpus, runners, and statistical evidencearea: runtimeCore runtime and execution lifecycleenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions