Compare Code Context Providers

[sarthakagrawal927]

Spec-driven change: compare-code-context-providers

Proposal

Why

Code-context products make overlapping claims about helping coding agents, but
CodeVetter has no reproducible evidence that its graph—or any competing
provider—improves executable task outcomes. The qualified 30-task corpus,
provider-neutral runner, immutable receipts, and structural-context scorer now
make a bounded comparison possible without treating token savings, attractive
documentation, or model opinion as success.

What Changes

• Add a preregistered context-provider experiment that compares one no-special-
  context baseline with eligible agent-readable provider configurations while
  keeping task, agent, model, prompt, environment, revision, and trial identity
  fixed.
• Qualify provider eligibility through an explicit capability probe covering
  machine-readable access, exact version/configuration identity, snapshot
  freshness, bounded setup, tool-call observability, privacy, and repeatability.
• Reuse the existing agent-task runner and structural-context scorer; add only
  the provider identity, isolation, scheduling, and cross-provider aggregation
  needed for a deterministic multi-arm report.
• Run a free/local feasibility stage before any full-corpus or paid/hosted
  trial. Full trials require an exact cost/attempt plan and explicit approval.
• Treat executable hidden checks and preserved regressions as the primary
  outcome. Report setup success, invalid/contaminated trials, relevant-file
  recall where ground truth exists, files inspected/modified, tool calls,
  latency, tokens, and cost only as diagnostics.
• Keep human-wiki and enterprise products out of the first cohort unless they
  expose a reproducible agent-readable CLI, API, or MCP interface. Keep generic
  storage engines out of scope because they are building blocks rather than
  comparable context providers.
• Publish no winner or product-value claim from synthetic fixtures, incomplete
  arms, stale indexes, contaminated controls, unqualified A/A noise, or a
  feasibility-only run.

Capabilities

New Capabilities

• context-provider-comparison: Defines provider eligibility, immutable
  multi-arm experiment identity, isolation, deterministic scheduling, outcome
  authority, reporting, and staged claim gates for code-context comparisons.

Modified Capabilities

• None. Existing agent-task receipt evaluation and structural-context scoring
  remain authoritative and backward-compatible; the new capability composes
  them without weakening their contracts.

Impact

• Extends benchmarks/agent-tasks experiment metadata and adapter fixtures.
• Adds bounded orchestration/reporting beside
  scripts/agent-task-corpus/ and benchmarks/structural-context/.
• Reuses the 30-task corpus, v2 run receipts, evaluation bundles, hidden checks,
  and current structural-context score rather than introducing a second runner
  or grader.
• May invoke separately installed local or hosted provider interfaces only
  during explicitly approved trials; no provider becomes a production
  dependency and no credentials or provider data enter committed artifacts.
• Operationally advances GitHub issue Run repeated comparative trials across agent configurations #55. It does not publish results, run a
  paid provider, alter the desktop UI, or create an enterprise integration.

---

Design

Context

CodeVetter already has the pieces needed for a single-provider paired test:

• a 30-task, eight-category qualified corpus;
• deterministic planning and a one-attempt provider-neutral runner;
• immutable v2 run receipts with optional diagnostics;
• a receipt composer that rejects identity drift and contaminated controls;
• the structural-context A/B and A/A scorer with JSON, Markdown, and HTML
  projections.

The current evaluation bundle models one control/treatment pair and names
the treatment context as a graph. The planned experiment must compare several
agent-readable context systems without weakening that proven pairwise contract,
misclassifying infrastructure as a finished provider, or installing any
provider into the production application. See proposal.md for motivation and
specs/context-provider-comparison/spec.md for observable requirements.

Goals / Non-Goals

Goals:

• Compare a plain baseline, CodeVetter context, and eligible external context
  providers under identical executable tasks and agent configurations.
• Preserve the current hidden-check scorer as the outcome authority.
• Make provider eligibility, indexing freshness, tool access, order, cost, and
  contamination inspectable before interpreting results.
• Stage the work so a cheap local feasibility probe can stop an unsound or
  impractical experiment before full-corpus execution.

Non-Goals:

• Building another graph, wiki, search engine, or enterprise context service.
• Comparing human-facing documentation quality or visual graph quality.
• Treating pgGraph, HydraDB, or another storage primitive as an end-to-end
  context provider.
• Adding provider SDKs or CLIs as CodeVetter production dependencies.
• Running paid trials, publishing a leaderboard, or making provider claims in
  this planning change.

Decisions

1. Add a comparison layer over existing pairwise scores

The multi-provider experiment will introduce closed context-provider-plan,
context-provider-probe, and context-provider-comparison artifacts. Each
eligible provider is still evaluated as baseline-versus-provider evidence
through the existing receipt composer and structural-context scorer. The new
aggregator validates those pairwise score artifacts against one common plan
and produces the cross-provider scorecard.

This avoids changing the established A/B evaluator or creating a second task-
success definition. The legacy context.graph projection can carry the exact
provider engine/index identity internally for pairwise compatibility; the new
artifacts use the accurate public term context_provider and record whether
the source is graph, search, wiki/RAG, hybrid, or another declared kind.

Alternative considered: replace the pairwise evaluation bundle with a new
multi-arm schema. Rejected because it would duplicate pairing, check
projection, qualification, and invalid-evidence behavior that already passes.

2. Separate eligibility from outcome execution

A capability probe runs before experiment planning and emits a bounded record:

• provider/version/configuration and interface kind;
• local, hosted, or enterprise operating mode;
• index command and exact revision/freshness evidence;
• available tool names and whether calls are observable;
• setup time, bounded storage, data egress, authentication posture, and known
  limitations;
• eligible cohort or an explicit exclusion reason.

The first candidate inventory is:

Candidate	Initial classification
Plain repository tools	Required baseline
CodeVetter structural context	Required local treatment
CodeGraph	Local agent-readable candidate
Graphify	Local agent-readable candidate
Repowise	Local/self-hostable agent-readable candidate
DeepWiki	Separate candidate only if its MCP/CLI path is reproducible
Sourcegraph	Separate hosted/enterprise cohort requiring explicit approval
pgGraph / HydraDB	Excluded infrastructure, not end-to-end providers

Eligibility is determined from a live probe at implementation time; this table
is not a compatibility or quality claim.

3. Use a staged, deterministic crossover design

Stage 0 performs capability probes and produces plans only. Stage 1 is a
free/local feasibility run over a preregistered four-task slice: two API and
two browser tasks spanning at least four failure categories. It uses two
repetitions per arm and admits at most the baseline plus three treatment arms.

Stage 2 may run all 30 tasks with at least three repetitions per admitted arm
only after Stage 1 passes setup, freshness, contamination, cleanup, and
observability gates. The planner calculates exact attempts and conservative
cost bounds before requesting approval. Hosted or paid providers are a
separate cohort so local/privacy differences are not hidden in one ranking.

The implemented Stage 1 crossover schedule contains A/B arms only. It can test
adapter feasibility and report descriptive outcomes, but it cannot qualify a
provider because the existing pairwise policy requires independent A/A noise
evidence. Before any Stage 2 approval, its plan must therefore add a
preregistered A/A schedule with fresh workspaces and sessions; repeating or
relabeling Stage 1 A/B evidence is not sufficient.

Arm order is derived deterministically from the experiment identity using a
balanced Latin-square schedule where possible and a declared balanced rotation
otherwise. Fresh workspaces and new agent sessions prevent conversation or
generated-instruction carryover.

Each treatment binds a separate content snapshot to every task fixture. A
single provider-wide snapshot is invalid because the compact corpus tasks have
different source trees even when they share one provenance revision. A
content-addressed per-task snapshot may be reused read-only across repetitions;
provider configuration, agent session, and mutable workspace still remain
fresh for every arm.

Alternative considered: run providers sequentially in a convenient fixed
order. Rejected because model/provider drift, cache warmth, and learning or
rate-limit effects could become provider effects.

4. Keep task outcome primary and preregister diagnostics

Every arm uses the immutable task packet and hidden acceptance inventory.
Success requires all required checks and no regression. Operational failure,
setup failure, missing checks, check errors, timeouts, cancellation, and
cleanup failure retain their existing distinct states.

The plan preregisters secondary diagnostics:

• setup/index success and latency;
• tool calls and context-tool calls;
• files inspected and modified;
• relevant-file recall only for tasks with a predeclared relevant-file set;
• input/output tokens, elapsed time, and cost when actually captured;
• invalid, stale, missing, and contaminated arm counts.

Known-good changed files are not automatically used as retrieval ground truth:
a valid solution may touch a different boundary. Relevant-file sets require a
separate pre-run annotation and are diagnostic only.

5. Qualify each provider against baseline, then adjust the family

Each provider must pass the existing pairwise identity, sample, regression,
and A/A noise gates. The comparison plan additionally declares a family-wise
multiple-comparison method before runs begin. The initial design uses Holm's
step-down correction over provider-versus-baseline success hypotheses because
it controls family-wise error without assuming independent arms.

Raw outcome counts and unadjusted descriptive intervals remain visible, but a
provider is not ranked as qualified unless both its pairwise gates and adjusted
family decision pass. The report preserves negative, null, unavailable, and
excluded results.

Alternative considered: rank raw success rates. Rejected because repeated
provider comparisons make favorable noise increasingly likely.

6. Keep execution and scoring separated

flowchart LR
    A[Capability probes] --> B[Immutable experiment plan]
    B --> C[Deterministic arm schedule]
    C --> D[Existing agent-task runner]
    D --> E[Immutable v2 receipts]
    E --> F[Existing pairwise composer and scorer]
    F --> G[Multi-provider aggregator]
    G --> H[JSON / Markdown / local HTML]

Loading

Planning never launches agents. The runner never interprets success. The
pairwise scorer and new aggregator consume local immutable evidence and make no
network requests. This preserves deterministic rescoring and permits a ground-
truth correction without repeating expensive runs.

Risks / Trade-offs

• Provider interfaces change during the experiment → Pin exact versions,
  configuration hashes, tool inventories, and snapshot identities; invalidate
  drift instead of silently combining it.
• Global MCP configuration contaminates the baseline or another arm → Use
  isolated per-run agent configuration roots and reject undeclared context-tool
  calls or injected instruction files.
• Hosted and local products are not operationally comparable → Separate
  cohorts and expose privacy, setup, and data-egress differences beside outcome
  results.
• The 30 compact tasks favor localized fixes → Limit claims to this corpus
  and add broader tasks only through the existing qualification contract.
• Repeated multi-arm runs become expensive → Stop after Stage 1 unless
  exact attempt, time, and cost bounds justify Stage 2 and receive approval.
• Tool-call diagnostics may be incomplete → Mark missing measurements
  unavailable; do not infer them from model prose or stdout.
• External license or terms restrict automation/publication → Record the
  restriction in the probe and exclude or keep the provider's result private.

Migration Plan

1. Add new closed probe, plan, and aggregate-score contracts without changing
   existing run receipts or pairwise evaluation bundles.
2. Add hermetic synthetic fixtures proving eligibility, scheduling,
   contamination, missing-arm, aggregation, adjustment, and deterministic
   output behavior.
3. Add baseline and CodeVetter experiment adapters, then probe external
   candidates one at a time outside the production dependency graph.
4. Run Stage 0 and publish only the eligibility/feasibility plan locally.
5. Request explicit approval before Stage 1 execution and again before any
   Stage 2, paid, hosted, or public result.

Rollback is removal of the additive comparison contracts and scripts. Existing
corpus, runner, receipts, structural-context score, CLI, MCP, and desktop paths
remain unchanged.

Open Questions

• Which exact four qualified tasks best represent the Stage 1 slice will be
  selected from the current corpus by a deterministic coverage report during
  implementation.
• External candidates admitted after the capability probe may require a
  private-only result if their licenses or terms prohibit public comparison.

---

Specs

context-provider-comparison/spec.md

Purpose

Define a reproducible experiment that determines whether an agent-readable
code-context provider changes executable coding-task outcomes under fixed,
inspectable conditions without confusing retrieval activity with correctness.

ADDED Requirements

Requirement: Provider eligibility is explicit and evidence-backed

The experiment system SHALL include a context provider only after a bounded
capability probe records its provider identity, exact version, context kind,
machine-readable interface, configuration identity, repository-indexing mode,
freshness evidence, allowed tools, data-egress posture, setup result, and known
limitations. It MUST classify storage engines without a complete agent-facing
retrieval path as infrastructure and exclude them from provider outcome arms.

Scenario: Local MCP provider passes the probe

• WHEN a provider exposes a bounded MCP interface, indexes the exact task revision, reports an exact version and configuration identity, and permits tool-call observation
• THEN the system records it as eligible for the declared local provider cohort

Scenario: Human-only wiki has no reproducible agent interface

• WHEN a product produces human-readable pages but exposes no stable CLI, API, or MCP interface for the tested agent
• THEN the system excludes it from the agent-context cohort and records the missing capability rather than inventing an adapter

Scenario: Generic graph storage is proposed as a provider

• WHEN a system stores or queries graphs but does not construct and deliver task-relevant code context to the agent
• THEN the system classifies it as infrastructure and does not score it against complete context providers

Requirement: Multi-arm experiments bind immutable common and provider identities

The system SHALL bind every experiment to one corpus identity, task and ground-
truth identities, agent/model/prompt configuration, environment, schedule,
qualification policy, and declared provider cohort. Every provider arm SHALL
also bind its provider version, configuration, context snapshot or index,
indexed repository revision, context policy, and allowed tool set. Missing,
duplicated, stale, or mismatched identities MUST invalidate the affected arm.
Snapshot identity SHALL be task-bound; a provider-wide snapshot MUST NOT stand
in for several task fixtures that contain different source trees.

Scenario: Provider arms differ only in context policy

• WHEN baseline and provider runs share every common identity and each provider arm carries a current provider-specific context identity
• THEN the system admits the runs to the declared comparison

Scenario: Provider index is stale

• WHEN a provider snapshot or index names a repository revision different from the task revision
• THEN the system excludes the arm from outcome comparison and reports stale context

Requirement: Context isolation fails closed

Each arm SHALL run in a fresh workspace and agent session with only its declared
context interface available. The baseline MUST have all special context
providers disabled. A provider arm MUST NOT access another provider's tools,
cache, generated instructions, or retained conversation state. Any undeclared
context tool call or cross-arm state MUST mark the run contaminated.

Scenario: Baseline invokes a provider tool

• WHEN baseline diagnostics record a context-provider tool call or injected provider artifact
• THEN the system marks the baseline contaminated and refuses to use its paired outcomes

Scenario: One provider remains configured in another arm

• WHEN an arm can access a context interface not declared by its provider policy
• THEN the system invalidates the arm even if all executable checks pass

Requirement: Scheduling is deterministic, balanced, and staged

The system SHALL create a deterministic schedule that balances arm order across
tasks and repeated trials. It SHALL complete a preregistered free/local
feasibility stage before permitting a full-corpus or paid/hosted stage. Any
stage that can incur non-zero or unknown cost MUST expose exact run counts,
conservative cost bounds, credential-name availability, and an approval
identity before execution.

Scenario: Feasibility stage is planned

• WHEN eligible free/local arms and a bounded task subset are selected
• THEN repeated planning produces the same ordered schedule and plan identity without launching an agent or provider

Scenario: Full trial has unknown cost

• WHEN any selected arm has paid or unknown pricing and no matching approval is supplied
• THEN the system produces a blocked plan and starts no experiment run

Requirement: Executable behavior remains the outcome authority

The system SHALL count task success only from complete hidden acceptance checks
and preserved regression checks. It SHALL report provider setup success,
invalid and contaminated arms, treatment-only wins, baseline-only wins,
cross-provider outcome deltas, and per-task results. Retrieval recall, files
inspected or modified, tool calls, latency, tokens, and cost SHALL remain
secondary diagnostics and MUST NOT be substituted for executable success.

Scenario: Provider uses fewer tokens but fails a hidden check

• WHEN a provider arm records lower token use than baseline but misses any required acceptance check
• THEN the system reports the failed task outcome and treats token use only as a diagnostic

Scenario: Relevant-file ground truth is absent

• WHEN a task has no preregistered relevant-file set
• THEN the system reports retrieval recall as unavailable instead of deriving it from the agent's edits or known-good patch after the run

Requirement: Qualification controls claims and multiple comparisons

The experiment SHALL preregister minimum task breadth, complete repetitions,
A/A noise limits, success-rate and regression gates, provider eligibility, and
the method used to control multi-provider comparisons. Synthetic, feasibility,
underpowered, invalid, or excessively noisy evidence MUST remain descriptive.
The report MUST preserve negative and null results and MUST NOT rank providers
whose evidence failed qualification.

A feasibility schedule without independent A/A arms MUST remain descriptive.
Any later plan seeking a qualified comparison SHALL preregister and execute
independent A/A arms under the same isolation, identity, and completeness
rules; it MUST NOT derive A/A noise from A/B repetitions after execution.

Scenario: One provider has a favorable but underpowered result

• WHEN its descriptive success delta is positive but the declared sample or noise gate is not met
• THEN the system labels the result unqualified and publishes no winner claim

Scenario: Several providers are compared

• WHEN the experiment evaluates more than one provider against the same baseline
• THEN the report applies the preregistered multiple-comparison policy and exposes both raw and adjusted qualification results

Requirement: Scoring and reporting are deterministic and non-executing

The comparison system SHALL consume only explicitly supplied local plans,
capability probes, immutable receipts, and existing evaluation scores. It SHALL
perform no agent launch, provider call, repository mutation, hidden-check
execution, or network request while scoring. JSON, Markdown, and optional
self-contained HTML outputs MUST share one normalized scorecard and expose
provider identities, evidence gaps, invalid arms, scheduling, outcome deltas,
diagnostics, qualification, and limitations.

Scenario: The same evidence is rescored

• WHEN identical plans, probes, receipts, ground truth, and scorer bytes are supplied twice
• THEN the normalized comparison and evidence identities are byte-stable

Scenario: Provider receipt is missing

• WHEN a scheduled arm has no declared terminal receipt
• THEN the report records the missing arm and refuses a complete-provider comparison rather than silently shrinking the denominator

Requirement: Trial artifacts preserve privacy and dependency boundaries

Committed experiment artifacts MUST NOT contain credentials, repository secrets,
raw private source, absolute paths, provider account identifiers, or unbounded
model output. Provider integrations SHALL remain experiment adapters rather
than production dependencies, and hosted data egress MUST require an explicit
declared policy and approval.

Scenario: Provider requires a credential

• WHEN a plan references a required credential name
• THEN it records only bounded availability and hashed environment identity, never the credential value

Scenario: Hosted provider would ingest task source

• WHEN the provider's declared data-egress policy is absent or unapproved
• THEN the experiment blocks that arm before indexing or agent execution

---

Spec-driven tracking issue. All design content lives here; no local openspec/ docs.