Skip to content

feat(context-providers): preregister A/A noise arms and recoverable failure diagnostics - #168

Closed
sarthakagrawal927 wants to merge 711 commits into
mainfrom
feat/context-provider-aa-noise-and-diagnostics
Closed

feat(context-providers): preregister A/A noise arms and recoverable failure diagnostics#168
sarthakagrawal927 wants to merge 711 commits into
mainfrom
feat/context-provider-aa-noise-and-diagnostics

Conversation

@sarthakagrawal927

Copy link
Copy Markdown
Member

Closes the two Stage 1 gaps recorded in benchmarks/context-providers/README.md so a later Stage 2 can be planned. Harness only — this PR executes no benchmark run and publishes no result. Stage 2 (and any paid, hosted, or public run) still needs its own explicit approval per #159.

Preregistered A/A noise arms

  • The planner schedules independent treatment-vs-treatment A/A arms beside the A/B crossover (--aa-repetitions; default 2 at the full stage, opt-in at feasibility). A full-stage plan declaring none is blocked with aa-schedule-missing; 1 repetition is rejected outright.
  • Every A/A arm gets its own workspace, agent session, and tool-configuration identity, so A/A noise cannot be derived by relabelling A/B repetitions — the projector rejects any attempt whose comparison/arm disagrees with the schedule, and family_qualified now requires complete A/A pairs.
  • Plans without A/A arms keep their previous identity and counts, so the pinned Stage 0 and Stage 1 plan identities are unchanged.
  • Cost derivation and attempt bounds account for the doubled treatment attempts, so a Stage 2 plan re-states its own cost for re-approval.

Recoverable failure evidence

Stage 1 could not explain its two treatment agent failures because the immutable receipt keeps only bounded hashes. The runner now also writes redacted, bounded stdout/stderr to <run-root>/diagnostics/attempt-<n>.json, verified against the receipt's stdout_sha256/stderr_sha256 before retention. Receipts are unchanged; a hash mismatch fails the run rather than storing unbound output.

Verification

Run in a clean worktree at origin/main with the patch applied, compared against the same worktree without it:

Check origin/main baseline This branch
pnpm test:corpus-contracts 61/61 pass 67/67 pass
pnpm test:context-provider-plan 11/11 pass
pnpm exec biome check . clean (776 files) clean (776 files)
pnpm quality:cycles clean clean
pnpm quality:duplication exit 0 exit 0
pnpm knip:strict fails (pre-existing) byte-identical output to baseline

Refs #159.

sarthakagrawal927 and others added 30 commits July 15, 2026 17:59
Ship fleet GEO surfaces so AI crawlers can read product truth without
JS. Includes agent-edge worker wiring where applicable, robots/sitemap
pointers, and static fallbacks for SPA shells.
Crisp path-based monograms for browser tabs and bookmarks; SVG primary, ICO legacy fallback.
The pre-push secret scan greps all tracked files and tripped on the
hardcoded detector test fixtures (AKIA…/xoxb-…) inside the product's own
secret-detection module, blocking every push. Scope the exclusion to that
one file; real secrets elsewhere still abort the push.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Social crawlers poorly support SVG og:image.
Remove the single hand-written SoftwareApplication JSON-LD block from
Layout.astro and inject the fleet-generated marked block
(<!-- fleet-jsonld:start/end -->) from apply-agent-surfaces.mjs.

The new block uses a @graph with the fleet Organization publisher
(SaaS Maker/Foundry, sameAs → sassmaker.com + GitHub repo) plus the
product SoftwareApplication node. Preserves applicationCategory and
offers from the hand-written block via registry fields.

Build verified: pnpm build passes, dist/index.html contains exactly
one fleet-generated ld+json block (the FAQPage page-level schema
remains separate and unaffected). Idempotent on re-run.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Install SVG/ICO/apple-touch from fleet brand marks and ensure layout
metadata/links point at them.
Expand the /benchmark stub into the full story anchored to real scored data
from benchmark/cases (27 cases, 29 hand-labeled findings) and
benchmark/reviews (codevetter + raw-claude).

- apps/landing-page-astro/src/pages/benchmark.astro: intro, methodology,
  headline results (CodeVetter 29/29 = 100% catch, 0.433 precision; raw
  Claude 27/29 = 93.1%, 0.397 precision), per-reviewer per-case table, the
  27 cases summary, an honest limitations section, and a download link.
  Adds a schema.org Dataset JSON-LD block.
- public/benchmark/codevetter-benchmark-v1.json: downloadable dataset
  (cases + expected findings + scoring method + CC0 license note).
- src/data/benchmark-results.json: per-reviewer scorecards consumed by the
  page, generated from scripts/run-public-benchmark.mjs output.
- scripts/generate-benchmark-dataset.mjs: reproducible generator for both
  JSON artifacts.
- docs/show-hn.md: title options + first-comment draft (plain, technical).

Verified: pnpm build passes (5 pages), dataset link 200s locally, JSON
parses, Dataset JSON-LD present on /benchmark.
Layout.astro: add clampDescription() to enforce 70–160 char range
on meta, og, and twitter descriptions. Fixes /download page having
a 48-char description.

Hero.astro + Bento.astro: wrap code-display divs in <code> tags so
seo-audit.sh correctly identifies them as code samples and doesn't
flag SQL template literals (${token}, ${id}) as SSR leaks.
'/benchmark/' anywhere was broader than needed — a real secret under any
future benchmark-named dir would slip through. Scope to the case corpus
and the published dataset dir.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Static /faq page answering the top questions users ask AI assistants
about AI code review tools. Includes FAQPage structured data for LLM
and search engine indexing.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
sarthakagrawal927 and others added 24 commits August 16, 2026 04:31
Adds the OpenSpec proposal/design/specs/tasks for the ccusage accounting
change on the verification-experiments branch.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds the ccusage dependency for the verification-experiments branch.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds the ccusage sidecar preparation script + tests, updates Tauri config,
CI, release workflow, .gitignore, and dependabot config for the new
dependency.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Biome formatting fixes from the pre-push hook.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds local_usage command + tests and UI surface for ccusage sidecar
integration on the verification-experiments branch.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Iterates local-usage UI and Home.tsx integration for the ccusage sidecar.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Iterates history.rs and main.rs integration for the ccusage sidecar.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Iterates local-usage command, desktop UI wiring, data model docs,
OpenSpec adoption plans, and project status around the ccusage accounting
sidecar.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The change-size step needs the previous commit (github.event.before), but
the shallow checkout does not always retain it. Add an explicit fetch so
CODE_HEALTH_BASE resolves.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Migrated spec content to GitHub tracking issues. No local openspec/
directory, no openspec CLI. See fleet-workspace AGENTS.md for the new
spec-driven workflow.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… agent-friendly 404s

Improves is-agentic.com score by adding:
- /openapi.json endpoint
- JSON error responses for unknown /api/* paths
- Vary: Accept header for markdown negotiation
- Agent-friendly 404 with markdown body
- "When to use this" guidance in llms.txt
- openapi field in api-ai.json

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…d search, CLI docs

- Add response schemas and error models to OpenAPI spec
- Add RateLimit headers to API responses
- Fix markdown negotiation Vary header (run_worker_first)
- Add product name to H1 for brand search accuracy
- Add CLI section to llms.txt

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ailure diagnostics

Closes the two Stage 1 gaps recorded in benchmarks/context-providers/README.md
so a later Stage 2 can be planned. Harness only — no benchmark run is executed
and no result is published by this change.

Preregistered A/A noise arms:
- The planner schedules independent treatment-vs-treatment A/A arms beside the
  A/B crossover (`--aa-repetitions`, default 2 at the full stage, opt-in at
  feasibility). A full-stage plan declaring none is blocked with
  `aa-schedule-missing`, and 1 repetition is rejected outright.
- Every A/A arm gets its own workspace, agent session, and tool-configuration
  identity, so A/A noise cannot be derived by relabelling A/B repetitions: the
  projector rejects any attempt whose comparison/arm disagrees with the
  schedule, and `family_qualified` now requires complete A/A pairs.
- Plans without A/A arms keep their previous identity and counts, so the pinned
  Stage 0 and Stage 1 plan identities are unchanged.
- Cost derivation and attempt bounds account for the doubled treatment
  attempts, so a Stage 2 plan re-states its own cost for re-approval.

Recoverable failure evidence:
- The runner writes redacted, bounded stdout/stderr to
  `<run-root>/diagnostics/attempt-<n>.json`, verified against the receipt's
  `stdout_sha256`/`stderr_sha256` before retention. Receipts are unchanged and
  a hash mismatch fails the run rather than storing unbound output.

Verification on this branch: pnpm test:corpus-contracts 67/67 pass (61 on
origin/main), pnpm test:context-provider-plan 11/11, biome check clean,
quality:cycles clean, quality:duplication within threshold, knip:strict
unchanged from the origin/main baseline.

Refs #159.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sarthakagrawal927

Copy link
Copy Markdown
Member Author

Not merging — quality:complexity fails, and the change causes it

CI lint-and-typecheckCode health step, verbatim (run 32602737946):

> code-reviewer@1.1.2 quality:change-size
Change-size gate: PASS (11/40 files, 809/4000 additions, 889/6000 gross lines)

> code-reviewer@1.1.2 knip:strict
Configuration hints (82)      <- unchanged from the origin/main baseline

> code-reviewer@1.1.2 quality:complexity
scripts/agent-task-corpus/context-provider-evaluation.mjs:182 projectContextProviderEvaluationBundle has complexity 38 (new violation)
scripts/agent-task-corpus/context-provider-plan-cli.mjs:40 parseArgs has complexity 28 (baseline 25)
scripts/agent-task-corpus/context-provider-run-cli.mjs:27 runContextProviderPlan has complexity 21 (new violation)
scripts/agent-task-corpus/contracts.mjs:1879 validateContextSchedule has complexity 34 (new violation)
 ELIFECYCLE  Command failed with exit code 1.

scripts/check-changed-complexity.mjs grandfathers a changed function at its origin/main score but caps everything at max(20, previous) (biome.code-health.jsonnoExcessiveCognitiveComplexity.maxAllowedComplexity: 20). Three of these four functions were under 20 on origin/main and this branch pushes them over; parseArgs regresses 25 → 28. There is no accepted-debt allowlist to record an exception in, so the gate can only be satisfied by extracting the new branching.

Everything else is green — see the verification table in the PR description. Leaving this open rather than merging or force-merging: reducing cognitive complexity inside the A/A scheduling, projection, and schedule-validation paths is a real refactor of correctness-bearing statistics code and should be a deliberate follow-up, not a rubber stamp.

Suggested extractions, smallest first:

  • contracts.mjs validateContextSchedule — split the per-entry A/A vs A/B key/group derivation into a helper alongside the existing validateContextScheduleArm.
  • context-provider-evaluation.mjs projectContextProviderEvaluationBundle — the A/B pair loop and the A/A pair loop are independent; each can become its own function returning runs.
  • context-provider-run-cli.mjs runContextProviderPlan — the new diagnostics write is a natural extraction point.
  • context-provider-plan-cli.mjs parseArgs — the value-taking flag list is now long enough to be a table lookup.

sarthakagrawal927 added a commit that referenced this pull request Aug 23, 2026
…ailure diagnostics

Lands the change from #168, whose branch pointed into history that a force-push of
main had orphaned, so GitHub computed its diff as 100 files across two divergent
ancestries. The actual change is the 11 files here.

Harness only. It executes no benchmark run and publishes no result.

The planner schedules independent treatment-vs-treatment A/A arms beside the A/B
crossover. Each A/A arm gets its own workspace, agent session and tool-configuration
identity, so A/A noise cannot be derived by relabelling A/B repetitions — the
projector rejects any attempt whose comparison or arm disagrees with the schedule.
A full-stage plan declaring no A/A arms is blocked; one repetition is rejected
outright. Plans without A/A arms keep their previous identity and counts, so pinned
plan identities are unchanged.

The runner also writes redacted, bounded stdout and stderr to
diagnostics/attempt-<n>.json, verified against the receipt's hashes before
retention. Stage 1 could not explain its two treatment agent failures because the
immutable receipt keeps only bounded hashes. Receipts are unchanged, and a hash
mismatch fails the run rather than storing unbounded output.

Four functions in the original change were above this repository's cognitive
complexity ceiling of 20 — one at 38 — and none had been caught, because the
change verified lint, cycles, duplication and the test suites but not
quality:complexity. All four are split here: schedule validation into per-entry,
key-derivation and tally steps; the evaluation projector into separate A/B and A/A
collectors; the plan CLI's flag chain into a table; and the runner's preflight into
plan approval and adapter pinning.

67/67 corpus contracts and 11/11 plan tests pass, as the original change reported.
Complexity, duplication, cycles, docs, change-size and lint are green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sarthakagrawal927

Copy link
Copy Markdown
Member Author

Landed on main as e7ccfc3 rather than merged here.

Why the merge button could not be used: I force-pushed main earlier to purge private-repository artifacts from history, which rewrote every commit SHA. This branch still points into the orphaned history, so GitHub computed the diff across two divergent ancestries — 711 commits each way, 100 files, and none of the files this PR's own description names. The change itself was always the 11 files it claimed. That breakage is mine, not this branch's.

Review notes, all acted on before landing:

Four functions were above the repo's cognitive complexity ceiling of 20 — projectContextProviderEvaluationBundle at 38, validateContextSchedule at 34, parseArgs at 28 (from a baseline of 25), runContextProviderPlan at 21. The verification table in the description covers lint, cycles, duplication and both test suites but not quality:complexity, which is why none surfaced. All four are split in e7ccfc3: schedule validation into per-entry/key-derivation/tally steps, the projector into separate A/B and A/A collectors, the flag chain into a table, and the runner preflight into plan-approval and adapter-pinning halves.

The A/A design holds up on reading. Separate workspace, session and tool-configuration identity per arm is the right call — deriving A/A noise by relabelling A/B repetitions is precisely what the measurement exists to rule out, and the projector rejecting a mismatched comparison/arm closes that off structurally rather than by convention.

67/67 corpus contracts and 11/11 plan tests pass, matching what you reported. Complexity, duplication, cycles, docs, change-size and lint are green on main.

@sarthakagrawal927
sarthakagrawal927 deleted the feat/context-provider-aa-noise-and-diagnostics branch August 23, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant