feat(context-providers): preregister A/A noise arms and recoverable failure diagnostics - #168
feat(context-providers): preregister A/A noise arms and recoverable failure diagnostics#168sarthakagrawal927 wants to merge 711 commits into
Conversation
Ship fleet GEO surfaces so AI crawlers can read product truth without JS. Includes agent-edge worker wiring where applicable, robots/sitemap pointers, and static fallbacks for SPA shells.
Crisp path-based monograms for browser tabs and bookmarks; SVG primary, ICO legacy fallback.
The pre-push secret scan greps all tracked files and tripped on the hardcoded detector test fixtures (AKIA…/xoxb-…) inside the product's own secret-detection module, blocking every push. Scope the exclusion to that one file; real secrets elsewhere still abort the push. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Social crawlers poorly support SVG og:image.
Remove the single hand-written SoftwareApplication JSON-LD block from Layout.astro and inject the fleet-generated marked block (<!-- fleet-jsonld:start/end -->) from apply-agent-surfaces.mjs. The new block uses a @graph with the fleet Organization publisher (SaaS Maker/Foundry, sameAs → sassmaker.com + GitHub repo) plus the product SoftwareApplication node. Preserves applicationCategory and offers from the hand-written block via registry fields. Build verified: pnpm build passes, dist/index.html contains exactly one fleet-generated ld+json block (the FAQPage page-level schema remains separate and unaffected). Idempotent on re-run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Part of fleet-wide logo wrap-up.
Install SVG/ICO/apple-touch from fleet brand marks and ensure layout metadata/links point at them.
Expand the /benchmark stub into the full story anchored to real scored data from benchmark/cases (27 cases, 29 hand-labeled findings) and benchmark/reviews (codevetter + raw-claude). - apps/landing-page-astro/src/pages/benchmark.astro: intro, methodology, headline results (CodeVetter 29/29 = 100% catch, 0.433 precision; raw Claude 27/29 = 93.1%, 0.397 precision), per-reviewer per-case table, the 27 cases summary, an honest limitations section, and a download link. Adds a schema.org Dataset JSON-LD block. - public/benchmark/codevetter-benchmark-v1.json: downloadable dataset (cases + expected findings + scoring method + CC0 license note). - src/data/benchmark-results.json: per-reviewer scorecards consumed by the page, generated from scripts/run-public-benchmark.mjs output. - scripts/generate-benchmark-dataset.mjs: reproducible generator for both JSON artifacts. - docs/show-hn.md: title options + first-comment draft (plain, technical). Verified: pnpm build passes (5 pages), dataset link 200s locally, JSON parses, Dataset JSON-LD present on /benchmark.
Layout.astro: add clampDescription() to enforce 70–160 char range
on meta, og, and twitter descriptions. Fixes /download page having
a 48-char description.
Hero.astro + Bento.astro: wrap code-display divs in <code> tags so
seo-audit.sh correctly identifies them as code samples and doesn't
flag SQL template literals (${token}, ${id}) as SSR leaks.
'/benchmark/' anywhere was broader than needed — a real secret under any future benchmark-named dir would slip through. Scope to the case corpus and the published dataset dir. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Static /faq page answering the top questions users ask AI assistants about AI code review tools. Includes FAQPage structured data for LLM and search engine indexing. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds the OpenSpec proposal/design/specs/tasks for the ccusage accounting change on the verification-experiments branch. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds the ccusage dependency for the verification-experiments branch. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds the ccusage sidecar preparation script + tests, updates Tauri config, CI, release workflow, .gitignore, and dependabot config for the new dependency. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Biome formatting fixes from the pre-push hook. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds local_usage command + tests and UI surface for ccusage sidecar integration on the verification-experiments branch. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Iterates local-usage UI and Home.tsx integration for the ccusage sidecar. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Iterates history.rs and main.rs integration for the ccusage sidecar. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Iterates local-usage command, desktop UI wiring, data model docs, OpenSpec adoption plans, and project status around the ccusage accounting sidecar. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The change-size step needs the previous commit (github.event.before), but the shallow checkout does not always retain it. Add an explicit fetch so CODE_HEALTH_BASE resolves. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Migrated spec content to GitHub tracking issues. No local openspec/ directory, no openspec CLI. See fleet-workspace AGENTS.md for the new spec-driven workflow. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… agent-friendly 404s Improves is-agentic.com score by adding: - /openapi.json endpoint - JSON error responses for unknown /api/* paths - Vary: Accept header for markdown negotiation - Agent-friendly 404 with markdown body - "When to use this" guidance in llms.txt - openapi field in api-ai.json Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…d search, CLI docs - Add response schemas and error models to OpenAPI spec - Add RateLimit headers to API responses - Fix markdown negotiation Vary header (run_worker_first) - Add product name to H1 for brand search accuracy - Add CLI section to llms.txt Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ailure diagnostics Closes the two Stage 1 gaps recorded in benchmarks/context-providers/README.md so a later Stage 2 can be planned. Harness only — no benchmark run is executed and no result is published by this change. Preregistered A/A noise arms: - The planner schedules independent treatment-vs-treatment A/A arms beside the A/B crossover (`--aa-repetitions`, default 2 at the full stage, opt-in at feasibility). A full-stage plan declaring none is blocked with `aa-schedule-missing`, and 1 repetition is rejected outright. - Every A/A arm gets its own workspace, agent session, and tool-configuration identity, so A/A noise cannot be derived by relabelling A/B repetitions: the projector rejects any attempt whose comparison/arm disagrees with the schedule, and `family_qualified` now requires complete A/A pairs. - Plans without A/A arms keep their previous identity and counts, so the pinned Stage 0 and Stage 1 plan identities are unchanged. - Cost derivation and attempt bounds account for the doubled treatment attempts, so a Stage 2 plan re-states its own cost for re-approval. Recoverable failure evidence: - The runner writes redacted, bounded stdout/stderr to `<run-root>/diagnostics/attempt-<n>.json`, verified against the receipt's `stdout_sha256`/`stderr_sha256` before retention. Receipts are unchanged and a hash mismatch fails the run rather than storing unbound output. Verification on this branch: pnpm test:corpus-contracts 67/67 pass (61 on origin/main), pnpm test:context-provider-plan 11/11, biome check clean, quality:cycles clean, quality:duplication within threshold, knip:strict unchanged from the origin/main baseline. Refs #159. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Not merging —
|
1517be7 to
b4284d2
Compare
…ailure diagnostics Lands the change from #168, whose branch pointed into history that a force-push of main had orphaned, so GitHub computed its diff as 100 files across two divergent ancestries. The actual change is the 11 files here. Harness only. It executes no benchmark run and publishes no result. The planner schedules independent treatment-vs-treatment A/A arms beside the A/B crossover. Each A/A arm gets its own workspace, agent session and tool-configuration identity, so A/A noise cannot be derived by relabelling A/B repetitions — the projector rejects any attempt whose comparison or arm disagrees with the schedule. A full-stage plan declaring no A/A arms is blocked; one repetition is rejected outright. Plans without A/A arms keep their previous identity and counts, so pinned plan identities are unchanged. The runner also writes redacted, bounded stdout and stderr to diagnostics/attempt-<n>.json, verified against the receipt's hashes before retention. Stage 1 could not explain its two treatment agent failures because the immutable receipt keeps only bounded hashes. Receipts are unchanged, and a hash mismatch fails the run rather than storing unbounded output. Four functions in the original change were above this repository's cognitive complexity ceiling of 20 — one at 38 — and none had been caught, because the change verified lint, cycles, duplication and the test suites but not quality:complexity. All four are split here: schedule validation into per-entry, key-derivation and tally steps; the evaluation projector into separate A/B and A/A collectors; the plan CLI's flag chain into a table; and the runner's preflight into plan approval and adapter pinning. 67/67 corpus contracts and 11/11 plan tests pass, as the original change reported. Complexity, duplication, cycles, docs, change-size and lint are green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Landed on main as e7ccfc3 rather than merged here. Why the merge button could not be used: I force-pushed main earlier to purge private-repository artifacts from history, which rewrote every commit SHA. This branch still points into the orphaned history, so GitHub computed the diff across two divergent ancestries — 711 commits each way, 100 files, and none of the files this PR's own description names. The change itself was always the 11 files it claimed. That breakage is mine, not this branch's. Review notes, all acted on before landing: Four functions were above the repo's cognitive complexity ceiling of 20 — The A/A design holds up on reading. Separate workspace, session and tool-configuration identity per arm is the right call — deriving A/A noise by relabelling A/B repetitions is precisely what the measurement exists to rule out, and the projector rejecting a mismatched 67/67 corpus contracts and 11/11 plan tests pass, matching what you reported. Complexity, duplication, cycles, docs, change-size and lint are green on main. |
Closes the two Stage 1 gaps recorded in
benchmarks/context-providers/README.mdso a later Stage 2 can be planned. Harness only — this PR executes no benchmark run and publishes no result. Stage 2 (and any paid, hosted, or public run) still needs its own explicit approval per #159.Preregistered A/A noise arms
--aa-repetitions; default 2 at the full stage, opt-in at feasibility). A full-stage plan declaring none is blocked withaa-schedule-missing; 1 repetition is rejected outright.comparison/armdisagrees with the schedule, andfamily_qualifiednow requires complete A/A pairs.Recoverable failure evidence
Stage 1 could not explain its two treatment agent failures because the immutable receipt keeps only bounded hashes. The runner now also writes redacted, bounded stdout/stderr to
<run-root>/diagnostics/attempt-<n>.json, verified against the receipt'sstdout_sha256/stderr_sha256before retention. Receipts are unchanged; a hash mismatch fails the run rather than storing unbound output.Verification
Run in a clean worktree at
origin/mainwith the patch applied, compared against the same worktree without it:origin/mainbaselinepnpm test:corpus-contractspnpm test:context-provider-planpnpm exec biome check .pnpm quality:cyclespnpm quality:duplicationpnpm knip:strictRefs #159.