Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions PROJECT_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,26 @@ Internal (fleet):
with 913 Rust tests, 665 frontend tests plus the 20-scenario live
qualification, lint, typecheck, production build, docs, and strict OpenSpec.

- **2026-08-10 — Revision-bound optimization contribution closeout:** extended
the local performance campaign with a mandatory post-promotion candidate
challenge and three closed CLI/MCP operations. The new local receipt binds
the kept campaign record, baseline, candidate commit, diff, optional T-Rex
preview evidence, and one canonical GitHub PR while preserving correctness,
performance, patch quality, head freshness, checks, review threads,
approvals, and merge authority as independent fail-closed gates. Candidate
source must be committed before promotion/challenge; stale heads, missing
required T-Rex evidence, unobserved checks, actionable feedback, and
symlink/path escapes cannot become ready. GitHub access is one fixed read-only
GraphQL query with explicit refresh—no polling, app install, comment, review
request, thread resolution, required check, merge, or deploy. Live read-only
dogfood against Marked PR #4048 caught and fixed a classifier error: Vercel
fork authorization is now `approval_required`, not a code failure; two older
inline threads remain visible as outdated. Raw evidence stays local and
upstream maintainers receive no additional workflow. Qualified candidate
comparisons retain target plus available smaller-input/allocation controls,
and the full Marked-shaped fixture proves reviewed-head invalidation,
simpler-candidate selection, bounded feedback learning, and regeneration of
a current receipt-backed publication projection.
- **2026-08-09 — Shared lint baseline:** Adopted the Fleet Ultracite baseline
for core TypeScript, React, and test code. Explicit compatibility exceptions
preserve current behavior while 662 files pass with zero diagnostics;
Expand Down
68 changes: 67 additions & 1 deletion docs/development/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -863,7 +863,7 @@ pnpm --silent runtime:campaign -- promote \
--hypothesis "Remove one redundant allocation without changing output." --json
```

The same six operations are available from `runtime:mcp`. Start that server
The same six campaign operations are available from `runtime:mcp`. Start that server
with `--repo /path/to/candidate --incumbent-repo /path/to/incumbent` so the
promotion checkout is fixed outside tool arguments. The checked-in
[agent program](../../scripts/runtime-failure-capsule/AUTONOMOUS_OPTIMIZATION_PROGRAM.md)
Expand All @@ -881,6 +881,72 @@ finished campaign is a manual local cleanup choice. The engine installs nothing,
invokes no production or cloud endpoint,
and never commits, pushes, resets source, or manufactures a checkout.

### Candidate challenge and contribution closeout

A campaign `keep` proves only the declared local scopes. Commit the candidate
before paired promotion so the kept record names the future PR head; uncommitted
source at challenge time fails closed. Before publishing, challenge that exact
retained SHA:

```bash
pnpm --silent runtime:campaign -- challenge \
--repo /path/to/candidate \
--campaign .codevetter/optimization-campaigns/parser-loop \
--selected-sequence 4 \
--justification "The direct lookup adds no cache or fallback state." --json
```

The challenge binds the kept record, current commit, diff digest, complexity,
and deterministic risk observations. It requires either a directly comparable
qualified candidate or a bounded reason that a simpler comparison is not
applicable. Candidate comparison retains the largest scale or latency value as
the target and treats smaller scale points, bytes/op, and allocations/op as
controls when they exist. The simpler candidate must stay within the same 5%
tolerance on every retained metric. Risk tokens request evidence; they are not
a model score and cannot offset correctness or performance.

Inspect one published pull request with the returned challenge path:

```bash
pnpm --silent runtime:campaign -- inspect-contribution \
--repo /path/to/candidate \
--campaign .codevetter/optimization-campaigns/parser-loop \
--challenge .codevetter/optimization-campaigns/parser-loop/closeout/challenge-<sha>-<digest>.json \
--pr https://github.com/owner/repository/pull/123 \
--trex-policy optional --json
```

`trex-policy` is `optional`, `required`, or `not_applicable`. A supplied T-Rex
preview receipt is read from a contained local path and must identify the same
candidate SHA. CodeVetter does not launch T-Rex, a browser, preview, or hosted
load from this operation.

The contribution receipt keeps correctness, performance, patch quality, T-Rex,
head freshness, checks, review threads, and merge authority as independent
gates. It distinguishes failed checks from fork-workflow approval, retains
outdated and resolved feedback without treating either as current work, and
reports `waiting_for_maintainer` when upstream owns merge. Use
`refresh-contribution` for one explicit reread; there is no polling.

The append-only receipt also carries bounded maintainer-feedback learning when
a reviewed candidate is superseded: exact before/after SHAs and complexity,
the actionable thread, prior deterministic risk signals, the revised campaign
hypothesis, repeated local gates, and observed upstream disposition. This is
evidence from one contribution, not a universal style rule.

`closeout/publication.json` is a concise projection of the newest current
receipt. A different PR head marks an existing projection `stale` while keeping
its original receipt digest; only a newly challenged, current receipt can
regenerate it. The receipt ledger remains the authority.

The GitHub adapter uses one fixed read-only GraphQL query. It cannot comment,
resolve a thread, request review, approve, merge, deploy, install an app, create
a required check, or change pull-request metadata. Raw receipts remain local;
an author can copy a concise summary separately.

The challenge, inspection, and refresh operations are also available through
`runtime:mcp`; their repository is fixed when the server starts.

## 13. Qualifying a workload before profiling

Runtime detection proves that a repository contains a supported toolchain; it
Expand Down
35 changes: 21 additions & 14 deletions docs/knowledge/performance-lab-publication.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,23 +63,26 @@ disposition is retained in

[markedjs/marked#4048](https://github.com/markedjs/marked/pull/4048) was the
first external upstream PR created by this campaign. It optimizes reference-link
membership during the normal post-block inline phase while retaining the
dynamic direct-call path and clearing temporary state in `finally`.
membership by checking the existing link table directly instead of rebuilding
and linearly searching its keys for every queued inline source. The revised
patch contains no duplicate lexer state.

Five-sample paired verification measured:
Thirty alternating fresh-process pairs with per-process warmup measured:

| References | Baseline | Candidate | Change |
| References | Baseline | Candidate | Paired median change (95% interval) |
| ---: | ---: | ---: | ---: |
| 100 | 1.577 ms/op | 0.869 ms/op | -44.9% |
| 500 | 15.472 ms/op | 3.326 ms/op | -78.5% |
| 2,000 | 217.253 ms/op | 12.761 ms/op | -94.126% |

The endpoint exponent fell from 1.644 to 0.897. A representative Markdown
control moved +2.722%, below the 10% materiality threshold. Marked's complete
`npm test` passed: 190 unit tests, 1,779 specification tests, ESM/UMD/CJS and
type builds, lint, and generated-output checks. The PR is open and no longer a
draft as of this record. Its visible unit, specification, security, and Vercel
checks pass; upstream maintainers still own the merge decision.
| 100 | 0.805 ms/op | 0.489 ms/op | -39.2% (-40.3, -38.8) |
| 500 | 11.512 ms/op | 2.265 ms/op | -80.4% (-80.5, -80.2) |
| 2,000 | 164.115 ms/op | 10.095 ms/op | -93.8% (-93.9, -93.8) |

A 2,000-paragraph no-definition control moved +1.3% with a +0.3% to +2.3%
interval. Reintroducing `Object.keys` or using `for...in` restored that fast path
but re-enumerated the full link table and erased most of the target improvement,
so neither experiment was retained. Marked's complete `npm test` passed: 190
unit tests, 1,779 specification tests, ESM/UMD/CJS and type builds, lint, and
generated-output checks. The PR is open and no longer a draft. Two maintainer
threads on the earlier implementation are outdated. Snyk passes; Vercel's fork
preview requires upstream authorization and is not a code-test failure.

### qs

Expand Down Expand Up @@ -143,6 +146,10 @@ optimization laboratory:
10. **Self-profiling** found and fixed repeated source-offset scans in V8
function coverage. The representative 91-document replay improved 18.11%;
the 98.808% number belongs only to the adversarial regression fixture.
11. **Contribution closeout** now challenges retained patch complexity, binds
local and optional T-Rex evidence to the pull-request head, reads current
and outdated review threads, distinguishes failed checks from fork approval,
and stops at upstream ownership without posting or assigning maintainer work.

The most important tooling improvements came from false or incomplete early
results: startup-dominated tests now fail closed, Vitest names must identify one
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-08-10
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
## Context

The local runtime campaign already keeps immutable manifests, append-only
experiment records, exact correctness results, paired performance evidence,
repository identities, and deterministic promotion decisions. It deliberately
does not own source edits or GitHub publication. T-Rex independently emits a
canonical change-plus-preview receipt with an exact source head and
deterministic browser-flow verdict.

The missing boundary is between a campaign `keep` and an honest external
contribution closeout. See `proposal.md` and the optimization-contribution-
closeout specification.

## Goals / Non-Goals

**Goals:**

- Make patch quality a required, evidence-backed gate after promotion.
- Bind local optimization and optional T-Rex evidence to one candidate SHA.
- Normalize enough GitHub state to distinguish code failure, review work,
approval, and upstream ownership.
- Keep the full loop available through deterministic CLI/MCP JSON without a
desktop dependency.

**Non-Goals:**

- Generating or editing patches, reviewing arbitrary style, or scoring code
quality with a model.
- Running T-Rex, deploying a preview, or load-testing hosted systems.
- Posting or resolving GitHub feedback, approving workflows, or merging PRs.
- Adding a second persistence system or modifying existing T-Rex receipts.

## Decisions

### 1. Add a companion closeout artifact instead of changing campaign v1 records

The campaign ledger remains immutable and backward compatible. A new bounded
challenge artifact references one or two existing `keep` records by sequence,
record digest, repository revision, and diff digest. The contribution receipt
references the challenge and the selected record.

Changing the closed v1 campaign record was rejected because old campaign
artifacts would become unreadable or require a migration that adds no runtime
evidence. Treating line count as part of the existing performance decision was
rejected because complexity must not compensate for correctness or speed.

### 2. Use deterministic risk signals as prompts for evidence

The challenge computes files and line movement from recorded campaign data and
scans the selected Git diff for bounded language-neutral risk tokens such as
new cache/state identifiers, `finally`/`defer` cleanup, and fallback branches.
Signals are observations, not a quality score. A flagged candidate needs either
a qualified simpler comparison or a bounded invariant justification.

The comparison retains the largest scale or latency value as the target and
smaller scale points, bytes/op, and allocations/op as controls when the
promotion evidence contains them. A simpler candidate must stay inside the
same tolerance on every recorded value. This avoids accepting a fast target
that quietly regresses the no-op, small-input, or allocation surface.

An AST dependency was rejected for the first slice because it would increase
packaging and language scope. An LLM quality verdict was rejected because it
would make the publication gate nondeterministic.

### 3. Keep contribution inspection in a separate pure service

`contribution-contracts.mjs` owns strict schemas and deterministic verdict
derivation. `contribution.mjs` loads campaign/challenge/T-Rex artifacts,
invokes a narrow injected GitHub inspector, normalizes observations, and writes
one atomic receipt under the campaign directory. Tests inject fixtures; the
default adapter calls local `gh` with argument arrays and a fixed GraphQL query.

The service accepts one canonical PR URL and verifies that its repository and
head match the locally selected evidence. It does not use a general GitHub SDK
or add a production dependency.

### 4. Compose T-Rex by receipt, not by execution

The contribution input declares `optional`, `required`, or `not_applicable` and
may identify one contained T-Rex JSON receipt. The adapter reads only the fields
needed for source identity, verdict, preview identity, limitations, and receipt
identity. Unknown schema or malformed evidence becomes no-confidence.

This keeps T-Rex beside performance while preserving authority: performance
cannot turn a failed browser flow into a pass, and T-Rex cannot claim a speedup.
Launching T-Rex from the runtime MCP was rejected because it would expand a
closed local performance server into a browser/network execution authority.

### 5. Derive readiness lexicographically

```mermaid
stateDiagram-v2
[*] --> LocallyProven
LocallyProven --> QualityRequired: promotion keep
QualityRequired --> ContributionObserved: challenge qualified
ContributionObserved --> Stale: PR or T-Rex head differs
ContributionObserved --> ReviewAction: current maintainer thread
ContributionObserved --> ChecksPending: pending or approval required
ContributionObserved --> FlowBlocked: required T-Rex fails or is missing
ContributionObserved --> Ready: every configured gate passes
Ready --> WaitingForMaintainer: external merge authority
Ready --> Merged: current head merged
Stale --> ContributionObserved: new head locally reverified
ReviewAction --> ContributionObserved: revised head or resolved feedback
ChecksPending --> ContributionObserved: checks refresh
FlowBlocked --> ContributionObserved: current T-Rex receipt supplied
```

The receipt keeps all gates even when an earlier gate controls the overall
status. A weighted score was rejected because a large speedup must never offset
wrong behavior, stale identity, or actionable review.

### 6. Make refresh explicit for the local MVP

CLI and MCP expose challenge, inspect, and refresh operations. No daemon or
hosted webhook is added. Each refresh reads the current PR once and emits a new
current receipt while retaining earlier receipts as bounded history.

Background polling was rejected because it creates hidden resource and API
cost, conflicts with the local-first product boundary, and is unnecessary to
prove the contract.

### 7. Optimize contributor preparation, not maintainer ceremony

Candidate challenge runs before publication whenever possible. GitHub refresh
only consumes evidence that already exists, and a maintainer-owned terminal
state ends in quiet waiting. The adapter contains no mutation command and the
MCP annotations remain read-only for inspection and refresh.

Installing a GitHub App, creating a required status check, posting a large
benchmark report, automatically requesting re-review, or reminding maintainers
was rejected. Those actions externalize the contributor's verification cost
onto open-source projects. A future explicit author action may copy a concise
summary, but it is not part of this service.

### 8. Derive learning and publication from receipt transitions

When a reviewed candidate is superseded, the service reads the prior immutable
challenge and carries the actionable thread into a bounded learning record. It
records exact before/after revisions and complexity, prior risk signals, the
revised campaign hypothesis, repeated gate status, and current upstream
disposition. It does not infer a universal style rule or ask a maintainer to
classify the feedback.

`publication.json` is a replaceable projection, not evidence authority. A
current receipt regenerates it. Head drift only marks the existing projection
stale while preserving its source digest; no stale receipt can silently become
the new publication. The append-only receipt ledger and immutable challenges
remain authoritative.

## Risks / Trade-offs

- **Diff-token signals can over-report complexity** → treat them as evidence
requests, never automatic rejection; allow explicit invariant justification.
- **GitHub status models vary across check runs and status contexts** → preserve
raw bounded observed kinds and normalize only the small readiness taxonomy.
- **T-Rex is not applicable to every Node or Go project** → require an explicit
policy and keep `not_applicable` distinct from missing evidence.
- **A PR can change immediately after inspection** → bind the receipt to the
observed head and require refresh before terminal closeout.
- **Repeated refreshes can consume GitHub API quota** → remain operator-
triggered, make normalization idempotent, and perform no background polling.
- **Issue #111 is broad** → ship one hermetic Marked-shaped fixture and closed
operations first; UI and hosted monitoring remain out of scope.

## Migration Plan

The change is additive. Existing campaign and T-Rex artifacts remain readable
and unchanged. New challenge and contribution files live under the existing
contained campaign directory. Rollback removes the new CLI/MCP operations and
artifacts without touching campaign history, T-Rex storage, desktop data, or
external pull requests.
Loading
Loading