Skip to content

feat(bench): add strict budget collection and nightly evidence checks - #1160

Draft
zzet wants to merge 27 commits into
mainfrom
feat/performance-budgets
Draft

zzet wants to merge 27 commits into
mainfrom
feat/performance-budgets

Conversation

@zzet

@zzet zzet commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Adds strict v0.64.12 performance budgets, comparable evidence collection and staged PR/nightly/release plumbing. This remains a draft slice of #997; measured lane and release acceptance remain open.

The policy now declares 837 rows: 689 block rows and 148 disk/memory guard rows, with three separately named allowances. Its 38 PR targets use the issue's public ubuntu-24.04-arm profile, hosted_arm_smoke; every target, unit, comparison and repeat minimum is unchanged. Native four-CPU observations and the current GitHub-hosted Linux ARM job identity are retained truthfully. Reference Linux still requires validated eight-CPU/16-GiB/57–60-GiB capacity, and hosted smoke cannot satisfy nightly/release checks. Linux OOM safety requires both the authoritative cgroup event delta and log census; unavailable Mac authoritative OOM evidence remains an explicit prerequisite.

The Go checker retains medians of at least three pinned, comparable standalone #996 report-v1 repetitions, absolute block limits, named guard allowances, +20% smoke p95 comparisons and per-repeat hard/safety counters. Missing/incomparable observations, omitted repetitions and coupled paired-child capacity evidence fail. Collection preserves raw reports, failed attempts, logs, stop receipts, budget tables and wall time. The release entry forwards targeted SIGTERM/Interrupt to the directly launched collector and joins cleanup, including entry compilation in its cleanup reserve and wall receipt. Nightly history requires five adjacent scheduled main dates and exact raw-evidence replay; readiness/dispatch/duplicate-day/altered-table evidence cannot count.

The build-only perf-main.yml producer binds a clean daemon binary to its exact trusted main-push workflow/repository/revision/run/attempt, platform, pinned Go/C/C++ settings and SHA256. fetch-main refuses older-green fallback, forks/PRs, wrong workflows/platforms, expired or ambiguous artifacts, missing digests, binary tampering and unsafe ZIP entries. No artifact-host timing is used. The producer remains opt-in and disabled; the current main artifacts contain no usable daemon/benchmark bundle. A future consumer needs only job-scoped read access to Actions artifacts, not a write PAT.

Smoke collection requires that downloaded main binary and matching candidate build settings in the same PR run/attempt/job. Every raw report must fall inside its recorded task interval; verify-smoke checks retained provenance and replays the full policy. The existing strict limits remain authoritative.

The Go microbench runner executes exact-main/candidate test binaries for the existing three CI packages sequentially on the same host, with matching pinned fixture/CWD and explicit profile. It rotates order, retains at least three complete repeats and refuses missing/different benchmark-unit censuses or fixture changes before/after runs. It invokes only the existing benchstat pin, v0.0.0-20260409210113-8e83ce0f7b1c. Interrupt-ignoring benchmark processes receive a bounded failed kill/join; abnormal/context errors cannot become success. Raw samples and benchstat text are retained, and microbenchmark completion never claims formal workload acceptance.

perf-gate.yml publishes readiness explicitly named not acceptance. Owned reference/Mac jobs remain skipped unless separately reviewed host/config variables are enabled; no runner, VM, service or infrastructure is registered/adopted/provisioned. Existing affected/full test selection is unchanged. The load harness now has an explicit PR read-only offered-window/drain mode. With --duration 20s/30s --pr-smoke-drain, it stops new offers while joining admitted reads under their original product deadlines and a bounded drain context. Actual elapsed time, admissions and every busy/error/expiry/censor remain recorded; complete reports require consistent receipts. Edit/edit-and-probe, paired, reference/floor, saturation and other unsupported modes refuse the flag. The default remains unchanged. One optional workload config binds the new semantics; both schemas were generated with exactly that optional addition, preserving every historical value/required list and the legacy JSON/request hash.

Validation at c1e78dc66bcd904c7c2768800a52cea4705a250a:

  • Pinned Go 1.27.3 whole-package race tests passed: load harness 17.584s; checker 4.201s; release entry 2.352s. New controls cover successful admitted replies after offer end, no later admissions, product expiry, parent/overdrain censoring, unsupported-mode refusal, timing/admission receipts and frozen workload identity.
  • Both schemas came from the existing generators. Structural comparison confirms only the optional drain property is new, with every old value/required list unchanged. Legacy report golden/hash controls passed.
  • Full pinned golangci-lint 2.14.0 passed with zero issues. Changed gofmt -s, diff, CGO, production CI-workflow and both performance-workflow YAML/security guards passed. No new-head hosted CI result is claimed.

Still open:

  • The real five-step PR driver and measured 15-minute profile. An explicit 20/30-second PR-only window is the first experiment; docs quantify expected call counts and p99 limitations without claiming adequacy. The read offering/drain mode is implemented; edit/probe drain and exact probes still require implementation and integration with edit/readback instrumentation. Nightly/release profiles remain frozen.
  • Trusted publication/retrieval of the three main benchmark executables and the pinned fixture/tool setup. The current artifact format is daemon-only.
  • Exact readback/output-mode correctness, publication/reclaim/read-gate/cancellation and configured-root restart observations, scoped six-repo sweeps, editing-worktree and lifecycle/idle fixtures. Tag lacks new patch-write counters; equivalent baseline strategy needs review. Missing observations cannot become zeros or weaker proxies.
  • Reviewed nightly inventory/controls and effective 90-day retention. Commit an agent-profile load client and record v0.64.10 vs main baselines with cgroup, crash and CPU profiles #996 implementation and Count writes per edit (WAL, main file, rows, transactions) and add an edit-write benchmark #998 counters are merged; baseline/repeatability evidence is not assumed accepted.
  • Twenty unchanged smoke samples, both seeded PR failures, twenty-PR CI wall comparison, five consecutive main nightlies with seeded red, and a measured release subset under eight hours. Readiness starts no five-night calendar; it remains unstarted. Pre-enable CI metadata requires stratification and is not an accepted comparison baseline.

Refs #997. Keep the issue open.

zzet added 7 commits October 27, 2026 18:19
…formance-budgets

* commit 'c6c6de19da02fc698db7a89f9922a8ec49cc61b5': (205 commits)
  test(agents): refresh advertised-policy instruction goldens
  docs(profiles): preserve lean localization headroom
  docs(profiles): retain policy within existing byte budgets
  test(profiles): freeze supported legacy instruction bodies
  docs(mcp): tidy instruction refresh spacing
  test(guidance): pin reported full instruction profile
  test(guidance): cover managed inline refresh after upgrade
  docs(guidance): clarify missing compact aliases
  fix(guidance): follow advertised MCP tool policy
  perf(sqlite): cover proven-workspace dead-code probes
  test(sqlite): expose dead-code primary-row probes
  perf: seek eligible dead-code workspace candidates without rejected globals
  test: measure rejected-global candidate scan growth
  perf: exclude known ineligible dead-code candidates before usage reads
  test: expose dead-code hydration growth from rejected globals
  fix: preserve analysis for explicitly neutral index migrations
  test: retain active analysis across an index-only upgrade
  perf: demand interface evidence only for dead method receivers
  test: require demand-driven interface evidence for dead methods
  fix: register exact workspace planner and generation contracts
  ...
@zzet zzet changed the title feat(bench): add strict performance budgets and report checker feat(bench): add strict budget collection and nightly evidence checks Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant