C3: the v1 method benchmarked on a prepost campaign matrix by aclerc · Pull Request #143 · resgroup/wind-up

aclerc · 2026-09-15T13:14:16Z

C3: the v1 method, built and benchmarked on the prepost campaign matrix

This branch turns the method sketch into a working, documented and benchmarked prepost method
and the harness that judges it. docs/v1/method.md is the specification (steps 1 to 17 in four
parts) and the branch builds steps 1 to 15 at their defaults; the campaign runner and the prepost
campaign matrix both drive the same code. Read docs/v1/method.md first, then
docs/running-a-campaign.md, docs/analysis-period-and-reference-selection.md and findings
CF30 to CF34 in docs/v1/findings_campaigns.md.

Issue: C3 in docs/v1/issues_campaigns.md. The branch does not yet close C3's original
"done when" (see the status note below).

What changed

Method and documentation

docs/v1/method.md: the method, its defaults, what the code does for each edge case, the
validation targets i to v, the decisions taken with their evidence, the validity domain and the
open questions.
docs/analysis-period-and-reference-selection.md, docs/prepost-campaigns.md,
docs/running-a-campaign.md, docs/designing-a-campaign.md: declaring a campaign by its works
windows and running it. docs/estimating-uplift.md is retired.
Findings CF23 to CF34 in docs/v1/findings_campaigns.md.

Part A, data preparation (src/wind_up, run once per site)

Step 1: changes to the farm declared as works windows, with per-turbine change histories.
Step 2: operating states and data validity labels (operating_states.py), with the Hill of Towie
labeller (noise mode, BM curtailment, icing) and corrected Greenbyte fields for Kelmarsh and
Penmanshiel.
Step 3: reanalysis aligned once, with an offshore flag on the site.
Step 4: the whole record northed once; candidate yaw-alignment changes reported
(yaw_changes.py); golden north tables regenerated.
Step 5: waking direction, upwind turbines and waking scenarios per turbine (waking.py).
Step 6: the power model's features engineered once; ERA5 is out of the model and the sun's
elevation in (CF34).

Part B, the campaign for one test turbine

Step 7 and 8: analysis_period.py chooses each test turbine's before and after periods and ranks
the candidate references together, honouring works windows, exclusions and the wake reach of
every change (C10, campaign_design.py).
Step 9: the data screen and the measurement screen replace references the screen cannot judge
or rules out; a late change counts only after it.

Part C, measuring one turbine (benchmarking/baselines/power_model)

Step 11: record selection with the usable-hours floors and the waking-configuration rule; no
weather trimming (CF33).
Step 12 and 13: the LightGBM counterfactual, forward direction, corrected by the kept
references' mean reading (option C).
Step 14: a sampling sigma from a block bootstrap, a holdout sigma from the before period, the
references' spread, joined with rho 0.5.
Step 15: one uplift shape against predicted power, gated against the references' shape and cast
record by record to free-stream wind speed, direction, solar elevation and wake exposure.

The prepost campaign matrix (benchmarking/baselines/study_prepost_campaign_matrix.py,
prepost_matrix/)

Synthetic trial-then-rollout campaigns (campaigns/rollout.py) on Hill of Towie, Kelmarsh and
Penmanshiel, plus the two real campaigns (AeroUp T13, TuneUp), over seeds, K references, before
length P, after length L and season; sizes small, big, season, directions.
plan, download, run, merge, compare, validate, run-cell, plot-cell, list.
validate judges a study against method.md's targets on the validation domain with standard
errors clustered by campaign.
The matrix and the composed wind-up run share Part A and the cell executors; the real T13 cell
reproduces the composed run to 0.005 pp (CF30).

Removed

The finished probes shift_probe, disguise_probe and v0_probe.
The old study scripts and their baselines (study_overnight_*, study_power_model_compare,
study_toggle_methods_compare, study_toggle_specialist_uncertainty, inspect_*,
example_*_study), era5_derived, era5_sync, time_features, the conditional, matching and
waking modules of power_model, and the one-off outage_probe, screen_calibration and
screen_roadtest scripts.

Where it stands against the validation targets (run 9, `55386ca`, 4390 cells, 0 failures)

Run 9 is the big matrix with step 12's swap and the 9-month floor: 4363 cells ok, 27 with no test turbine
measured, 0 failed, about 18 h on 10 workers. The domain is now per reading, a training period of at least 9
months at 50 % coverage (3184 cells), so 3 months before with 9 or more after counts when the after period
trains.

Target (in the domain)	Run 9
i references read zero	before the screen +0.03 ± 0.05 pp, after +0.02 ± 0.05 pp: pass
i placebo test turbines read zero	-0.01 ± 0.05 pp: pass
ii measured on true uplift	slope 1.016 ± 0.010, intercept -0.01 ± 0.05 pp: pass
iii no dependence on P, L, K	pass (bias ranges 0.11, 0.23, 0.01 pp)
iii no dependence on season	FAIL: DJF +0.19 pp (z 2.5); the matrix confounds season with campaign, as in runs 7 and 8 (a season study cleared spring)
iv test sigma coverage	0.76 ± 0.03 (target 0.68; a sigma a little too large passes)
iv reference sigma coverage	FAIL, 0.44 (unchanged; the leave-one-out reference sigma is open)
v step 15 shapes beat the flat headline	predicted power 1.36 vs 1.54 pp, solar elevation and waked beat flat; direction and free-stream wind speed do not, through a few blown-up bins near cut-in (open, as in run 8)

Per period (test bias ± se, RMSE pp, coverage): P3 L9 -0.05 ± 0.11, 0.88, 0.66; P3 L12 -0.04 ± 0.10, 0.81,
0.67; P12 L12 +0.02 ± 0.07, 0.46, 0.82; P24 L12 +0.01 ± 0.07, 0.42, 0.71. P3 L6, outside the domain, reads
-0.29 ± 0.20 with RMSE 1.52.

Real campaigns at the defaults: AeroUp T13 +2.84 % ± 0.56 pp at 24 months before and 12 after (v0
published +3.1 %), +2.59 % at 12 and 12; with 3 months before and 9 after, where the after period trains,
+3.77 % ± 2.08 pp, the four references -0.6 to +1.4 %, no high-severity warning. TuneUp farm +1.14 %.

Step 12's swap, the validity floor and the analysis warnings register (2026-10-27)

PowerModelMethod.training_period ("auto", "before", "after"): the after period trains the model
when it spans at least 9 months, is longer than the before period and holds more usable hours, decided
once per test turbine on the candidate pool and shared with every reference reading; steps 13 to 15 then
run with the periods exchanged and every uplift is inverted to read as upgraded over baseline, sigmas
carried over. The uncertainty row records which period trained, its months, coverage and usable hours.
A training period under 9 months at 50 % coverage raises a high-severity AnalysisWarning
(wind_up.analysis_warnings) and still measures.
CampaignReport.warnings gathers the run's register, ranked: the methods' warnings, a reference the
screens ruled out (medium, step 9), an indicator of an undeclared change (medium, step 8), a candidate
yaw-alignment change (low, step 4). analysis_warnings.csv in the report, the summary in the log and at
the top of the run's README.
The validator's domain is now per reading: a training period of at least 9 months at 50 % coverage.
The rejected directions diagnostic (record_directions, DIRECTION_COLUMNS, symmetric_uplift, the
matrix override, the directions size and validator section vi) is removed.

Not in this PR

an accepted big baseline: run 9 (55386ca) validates the swap, but later commits on this branch move the
Hill of Towie synthetic cells (rollout campaigns now pass the site's excluded turbines), so the baseline
comes from the re-run after merge.

How to run it

Paths come from the environment or a git-ignored .env at the repo root (see .env.example):
WIND_UP_BENCHMARKING_DATA_DIR (each Zenodo record downloads into <dir>/<record id>/),
WIND_UP_BENCHMARKING_OUTPUT_DIR and WIND_UP_CACHE_DIR (the reanalysis cache). Install with uv sync.

A specific analysis, the real Hill of Towie AeroUp T13 campaign. Write the campaign folder from the open
data (SCADA parquet, turbines, works and the declaration), then run it through the composed wind-up method:

uv run python -m benchmarking.campaigns.real OUT_DIR/campaign            # or --campaign hot_tuneup
uv run python -m benchmarking.campaigns run OUT_DIR/campaign/campaign.yaml --out OUT_DIR/run

The run folder is laid out by the method's parts (its README.md opens with the analysis warnings):
A_data_preparation/ (steps 1 to 6), B_campaign/ (steps 7 to 9 and per_turbine.csv,
analysis_warnings.csv), C_turbine_measurement/<turbine>/ (steps 11 to 15 with plots) and D_farm/
(step 16). --reference-plots also writes each reference measurement's plots; run.log is the log.
To run a shorter campaign, edit campaign.yaml (a declared timing period, exclusions) or trim the
parquet; the planner's 3-month side floor can be lowered through run_declaration(plan_settings=...).
About 11 minutes for the default T13 run on a laptop, 2 minutes with 3 months before and 9 after.

The prepost campaign matrix. One study runs synthetic rollout campaigns on Hill of Towie, Kelmarsh and
Penmanshiel plus the two real campaigns over seeds, K references, before length P, after length L and
season, each cell through the same composed method:

uv run python -m benchmarking.baselines.study_prepost_campaign_matrix plan --size big       # cells, hours, memory
uv run python -m benchmarking.baselines.study_prepost_campaign_matrix download --size big   # open data + reanalysis
uv run python -m benchmarking.baselines.study_prepost_campaign_matrix run --size big        # resumable; --limit N for a trial
uv run python -m benchmarking.baselines.study_prepost_campaign_matrix validate STUDY_DIR    # method.md targets i to v
uv run python -m benchmarking.baselines.study_prepost_campaign_matrix compare STUDY_DIR [--accept-candidate]
uv run python -m benchmarking.baselines.study_prepost_campaign_matrix plot-cell STUDY_DIR CELL_ID OUT_DIR

Sizes: small (40 cells, a laptop smoke test, about half an hour on 3 workers), big (4390 cells, about
18 h on 10 workers; on a shared server never pass --workers above 10 and leave a third of its memory
free) and season. run writes <output dir>/prepost_matrix__<commit>__<size>/ with cells/*.json
and run.log, plus a __detail sibling; validate writes validation/validation.md and its CSVs;
compare judges the candidate against the committed study_prepost_campaign_matrix_baseline_small.json
and --accept-candidate promotes it. plot-cell reruns one cell with every plot on, laid out as a
composed run.

Status log

2026-10-27 evening: 55386ca pushed (swap, floor, warnings register); run 9 of the big matrix started on the
server (4390 cells, about 14 h); the committed small baseline is being regenerated.
2026-10-27 night: merged v1 (bddf371); docs brought up to the method as built and pr-144-review.md removed
(56234d6). The local small-baseline run was lost to a laptop out-of-memory crash.
2026-10-28: --workers above 10 refused; --input-plots became --data-preparation-dir (all of part A, the
README follows it); a blank works date is refused by turbine; rollout campaigns pass the site's excluded
turbines; a Greenbyte power reduction with a blank end is dropped; _reading keeps a zero sigma and the run
config records conditions, target_cp, shape_min_hours; fast suite green (2504 passed); the matrix no longer forces record_reversal (56d73ee, b529461). Run 9 at
3613/4390 cells.
2026-10-28 afternoon: the code hygiene list swept (007ebbc): time decay and dead fitting helpers retired,
at_limit keeps the wake terms true at a tighter limit, keyword-only arguments, plan_analysis's
allow_works_overlap, docstrings trimmed; the matrix's rollout draws are unchanged. Northing bands
item answered (the fixture moved them). Small baseline re-recorded at b529461 (40 ok) and committed.
2026-10-28 midday: run 9 done (4363 ok, 27 unmeasured, 0 failed) and validated on the new domain (table
above); cell costs and peak memory re-recorded from it (ba317f9).
2026-10-28 afternoon: the PR shrunk from 83,672 to 31,216 lines added (1e456a1): the matrix baseline JSON
holds one table row per line (the small baseline 53,176 -> 3,012 lines, same values), the finished
shift_probe, disguise_probe and v0_probe are removed (findings that cite them say so), and C3's
superseded 2026-10-07 plan leaves issues_campaigns.md.
2026-10-28 afternoon: Copilot's three comments fixed (2c6245c): benchmarking.env's data_root,
output_root and cache_dir load .env on first use, so no runner can miss it (a server run had
re-downloaded from Zenodo when one did); pvlib in the dev group; the synthetic generator limits the
first treated record by its start setpoint.

TODO before merge

Blocking

Docs that contradict the method as built

docs/v1/issues_campaigns.md C3 "Status and next steps" (lines 312-467) stops at 2026-10-07 with an
8-step plan the branch did not follow; rewrite as a status at the merge commit pointing at
method.md "Decided so far" and CF30-CF34, and state what C3's original "done when" still lacks.
Mark C10 (line 711) done.
docs/prepost-campaigns.md: still documents headline_estimator="reversal" as the prepost option
(the default is reference_corrected), the "known weak spots" of the 82c6398 small run, waking
booleans and excl_booleans, and a TODO at line 54.
docs/running-a-campaign.md: lines 156-159 ask for equal-length, same-months periods (method step 7
and CF33 say otherwise); line 288 says there is no uncertainty yet (step 14 ships a sigma); line 211's
three minutes per turbine predates the holdout fits; the output tree (214-230) disagrees with
docs/analysis-period-and-reference-selection.md 219-233.
docs/v1/README.md does not list method.md and still says uncertainty is deferred.
Decide whether docs/superpowers/specs/2026-10-07-step2-operating-state-design.md stays tracked (untracked in 55386ca)
(the convention is that specs stay local; its method content is in method.md step 2).
docs/v1/goals.md:87 names two commercial sites in a list of example reports (pre-existing on v1).

Code hygiene (can follow in a sweep)

After merge (already decided, in CLAUDE.md order)

a big run at the new commit validated on the new domain (P3 with L >= 9 included); then step 15's
blown-up bins, the reference sigma, the method.md TODOs, CF35/CF36, an accepted big baseline, the
ERA5-tail TODOs.

🤖 Generated with Claude Code

The reference screen took the worst-covered turbine in the pool as the whole campaign's data budget, so one turbine's outage switched screening off for every reference. It now applies screen_min_campaign_days candidate by candidate: a candidate with too little upgraded data is held out of the screen rather than standing it down for the rest, and the screen declines entirely only when no candidate holds enough. That exposed a second inconsistency. A screening estimate already runs with the held-out candidates demoted to their operating-state booleans, because _reference_input puts any candidate outside the pool it is handed among the wake contributors. The campaign's own estimate and its reference report did not -- they kept those candidates as full power references, so the screen ruled on a pool the campaign never used. All three now read the same pool through ScreenResult.power_free, and reference_stability.csv gains an `unjudged` column so a held-out reference stays distinguishable from a ruled-out one. A power-free reference also said only one thing: active power at or above 5% of rated. That conflates a turbine which is out of service with one which is merely becalmed, and the mix of those two states moves across a changeover, so the same boolean means different things on either side of it. It now says two things -- `waking` (is it making a wake, from active power) and `normal_operation` (was it able to run, from the availability counter). normal_operation_feature defaults on. It does not detect curtailment: a turbine held at zero while available reads as operating normally. Recorded benchmark baselines will move: any run with power-free references or wake contributors now carries an extra feature per such turbine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The shift probe slides one prepost placebo's changeover across a calendar year, so every arm is the same campaign meeting different weather, with nothing injected and a truth of 0. Each arm carries five measures of its own shift, and the driver fits the dose-response of reading against shift. It also reads the whole farm the way a campaign report does, one test turbine with every other turbine estimated against the rest, so a run returns twenty readings whose truth is 0 rather than one. The placebo declaration now takes explicit baseline and campaign lengths, and the composed method list can turn the reference screen and the per-reference uplift report off, so every arm of a study holds the same reference pool and pays for only the fits it scores. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The disguise probe cuts a window into alternating day blocks and reads it two ways: the toggle leg takes the record as it stands with the odd blocks toggled on, the prepost leg takes it with every timestamp remapped, SCADA and reanalysis together, so the even blocks become a contiguous pre period and the odd blocks a contiguous post period. Both legs therefore contrast the same two halves of the same rows, and over the default two-year window the halves hold the same number of records of every calendar month. Nothing is injected, so every turbine's truth is 0 and a prepost reading away from 0 belongs to the prepost path. Every step that reads the whole timeline at once is settled once on the undisguised record and handed to both legs: the reanalysis lag match, and northing discovery, whose changepoint search meets different steps when the days are interleaved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The campaign-proximity weighting tilts the baseline toward the changeover, which a prepost fit reads as uplift: on a record whose truth is 0 it moved the reading by +0.154 pp. It is now off by default; a caller can still set an adaptive or a fixed half-life. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CF19 prices the campaign-proximity weighting on a record that must read 0. CF20 ablates the reference power minimum and the reanalysis features and finds neither is the source of the prepost bias. CF21 balances the two periods month for month and two thirds of the bias goes with them, the large per-turbine offsets included. R8 keeps the failure mode the weighting was there for: a very long baseline should not be able to dominate the fit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Scratch scripts live beside the private v1 notes; ruff skips them through gitignore and mypy needs telling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ecision Design for gating the prepost reversal correction with a computed shrinkage lambda instead of a hand-fit fade curve: log-space bias subtraction, an MSE-optimal lambda from forward/reverse uncertainties with a James-Stein positive-part bias term, a low/high effort lever for how sigma is estimated, and an enabled-but-not-yet-P95 headline band. Force-added past the usual specs/ gitignore at the user's request; may be removed from tracking later. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

headline_estimator="reversal" on PowerModelMethod also fits the reverse direction (train the upgraded rows, predict the baseline rows) and reports sqrt((1+r_fwd)/(1+r_rev))-1, cancelling a shrinkage common to both directions. Default stays "forward"; toggle is unaffected. This is the UNGUARDED form: the full correction is applied at every post-period length. The synthetic sweep showed it helps at >=6 months and harms below (over-correcting on a data-starved reverse fit). The design that gates it with a computed, precision-weighted shrinkage lambda is specified in docs/superpowers/specs/2026-09-17-reversal-shrinkage-design.md (commit 505839d); this commit is the foundation that spec's build order extends. Do not use "reversal" on short campaigns until that lambda lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

north_scada now requires a layout. source_northed_scada and source_record take one and the probe passes the placebo campaign's own. The campaign findings from this branch were renumbered CF22-CF24 on rebase, so the time-decay reference in issues_campaigns.md moves with them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…machine sync) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A campaign may now be declared with a per-turbine works table and exclusions instead of one changeover and analysis period. wind-up then chooses each upgraded turbine's analysis span and its nearest-K power references (src/wind_up/analysis_period.py), runs each turbine over its own span, and writes the plans into the report. - CampaignSpec / SyntheticCampaign carry works windows and exclusions; analysis_period is optional, or per turbine - the loader reads a works table, exclusions and the Greenbyte schema - generate_dataset injects each turbine from its own changeover - power_model refills a screened pool from reserves, reads each reference against its own pool, and keeps waking booleans over a reference's exclusions (exclusion_channels) - the real Hill of Towie AeroUp T13 campaign is written from open data (benchmarking/campaigns/real.py) Flat declarations take the same path as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

benchmarking/campaigns/rollout.py draws realistic prepost campaigns on Hill of Towie, Penmanshiel and Kelmarsh: a trial from design_campaign (1 to max-1 test turbines), its works scheduled on England and Wales working days by one or two teams, and optionally a full rollout of every other turbine 6-9 months later. The AeroUp Cp shape is injected, scaled by a multiplier, from each worked turbine's own works end, and the data is cut a chosen number of months after the last trial works. Works become a Zenodo-shaped works table, so the period selector chooses each span as for a real campaign. - SyntheticCampaign.other_upgraded: turbines injected but not analysed - real.py: module-level default dates (fixes the B008 lint failure) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Bank holidays follow the Scottish rules: 1 and 2 January, Good Friday, the first and last Mondays of May, the first Monday of August, St Andrew's Day and Christmas, with weekend substitutes. Easter Monday and the last Monday of August are working days. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Crews may come from either side of the border, so a working day avoids the bank holidays of Scotland and of England and Wales: Easter Monday and the last Monday of August join the Scottish set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

benchmarking/baselines/study_prepost_campaign_matrix.py runs shipped wind-up over the real HoT T13 campaign and synthetic trial-then-rollout campaigns on Hill of Towie, Penmanshiel and Kelmarsh, one cell per (campaign, multiplier, K, post length), plus an exclusion-channel A/B arm at K=4. It is one resumable invocation on a spawn process pool: cells that are ok are skipped, failed and interrupted ones rerun, and a cell never raises. The cells are merged into a candidate baseline (test and farm bias/spread/score, the line fit across multipliers, reference mean/median/sd/worst_abs/n, real T13 rows) with a leak check, and compared with the committed baseline. - the matrix's period selector takes each cell's K and a 14-day shortest side, so every post length is analysed - north_scada can write the discovered table without its plots; the runner and estimate_campaign pass that through, and the runner takes plan settings - psutil joins the dev group, for each cell's peak RSS Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The prepost campaign matrix now fetches what it needs: Hill of Towie year zips as before, and Kelmarsh and Penmanshiel SCADA from their Zenodo records (ensure_greenbyte_data, size-checked, no network once complete). A new `download` command fetches everything up front; `run` does the same before its first cell. Locations come from the environment or a git-ignored .env at the repo root (benchmarking/env.py, template in .env.example): the Zenodo data directory, the output directory and the reanalysis cache. The shell wins over the file. - download_zenodo_data takes a per-record metadata filename, so two records can share a directory; zenodo_record_files returns a record's file list - python-dotenv joins the dev group Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Open data now lands in <data root>/zenodo/<record id>/ -- 20204946 for Hill of Towie, 5841834 for Kelmarsh, 5946808 for Penmanshiel -- so records never share a directory and their files cannot collide. Each directory keeps its own record metadata, so the per-record metadata filename is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

<data root>/<record id>/, with no zenodo/ level in between. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A leftover folder such as wind_up/plots/__pycache__ from before the src rename was found as a namespace package and failed the build with "package directory src/wind_up/plots does not exist". The wheel's contents are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Its data files are byte-identical to 20204946 (v2.0.0); v2.1.0 updates the README and adds a changelog, a data dictionary and a photo. The CI cache key follows the record id, so CI builds a fresh cache in the per-record layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ummary The full matrix needs about 530 CPU-hours, too long for 16 workers. A study now runs at a size: small, a laptop smoke test of every arm, or big, projected at about 11.5 h on 16 workers. A plan command projects a size's hours and memory from measured cell costs, and cells run most expensive first. Each study keeps its records and merged results in one directory to download, while the sources and per-cell working files stay in a __detail sibling. Failed cells are kept on resuming unless --retry-failed, and each size has its own committed baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The pool rule only ranked spans, and length ranked above the reference count, so when no span met the pool rule the selector took the longest span even with a single reference. A chosen span now needs PlanSettings.min_references (3 by default), and a turbine with no such span raises instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ce in Validation Steps 6, 11, 12, 13 and 14 now state what the method does; the measured evidence behind their defaults moves to Validation's record of what has been decided. Step 13 leads with option C, step 11 ii separates its two rules, the waking configuration open question carries run 7's evidence, and step 14's floor names the two terms it applies to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The HPC server is shared; its jobs keep to 10 workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…, cast to the other axes Method step 15 is rebuilt. The after period's records are binned by the test turbine's predicted power (11 bins of rated); each bin with 24 hours reads the forward ratio, and the measured bins are shifted by one constant to meet the headline. Each record carries its power bin's uplift, and the free-stream wind speed (the power references' mean wind speed, scaled on the before period so the turbine's electrical Cp at 40-60 % of rated meets 0.46), direction, solar elevation and wake exposure report the energy-weighted mean over their records. Within each direction sector, solar band and wake side the power shape is measured again and flagged where it differs. Each bin's sigma is a block bootstrap of the whole chain joined with the headline's level terms. The two-direction, ERA5-matched estimator, its ERA5 axis and its matching module are removed, so step 15 no longer needs reanalysis. The harness scores each axis's truth on the method's own per-record labels. Reference anemometer drift is reported, not corrected. The prepost matrix gains the step 15 targets (per-bin bias, RMSE against the flat headline, per-bin coverage), and method.md step 15 and Validation describe the new method and the probe that chose it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The slope and intercept of target ii were fitted per campaign turbine across before-period lengths (and, in a season study, season shifts), so a correlation across cells between the level of the true uplift and the error tilted the line: the season study read slope 1.012 (z 3) where every cell's own line averages 1.003. The line is now fitted per cell's data, so only the multiplier varies within a fit, in the validator and in the baseline tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ead to each bin's sigma Every measurement now carries its predicted-power bins' deviations from its own overall uplift. A test turbine's measured bin keeps its own uplift only where it differs from the kept references' mean deviation by more than two sigma; the other measured bins read the headline, and all measured bins move by one constant so the shape still combines to the headline. Each bin's sigma takes the references' spread of deviations in quadrature, cast to the other axes by predicted energy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…gate The level sigma moves every bin alike, so it says nothing about whether a bin's deviation differs from the references' shape. The gate now tests against the bin's sampling sigma and the references' spread alone; the level sigma stays in each bin's reported sigma. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ines' uplift distributions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…, and no change at rated Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…total, with a directions size The before-length axis gains 3 months and a cell whose before and after lengths together fall short of min_total_months (9) is left out; a real campaign's Pmax and Lmax count as its longest periods. The directions size runs each campaign at K4, the 5 % campaign and its placebo, over every period shape, with both real campaigns. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…d judge them per period shape PowerModelMethod.record_directions holds out the after period as the before period already is, reads each model on its own period, corrects the forward and reverse ratios by those readings, combines the two by precision and records it all as DIRECTION_COLUMNS without moving the headline; each reference reading carries them too, over at most 4 windows a side. The matrix records them, and the validator's vi_directions table and section score the headline, each corrected direction, their combination and the longer side per before and after length. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…he analysis warnings ranked Method step 12: PowerModelMethod.training_period (auto, before, after). Auto trains on the after period when it spans at least 9 months to within half a month, is longer than the before period and holds more usable hours, decided once per test turbine on the candidate pool before the screen and shared by every reference reading. With the after period training, steps 13 to 15 run with the periods exchanged and every uplift is inverted to read as upgraded over baseline; the sigmas carry over. The uncertainty row records which period trained, its months, coverage and usable hours. A training period under 9 months at 50 % coverage returns a high-severity AnalysisWarning (wind_up.analysis_warnings) and still measures. The campaign run gathers every warning ranked by severity: the methods' own, a reference the screens ruled out (medium, step 9), an indicator of an undeclared change (medium, step 8) and a candidate yaw-alignment change (low, step 4); the report writes analysis_warnings.csv, the log carries the summary and the run's README opens with it. The validator judges the targets per reading on a training period of 9 months at 50 % coverage. The recorded directions, their columns, symmetric_uplift, the matrix override, the directions size and validator section vi are removed. Three annotations fix mypy under Python 3.10. The step 2 design spec is untracked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

# Conflicts: # benchmarking/diagnostics/common.py # benchmarking/diagnostics/curves.py

…eview note The outputs table follows the run's four parts; the prepost page describes the reference correction, the training-period swap and the held-out training period in place of the retired reversal headline and the 82c6398 weak spots; C3 carries a current status above its superseded 2026-10-07 plan; C10 is marked done; docs/v1/README.md lists method.md first and no longer defers uncertainty; scripts since removed are named as such where a finding cites them; two commercial site names leave goals.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ation-dir, and fix PR 143's small defects A works row with a blank date is refused by turbine; rollout campaigns pass the site's excluded turbines, so part A no longer anchors on them; a Greenbyte power reduction with a blank end is dropped; _reading keeps a zero sigma; the run config records conditions, target_cp and shape_min_hours; the run README names part A's folder wherever it is placed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…at a tighter limit, and sweep PR 143's hygiene list Removed: adaptive_time_decay, time_decay_half_life_days and the sample-weight plumbing (off by default), purged_time_block_folds, make_propensity_model and _implied_shrinkage. campaign_design's at_limit recomputes the wake terms when the tighter limit changes a pool. analysis_period takes REFERENCES_PER_TEST_TURBINE from wake_reach, and plan_analysis's forced= is allow_works_overlap=. Cell is keyword-only and the flagged helpers take keywords. A data screen with no candidates gives an empty table; the apparent-Cp scan skips a bin with a zero mean before. Docstrings and comments that argued or cited evidence are trimmed, RUN4_USABLE_BEFORE is gone, uncertainty_summary is tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

40 cells, all ok, run on the server; the rollout draws are unchanged at 007ebbc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Means at 55386ca project the big size at about 18 h on 10 workers, as it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

The documented default installation cannot run campaigns because pvlib is missing, and unresolved issues also affect upgrade limits, .env loading, and swapped-training shape scores.

3 open issues

What changed in this PR

This PR develops the v1 prepost uplift method and its shared campaign benchmarking workflow. The reviewed changes support campaign preparation, synthetic truth, uncertainty estimates, diagnostics, and regression coverage.

Changes:

Add campaign context, condition-label scoring, setpoint-aware synthetic upgrades, and bootstrap uncertainty.
Organize outputs by method step, add environment configuration, and extend campaign CLI options.
Expand tests, refresh northing fixtures and documentation, and remove retired benchmark utilities.

File	Description
`tests/wind_up/test_yaw_changes.py`	Tests yaw-change candidates and apparent power-coefficient shifts.
`tests/wind_up/test_wake_reach.py`	Tests wake reach and retained wind share.
`tests/wind_up/test_northing_real_data.py`	Updates recorded northing degradation results.
`tests/wind_up/test_campaign_design_outputs.py`	Tests wake-loss tables and coverage maps.
`tests/wind_up/test_analysis_warnings.py`	Tests warning ranking and summaries.
`tests/wind_up/test_analysis_period_plots.py`	Tests analysis timeline rendering.
`tests/wind_up/hot_northing.py`	Clarifies fixture documentation.
`tests/test_waking_state.py`	Tests cache separation across turbine sets.
`tests/test_era5.py`	Tests grid-cell selection cache keys.
`tests/test_design_hill_of_towie.py`	Tests designs at three wake reaches.
`tests/test_data/hot/northing/northing_farm_inputs.parquet`	Refreshes the northing input fixture.
`tests/test_data/hot/northing/golden_northing_corrections_penmanshiel.yaml`	Refreshes Penmanshiel corrections.
`tests/test_data/hot/northing/golden_northing_corrections_kelmarsh.yaml`	Refreshes Kelmarsh corrections.
`tests/benchmarking/test_env.py`	Tests environment-file loading and precedence.
`tests/benchmarking/synthetic/test_upgrades.py`	Tests setpoint-aware upgrade limits.
`tests/benchmarking/synthetic/test_ground_truth.py`	Tests supplied condition labels and ERA5 bins.
`tests/benchmarking/synthetic/test_generator.py`	Tests per-turbine changeovers.
`tests/benchmarking/synthetic/test_cp_core.py`	Tests power-limit tapering and clipping.
`tests/benchmarking/harness/test_wind_rose.py`	Tests reanalysis wind roses.
`tests/benchmarking/harness/test_toggle.py`	Tests toggle-cycle inference.
`tests/benchmarking/harness/test_scoring.py`	Tests truth scoring with method labels.
`tests/benchmarking/harness/test_northing.py`	Tests validity filtering and direction alignment.
`tests/benchmarking/harness/test_context.py`	Tests reserve-reference constraints.
`tests/benchmarking/harness/test_conditions.py`	Tests new condition axes and bins.
`tests/benchmarking/diagnostics/test_yaw_changes.py`	Tests yaw-change report outputs.
`tests/benchmarking/diagnostics/test_diagnostics.py`	Updates plot and stage expectations.
`tests/benchmarking/campaigns/timeline_fixtures.py`	Adds staggered-campaign fixtures.
`tests/benchmarking/campaigns/test_screen_roadtest.py`	Removes retired road-test coverage.
`tests/benchmarking/campaigns/test_runner.py`	Tests planned truth windows and exclusions.
`tests/benchmarking/campaigns/test_runner_pin.py`	Refreshes pinned campaign results.
`tests/benchmarking/campaigns/test_reference_fixture.py`	Removes retired calibration checks.
`tests/benchmarking/campaigns/test_preparation.py`	Tests shared data preparation and outputs.
`tests/benchmarking/campaigns/test_placebo.py`	Tests configurable campaign lengths.
`tests/benchmarking/campaigns/test_methods.py`	Tests power-model configuration forwarding.
`tests/benchmarking/campaigns/test_main.py`	Tests new campaign CLI options.
`tests/benchmarking/campaigns/test_handover.py`	Updates handover artifact expectations.
`tests/benchmarking/baselines/test_toggle_specialist.py`	Uses relocated toggle helpers.
`tests/benchmarking/baselines/test_time_features.py`	Removes retired time-feature tests.
`tests/benchmarking/baselines/test_power_model_waking.py`	Removes retired waking-diagnostic tests.
`tests/benchmarking/baselines/test_power_model_row_dump.py`	Tests optional per-record dumps.
`tests/benchmarking/baselines/test_power_model_fitting.py`	Reformats fitting imports only.
`tests/benchmarking/baselines/test_power_model_conditional.py`	Removes retired decomposition tests.
`tests/benchmarking/baselines/test_inspect_prepost_hard_case.py`	Removes retired inspector tests.
`tests/benchmarking/baselines/test_era5_derived.py`	Removes retired ERA5-feature tests.
`tests/benchmarking/baselines/test_block_bootstrap.py`	Tests single-period energy-ratio uncertainty.
`src/wind_up/analysis_warnings.py`	Adds ranked analysis warnings.
`src/wind_up_v0/waking_state.py`	Includes turbine names in cache keys.
`src/wind_up_v0/era5.py`	Supports land, sea, and nearest-cell selection.
`pyproject.toml`	Updates dependencies and tooling configuration.
`examples/v1/design_hill_of_towie.py`	Compares wind-rose-weighted campaign designs.
`docs/v1/README.md`	Links the method and updates uncertainty status.
`docs/v1/goals.md`	Updates site references and dataset citation.
`docs/v1/findings.md`	Edits a historical output-path reference.
`docs/northing.md`	Updates northing selection documentation.
`benchmarking/synthetic/upgrades.py`	Applies record-level power limits.
`benchmarking/synthetic/schema.py`	Adds the power-setpoint field.
`benchmarking/synthetic/ground_truth.py`	Supports externally supplied condition values.
`benchmarking/synthetic/generator.py`	Supports staggered turbine changeovers.
`benchmarking/synthetic/cp_core.py`	Tapers and clips gains against power limits.
`benchmarking/harness/wind_rose.py`	Builds reanalysis wind roses.
`benchmarking/harness/toggle.py`	Centralizes pairing and cycle inference.
`benchmarking/harness/scoring.py`	Scores conditional truth using method labels.
`benchmarking/harness/method.py`	Extends output contracts with labels and warnings.
`benchmarking/harness/context.py`	Adds reserves, changed intervals, and works windows.
`benchmarking/harness/conditions.py`	Defines shape axes and label alignment.
`benchmarking/env.py`	Adds root `.env` loading.
`benchmarking/diagnostics/stages.py`	Defines method-step output folders.
`benchmarking/diagnostics/reanalysis.py`	Adds reanalysis plots and summary tables.
`benchmarking/diagnostics/northing.py`	Updates plot stages and reference selection.
`benchmarking/diagnostics/histograms.py`	Includes all finite values in histogram ranges.
`benchmarking/diagnostics/coverage.py`	Updates coverage stages and reference selection.
`benchmarking/diagnostics/context.py`	Adds explicit plotted power references.
`benchmarking/diagnostics/common.py`	Registers staged diagnostics with skip support.
`benchmarking/campaigns/sensor_fixture.py`	Updates the reference-wind-speed description.
`benchmarking/campaigns/placebo.py`	Allows baseline and campaign length overrides.
`benchmarking/campaigns/methods.py`	Passes geometry and analysis options to methods.
`benchmarking/campaigns/handover.py`	Adds wake declarations and period validation.
`benchmarking/campaigns/context.py`	Builds measurement context from analysis plans.
`benchmarking/campaigns/__main__.py`	Adds preparation-directory and reference-plot options.
`benchmarking/baselines/toggle_specialist.py`	Uses shared pairing helpers and new stages.
`benchmarking/baselines/study_overnight_toggle.py`	Removes the retired toggle study driver.
`benchmarking/baselines/study_overnight_prepost.py`	Removes the retired prepost study driver.
`benchmarking/baselines/study_northing_doc_plot.py`	Updates a documentation reference.
`benchmarking/baselines/study_analysis_period_doc_plot.py`	Adds analysis-period documentation figures.
`benchmarking/baselines/prepost_matrix/plan.py`	Estimates matrix runtime and memory needs.
`benchmarking/baselines/prepost_matrix/__init__.py`	Introduces the matrix package.
`benchmarking/baselines/power_model/screening.py`	Tracks candidates the screen could not judge.
`benchmarking/baselines/power_model/fitting.py`	Simplifies fitting documentation and comments.
`benchmarking/baselines/power_model/conditional.py`	Removes retired decomposition helpers.
`benchmarking/baselines/power_model/__init__.py`	Removes the obsolete ERA5 exclusion export.
`benchmarking/baselines/old/migrate_toggle_baseline_v2_to_v3.py`	Removes the obsolete migration script.
`benchmarking/baselines/old/__init__.py`	Removes the retired scripts package marker.
`benchmarking/baselines/naive_ratio.py`	Uses the new diagnostic stages.
`benchmarking/baselines/make_northing_farm_fixture.py`	Clarifies fixture-selection documentation.
`benchmarking/baselines/filtering.py`	Delegates stuck-record filtering to shared logic.
`benchmarking/baselines/block_bootstrap.py`	Adds single-period energy-ratio bootstrapping.
`.github/workflows/lint-and-test.yaml`	Updates the Hill of Towie dataset identifier.
`.env.example`	Documents configurable data, output, and cache paths.

_{Review effort: Balanced}

…d the superseded C3 plan The matrix baseline JSON is written with one table row per line (the small baseline goes from 53,176 lines to 3,012, same values). shift_probe, disguise_probe and v0_probe and their tests are removed and the findings that cite them say so. C3's 2026-10-07 plan leaves issues_campaigns.md and its status moves to run 9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… and limit the first treated record by its start setpoint benchmarking.env's data_root, output_root and cache_dir load the root .env on first use, so no runner has to call load_env: every output root, the Zenodo record directory and benchmarking's ERA5 fetches go through them. pvlib joins the dev group. The generator computes each test turbine's power limits on its whole record, so a treated record that starts a period takes its start setpoint from the record before (Copilot review on PR 143). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

aclerc changed the base branch from main to v1 September 15, 2026 13:14

aclerc and others added 9 commits September 25, 2026 13:25

Keep mypy out of the untracked extra docs

c8f5f9d

Scratch scripts live beside the private v1 notes; ruff skips them through gitignore and mypy needs telling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

aclerc force-pushed the v1-C3 branch from b7af0eb to 11c5522 Compare September 25, 2026 12:35

aclerc and others added 19 commits September 25, 2026 15:47

Spec: prepost campaign matrix (awaiting review; temporary, for cross-…

5f06d4f

…machine sync) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Update 2026-09-25-prepost-campaign-matrix-design.md

3427877

Update 2026-09-25-prepost-campaign-matrix-design.md

f8200d0

Untrack the reviewed specs; specs stay local

10d6971

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Document declaring a campaign by its works, and the plan files

d17e332

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Put each Zenodo record's directory directly in the data directory

57e4aeb

<data root>/<record id>/, with no zenodo/ level in between. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge branch 'v1' into v1-C3

6ebecd5

Call the HPC by a generic name in the power-model compare study

99462b7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

aclerc and others added 11 commits October 22, 2026 14:32

Run the big and season matrix sizes on 10 workers

d49be5a

The HPC server is shared; its jobs keep to 10 workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Note in method.md that the whole-site results should combine the turb…

bd72ddb

…ines' uplift distributions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add method.md TODOs: kW gain beside the combined uplift distributions…

78fa571

…, and no change at rated Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

aclerc changed the title ~~V1 c3~~ C3: the v1 method, built and benchmarked on the prepost campaign matrix Oct 27, 2026

aclerc and others added 7 commits October 27, 2026 17:59

Merge remote-tracking branch 'origin/v1' into v1-C3

bddf371

# Conflicts: # benchmarking/diagnostics/common.py # benchmarking/diagnostics/curves.py

Stop the prepost matrix recording the reverse direction on every cell

b529461

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Re-record the small prepost matrix baseline at b529461

e3f0df5

40 cells, all ok, run on the server; the rollout draws are unchanged at 007ebbc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Re-record the prepost matrix cell costs and peak memory from run 9

ba317f9

Means at 55386ca project the big size at about 18 h on 10 workers, as it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

aclerc marked this pull request as ready for review October 28, 2026 11:21

aclerc requested a balanced review from Copilot October 28, 2026 11:21

Copilot started reviewing on behalf of aclerc October 28, 2026 11:21

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

Comment thread pyproject.toml Resolved

Comment thread benchmarking/campaigns/__main__.py Resolved

Comment thread benchmarking/synthetic/upgrades.py Resolved

aclerc and others added 2 commits October 28, 2026 11:41

aclerc changed the title ~~C3: the v1 method, built and benchmarked on the prepost campaign matrix~~ C3: the v1 method benchmarked on a prepost campaign matrix Oct 28, 2026

aclerc merged commit b76ebcc into v1 Oct 28, 2026
4 checks passed

aclerc deleted the v1-C3 branch October 28, 2026 14:53

This was referenced Oct 28, 2026

Post-merge sweep of PR 143: finite sigmas, configuration errors, the season axis #154

Merged

Diagnostics: restore the kept/removed operating-curve figures for runs without operating-state labels #155

Merged

Navigation Menu

Repository navigation

C3: the v1 method benchmarked on a prepost campaign matrix - #143

C3: the v1 method benchmarked on a prepost campaign matrix#143
aclerc merged 129 commits into
v1from
v1-C3

aclerc commented Sep 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

aclerc commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

C3: the v1 method, built and benchmarked on the prepost campaign matrix

What changed

Where it stands against the validation targets (run 9, 55386ca, 4390 cells, 0 failures)

Not in this PR

How to run it

Status log

TODO before merge

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aclerc commented Sep 15, 2026 •

edited

Loading

Where it stands against the validation targets (run 9, `55386ca`, 4390 cells, 0 failures)