Watch wandb/core for UI label drift and report it in a rolling PR - #3078
Open
mdlinville wants to merge 19 commits into
Open
Watch wandb/core for UI label drift and report it in a rolling PR#3078mdlinville wants to merge 19 commits into
mdlinville wants to merge 19 commits into
Conversation
Detects user-facing label changes in wandb/core that may have left docs stale. This is step 1 of 9: the deterministic funnel and its regression tests. No ledger, docs index, model, CI, or secrets yet. Over 60 days of origin/master it reduces 2,990 commits to 170 candidates (~20/week) in 27 seconds, with no network or token. Three things worth knowing: - Case is never normalized. Lowercasing would collapse MODELS SEAT into Models Seat, silently cancelling a real drift finding that has been wrong in manage-organization.mdx for six weeks. - Commit type is never filtered on. Half that finding arrived under refactor(app):. - Label-carrying attribute names are matched by suffix, not enumerated. A component library invents saveLabel/isPendingAriaLabel as it grows. The six fixtures are frozen `git show` output from real commits. They caught three bugs that design review missed, so they are the regression surface, not decoration. ADAPTING.md records what is generic and what is wandb/core-specific, for pointing at another repo later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Joins stage-1 candidates against published docs: a label that left the product and still appears in a doc page is drift. Over 60 days this takes 170 candidates down to 12 with a real occurrence, ~1.5/week. The join is deterministic, so it belongs before any model pass, not after. `python3 scripts/uidrift_watch.py --dry-run --since "60 days ago" --docs` Matching requires UI-emphasis context (bold, backticks, quotes, or "the X button") plus a two-token-or-all-caps gate. Naive substring matching puts `search` on 215 pages and is unusable. Two subtleties worth flagging for review: - The literal is matched case-sensitively. Asking "does the OLD string still appear?" means `Models Seat` must NOT match where docs already say `Models Seat` — otherwise we report drift that is already fixed, on precisely the case-only renames this is best at finding. - Frontmatter is blanked rather than deleted, so reported page:line stays exact. Deleting it shifted citations by five lines. Docs absence still cannot suppress anything: this module exposes no function returning a score, so the "undocumented so ignore it" loop is unrepresentable rather than merely discouraged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ifying Substring matching let `Add panel` match inside `Add panels`, which inflated that literal from 2 pages to 16 -- past the too-generic cap, so a real finding would have been suppressed -- and misfiled every bold `**Add panels**` as unemphasized prose, because the bold matcher then failed on the trailing `s`. Now bounded by lookarounds, so literals that start or end with punctuation still match. Adds `match_confidence` (high/medium/low) and `replace_targets`, encoding the rule that text marked up as a control must match the UI exactly, while text in a run of prose is governed by the style guide and should be left alone. A page ending up mixed -- bold **Models Seat** in a step, lowercase "models seat" in a sentence -- is correct, not inconsistent. The corpus has more context shapes than are worth a taxonomy (SDK output, MDX component props, headings, CSV enum values), so those report at low confidence rather than getting their own rules. Backticked strings are capped at medium: as often an API value as a button. Confidence here grades match quality only. Docs absence still cannot lower anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The docstrings and test corpus illustrated prose with "a models seat", which is wrong: Models is a product surface and stays capitalized in prose regardless of what the UI does. These examples are the sort of thing an agent reads as a pattern, so leaving it risked teaching the error. Replaced with a contrast that has no capitalization trap: the **Add panel** button versus "add a panel to your workspace", which is a verb phrase that happens to share the words. Also notes in match_confidence that prose is not free-form -- it answers to the style guide rather than to the UI, which is a different authority, not an absent one. The frozen fixtures are untouched. f4861ad genuinely contains "A models seat gives you write access", but that is real captured wandb/core output and rewriting it would break the diffs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tep 3) Turns the flat delta list into typed changes. Two findings worth review, both caught by fixtures rather than by design: Pairing has to be positional before it is similarity-based. The obvious approach — match a removed string to the added string it most resembles — fails on the case that matters most. "Hide manually hidden runs" -> "List only visible runs" scores 0.55, and "Only show visualized" -> "Hide manually hidden runs" scores 0.22. No threshold catches those and still refuses to pair two unrelated column headers. Git already answers it: an in-place edit is a `-` and the `+` that replaced it at the same offset in one change block. Similarity remains as a second pass, for renames that moved between fields — `header: 'WEAVE ACCESS'` became `name: 'Weave Access'`, which grouping by field name would miss. Not every conditional is a gate. `if (hideManuallyHidden)` is UI state, and reporting it as gating would mark half the app "not yet visible". A conditional only counts when its variable resolves to a gate hook; with no hook, no claim is made. Flag lifecycle reads adds and removes from the RampKey union. Static presence is still deliberately ignored — gates left at 100% forever carry no information. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is the v1 deliverable: a table, no Jira filing, with each row routed to agent / pair / human. Over 60 days of origin/master it produces 42 findings (7 agent, 12 pair, 23 human) from 170 stage-1 candidates. The first render was unusable and that is worth recording. It emitted 22 rows for three commits, of which 2 were real; the rest were every changed string that matched no doc page, filed as a coverage gap — `new Loading members`, `new Invited`, `PROFILE removed`. Drift requires docs to drift from. Renaming a label no page mentions makes nothing incorrect, so it is a statistic, not a row. Those are now counted in aggregate. That is not the suppression the one-directional rule forbids: docs absence still never hides a finding that exists, it just stops manufacturing findings that do not. Two related fixes came out of the same pass. New copy is aggregated per surface, because a settings panel that adds a heading, a description, two labels and a button is one docs task. And findings are keyed on the docs task rather than the code surface — three member tables render the same column that the docs name once, so keying on surface showed one edit as three rows. Triage is deliberately asymmetric: easy to fall out of the agent lane, hard to fall in. A wrong agent call puts a false statement in published docs unattended; a wrong pair call costs a writer fifteen minutes. Ownership resolves CODEOWNERS last-match-wins, which wandb/core relies on — line 284 names rampFeatureFlags.ts explicitly to override the *ramp** rule 20 lines in. Reviewer ranking widens from 6 months to all-time when the recent window is too thin to rank. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reviewers and owning team look like per-finding lookups and are not. Both answers are identical for every row in a scan, so both are now computed once and cached for the life of the process. CODEOWNERS was being fetched with `git show` and reparsed for every single finding; it is one file that cannot change mid-run. Authorship was one or two `git log` calls per path; it is now a single `git log --name-only` over the UI roots, parsed into a path -> author-counts index. Three paths cost 3 subprocesses instead of 9, and the ~130 calls a 42-finding scan made are the reason the live run was done with resolve_owners=False. Owner columns can populate now. Cached per run and deliberately not across runs. Team membership changes, and a stale owner cache is a wrong @-mention in a PR nobody can explain. This module had no direct test coverage — test_triage passes resolve_owners=False throughout — so it now has tests against a throwaway git repo rather than mocks. A mock of `git log --name-only` output would only restate the parser's own assumptions about that format, which is the thing most likely to be wrong. Included is a regression test asserting the subprocess count stays bounded, since that is the whole point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…p 5) The obvious design is a store that remembers every finding ever emitted. This stores only human decisions. For identity, dedupe, settledness, triage, ownership and docs coverage, a fresh scan recomputes the same answer from the commit history and the docs tree, so persisting them would only create a second copy that can disagree with the first — and the second copy is the one nobody notices is wrong. What cannot be recomputed is a human having said "I looked at this and it is fine." Narrowing the ledger to that turned out to matter, because recomputing dedupe is where two real bugs were hiding. `settled` was read from whichever commit the loop was on. A label renamed 60 days ago and renamed again yesterday reported as settled — eligible for the agent lane while still moving. It now comes from the last change, which is what the word was always supposed to mean, and is what finally resolves the two-commit ORG ROLE cluster. Content-keyed ids make A->B and B->C different findings, and only the first has docs evidence, because docs still say A. A scan spanning both commits would therefore have told a writer to publish B — a label the product had already stopped using. Chains now collapse to A->C. Forks and joins refuse to chain and drop to the pair lane rather than guess which target is current, and A->B->A is counted as a revert instead of reported, since docs were never wrong. Suppression gets the same one-directional discipline as the docs oracle, and the failure mode is subtler: dismiss a finding as a false positive, and six weeks later a page starts documenting that surface. The finding is now real and the stored dismissal would hide it silently — a suppression that gets more wrong over time. So a decision records the docs evidence it was made against, and expanding evidence reopens it. Only expansion; a page that stops mentioning the string means somebody did the work. Verified across four runs against a real file: found, dismissed, stays suppressed, reopens when a second page documents it, then self-extinguishes once docs are fixed — leaving an orphan flagged prunable but never deleted, because an orphan is ambiguous between "resolved" and "outside the window". Two problems only the rendered output showed, which is now twice this has happened. The empty-state prose claimed every change was reflected in docs or touched no documented surface, while a suppressed finding sat below it; suppression is now accounted for in the funnel line and the prose. And the report bucketed by lane while iterating only the three known lanes, so a finding with an unexpected lane vanished from the table while still being counted in the header — the one failure a reader cannot detect. Rename chains are covered by constructed cases, not fixtures. The closest pair in the corpus (c99e959 then ccd66e2) turns out to add a gated toggle and then rename it, which emits one rename, not two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Until now nothing called the pipeline; the 60-day numbers in step 4 came from an ad-hoc script. This wires commit iteration through extraction, docs lookup, merge, the ledger and the renderer, behind a CLI. Live against wandb/core at 71fa9d10, 60 days: 2995 commits -> 593 touching UI -> 170 stage-1 candidates (1180 changed strings) -> 42 findings. The finding count matches step 4 exactly, with 51 raw rows collapsing to 42 through the cross-commit merge. Runtime is 1m36s WITH reviewer and team resolution, against roughly two minutes without it before, so the owner columns populate now instead of rendering as em dashes. Incremental mode takes its base from the head SHA in the newest report filename rather than a state file. That is lesson 18 applied to the watermark: the reports are the record, so there is nothing to disagree with them, and a deleted report degrades to "scan further back" rather than to a silently wrong range. 1.3 seconds against 96, which is what makes a frequent cron affordable. The scan never writes the ledger. Only `decide` does, and it re-derives the finding by scanning rather than reading it out of a report, because the evidence fingerprint has to reflect the corpus as it is now — a decision stamped with stale evidence would never reopen. It refuses to overwrite an existing decision without --force, since that is someone else's call. `docsindex.find` is now memoized on the index for the same reason ownership is cached per run: the corpus does not change mid-run, and the repeats are structural rather than incidental. `build_findings` probes each literal twice by construction — once to decide whether it is documented, once to attach the evidence — and one label routinely changes across several commits in a window. Two counts are reported where there was one. The established funnel counts commits with candidate strings (170); the number that actually calibrates the extractor is changed strings (1180). Reporting only the second would have looked like a 7x regression against the figures the working group has already seen. Exit code 3 when a decision reopens, separate from 0 and from operator error, because a stored decision that no longer matches its evidence is the one outcome that should be able to fail a CI step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tep 7a) Two changes the workflow in step 7 needs, both of which stand on their own. `scan --summary-json PATH` writes the run's counts -- funnel totals, the per-lane breakdown, the scanned range, and the report's path relative to the docs root. A caller that has to decide whether to open a PR needs facts. The alternative is grepping the rendered report for "No drift to act on", which couples a workflow to a sentence that exists to be read by a person, not parsed by a shell. The report path is emitted only when a report is actually written, so --stdout and --summary-json together describe a run that wrote no file rather than naming one that does not exist. stats is copied before it is annotated: the caller's Result keeps the scan's numbers unmodified. And the summary is written before the reopened-decision exit, so a run that ends in exit 3 still leaves the counts behind -- that is exactly the run whose numbers a human wants. Separately, uidrift joins DOCS.exclude_dirs. A report quotes the labels it reports on, so indexing our own output would make every finding look already documented -- a feedback loop that quietly empties the report. Reports are .md and only .mdx is indexed today, which makes this insurance rather than a fix for a live bug. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Steps 1-6 built a detector that prints a report. This is the part that
puts the report in front of a writer without anyone remembering to run
it: a weekday cron clones wandb/core, runs the scan with --incremental,
and opens or updates a single draft PR on uidrift/drift-report.
The clone is the whole engineering problem, and shallow is a trap. Four
options were measured against wandb/core (2.4 GB, 50k commits) and the
table is in ADAPTING.md so nobody optimizes this back:
* A 7-month shallow clone is 136 MB and 10 seconds, and returns
different reviewers than complete history does. Ownership falls back
to all-time authorship exactly when a file has thin recent history,
which is the history a truncated clone does not have. 18 months was
still wrong, differently wrong.
* A blobless clone looks perfect -- 96 MB, complete commit history --
until iter_commits reads --numstat, which needs blob content. A
one-day window spent 82 seconds lazy-fetching and then died on the
promisor remote.
So: full history, --single-branch, --no-checkout. 1.3 GB, ~3.5 minutes,
and correct.
Reading wandb/core needs a credential this repo did not have. The
workflow prefers a wandb-docs-source-reader App token and falls back to
a WANDB_CORE_TOKEN secret, so installing the App later requires no edit
here. Neither present means the first step fails in seconds naming both
options, rather than burning four minutes on a clone that cannot
authenticate.
No [skip ci] on the report commit, which contradicts the anatomy table in
the workflows README -- so that row is updated here rather than left
disagreeing with the shipped workflow. A skipped workflow reports no
status at all, so required pull_request checks would sit pending forever
and the PR could never merge. Nothing expensive runs anyway: Validate MDX
sees no .mdx or .json in the diff and short-circuits. .mintignore gains
uidrift/ for the same reason the docs index excludes it -- reports are a
record for the docs team, not pages.
The run fails, after the PR exists, if any stored decision reopened. That
means a writer's earlier dismissal no longer matches the docs, which only
a human can settle.
Not yet done: no JIRA ticket is filed, which the anatomy table names as
part of a sink.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
HiveMind Sessions7 sessions · 3h 20m · $71
View all sessions in HiveMind → Run |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
📚 Mintlify Preview Links✨ Added (1 total)⚙️ Other (1)
📝 Changed (1 total)⚙️ Other (1)
🤖 Generated automatically when Mintlify deployment succeeds |
Seven findings, all confirmed against the code rather than taken on trust. Six are dead symbols and are simply removed: `field` in config, `Iterable` in report, `dataclasses` and `docsindex` in test_triage, a `index = self.index` in test_docsindex that is overwritten two lines later by the rebuild the test actually asserts on, and the mixed import style in test_ownership. The seventh, `_GATE_CALL` in structure.py, was worth measuring before deleting, because a dead regex can mean a missing capability rather than leftovers. It means both. Gate detection only handles the assigned form (`const x = useFooGate(...)`); a hook called inline is invisible. Run against core's UI tree, `_GATE_ASSIGN` matches 261 sites across 83 hooks and an inline matcher would add roughly 110 more -- but 19 of those are `useGatedValue`, an unrelated Weave utility that any `use*Gate*` pattern matches too. So the regex as written could not have been switched on as it stood. It is removed, and the comment in its place records the measurement and what widening would require: a name filter and its own tests. That is a follow-up, not a one-line change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
All seven CodeQL quality findings are addressed in aa19110, each verified against the code before acting rather than applied on trust. No false positives — all seven were genuinely unused. Six were dead symbols and are removed as suggested. The seventh, 222 tests still pass. I also ran an AST sweep for unused imports and dead private module globals across One dissent, noted in-thread and complied with anyway: |
`uidrift_watch.py` was the step-1 CLI: walk a range, run the stage-1 funnel, print what survives, and nothing else. Its whole purpose was to make the funnel's reduction ratio reviewable before anything depended on it, and that job is done. `python3 -m uidrift.scan` replaced it at step 6 with the same walk plus the docs join, the ledger, and the report. Nothing references this file -- the only mentions of it anywhere are the two example invocations in its own docstring -- and shipping two entrypoints invites the reasonable question of which one is real. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a deterministic UI-label drift detector for wandb/core, including reporting, decision persistence, ownership resolution, tests, and scheduled rolling draft PR automation.
Changes:
- Implements extraction, classification, documentation lookup, triage, and reporting.
- Adds decision-ledger and ownership workflows.
- Adds extensive fixtures/tests and scheduled GitHub Actions automation.
Reviewed changes
Copilot reviewed 31 out of 34 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
.mintignore |
Excludes generated drift records. |
.github/workflows/uidrift-scan.yml |
Runs scans and maintains the rolling PR. |
.github/workflows/README.md |
Documents workflow operation. |
scripts/uidrift_watch.py |
Adds legacy dry-run entrypoint. |
scripts/uidrift/__init__.py |
Defines the package. |
scripts/uidrift/ADAPTING.md |
Documents architecture and adaptation. |
scripts/uidrift/build.py |
Assembles findings. |
scripts/uidrift/config.py |
Defines repository settings and thresholds. |
scripts/uidrift/docsindex.py |
Indexes documentation references. |
scripts/uidrift/extract.py |
Extracts label changes from diffs. |
scripts/uidrift/finding.py |
Defines findings and triage. |
scripts/uidrift/ledger.py |
Persists human decisions. |
scripts/uidrift/ownership.py |
Resolves reviewers and CODEOWNERS. |
scripts/uidrift/report.py |
Renders Markdown reports. |
scripts/uidrift/scan.py |
Implements CLI and scan orchestration. |
scripts/uidrift/structure.py |
Detects renames, gates, and structural signals. |
scripts/uidrift/_vendor/__init__.py |
Defines vendored package. |
scripts/uidrift/_vendor/commit_text.py |
Vendors commit-message utilities. |
scripts/uidrift/_vendor/diff_signals.py |
Vendors diff parsing utilities. |
scripts/uidrift/_vendor/gitsource.py |
Vendors Git history access. |
scripts/uidrift/tests/__init__.py |
Defines test package. |
scripts/uidrift/tests/test_docsindex.py |
Tests documentation indexing. |
scripts/uidrift/tests/test_extract.py |
Tests label extraction. |
scripts/uidrift/tests/test_ledger.py |
Tests decisions and merging. |
scripts/uidrift/tests/test_ownership.py |
Tests ownership resolution. |
scripts/uidrift/tests/test_scan.py |
Tests CLI and watermark behavior. |
scripts/uidrift/tests/test_structure.py |
Tests structural analysis. |
scripts/uidrift/tests/test_triage.py |
Tests finding assembly and reporting. |
scripts/uidrift/tests/fixtures/9573f30.diff |
Provides move-detection fixture. |
scripts/uidrift/tests/fixtures/c99e959.diff |
Provides rename fixture. |
scripts/uidrift/tests/fixtures/cb100df.diff |
Provides gated-addition fixture. |
scripts/uidrift/tests/fixtures/ccd66e2.diff |
Provides rewording fixture. |
scripts/uidrift/tests/fixtures/e1bc1e6.diff |
Provides gated-setting fixture. |
scripts/uidrift/tests/fixtures/f4861ad.diff |
Provides case-only rename fixture. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Five of eleven review findings, each verified against the code first. The other six change what the detector reports and are answered in the review threads with evidence rather than patched here; they deserve a before/after run against core, which this PR cannot give them. **Nothing ran the tests.** uidrift-scan.yml triggers only on schedule and dispatch, so 224 tests existed and no pull request executed one. A regression could merge and first appear days later inside a rolling report, where a writer would read it as drift rather than as a broken detector. uidrift-tests.yml now runs the suite on any PR touching scripts/uidrift. It needs no network and no core checkout. **A safety floor that was not one.** MIN_AGENT_CONFIDENCE = 0.6 claimed that no finding below it could reach the agent lane. Nothing read it, and Finding.confidence is never assigned, so it was 0.0 everywhere -- had the check existed, nothing would have qualified at all. Agent eligibility is decided by triage() from structural facts. The constant is gone and the comment in its place says where a real confidence value would have to be wired in. **CODEOWNERS `**/` is zero-or-more.** Translating it to `.*` plus a literal slash required at least one directory, so `/src/**/*ramp*` missed `src/ramp.tsx`. CODEOWNERS is last-match-wins, so a missed pattern is a missed override: the report names the wrong team, not no team. Two regression tests cover zero and many directories. **config.SOURCE was not actually the single source of the repo name.** report.py hard-coded the commit URL and the report title, so retargeting the detector as ADAPTING.md describes would have produced reports linking to wandb/core regardless. Both derive from config now, and scan passes owner_repo explicitly rather than relying on the vendored default. **A gaps-only run hid its own explanation.** The empty-findings path returned before the undocumented-surfaces section, so a run where every changed label was undocumented printed "No drift to act on" and withheld the count that explains why. Both paths share _gaps_section now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were reported as plausible, and both reproduce. Each one sends a
finding to the wrong lane, and in both cases the wrong direction is
toward `agent` -- the lane meaning "safe to apply unattended."
**Emphasis was decided per line, not per occurrence.** Occurrences were
recorded once per line and took the context of the first matcher that hit
anywhere on it. In MDX a paragraph is normally a single line, so
Click **Add panel** to begin. Add panel opens the chart picker.
produced one `bold` occurrence instead of one `bold` and one `prose`, and
`all_occurrences_emphasized` came back true -- the predicate that says a
blind find-and-replace is safe, answered on evidence it never saw. Each
match is now classified by the emphasis enclosing its own span. A deictic
second mention ("The Add panel button") still counts as emphasis, which
is the case that keeps the fix from over-correcting.
**A closed block was still treated as an ancestor.** The backward scan
skipped every line indented at least as deep as the changed line, so it
walked past the `)}` that ends a sibling block and adopted that block's
conditional. Prettier makes this ordinary: a JSX attribute sits one level
deeper than its own element, so a changed label is routinely deeper than
the sibling gate above it. The scan now lowers a ceiling each time it
passes a closing delimiter. Verified on four shapes -- JSX and brace
forms, sibling and enclosing -- including that an outer gate is still
found across a nested sibling block.
231 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Copilot review addressed. Eleven findings, every one verified against the code before acting — and unlike the CodeQL round, none were false positives. Two I could only confirm by building a repro that reproduced the failure. Fixed (7)
Findings 4 and 5 both misroute toward Confirmed, deferred, answered in thread (4) — Two need a call before merge, not after. 231 tests pass. |
All three touch a format or a semantic that gets expensive to change once data exists. `uidrift/` is still empty -- no report written, no decision stored -- so this is the last moment any of them is free. **Settledness now measures time on master.** `_vendor/gitsource` reads `%aI`, the author date, which rebase and cherry-pick preserve; a commit authored in March and landed today would arrive already older than SETTLED_DAYS and skip the churn protection entirely. `scan` now annotates each commit with its committer date and `build._landed_date` prefers it, falling back to the author date. Deliberately not fixed inside `_vendor/`: that directory is a faithful copy of another repo's module, and a local edit there turns every future re-vendor into a merge. The key written is `commit.committer.date`, which is what the real GitHub API already calls this, so if the vendored reader ever supplies it nothing downstream changes. Cost is one extra `git log` with no `--numstat`. Worth recording honestly: this is hardening, not a live bug fix. Of 49,501 non-merge commits on core's master, 32 (0.06%) landed seven or more days after they were authored, and none in the last six months -- squash merges rewrite both dates. The protection matters if that merge strategy ever changes, and costs nothing while it does not. **Report names carry a UTC timestamp.** `_last_report` picked the newest with max() over (date, sha, path), so two reports merged on one date were ordered by head SHA -- content-addressed, so effectively a coin flip. Half the time that picks the older one, and a watermark that moves backwards re-reports drift a writer already dismissed. Names are now `YYYY-MM-DDTHHMMSS-<sha>.md` and ordering never consults the SHA. Untimed names still parse and sort first within their day, so the failure direction is a rescan rather than a skip. **Ledger evidence counts occurrences per page.** `targets` was a set of page names, which cannot tell one editable occurrence on a page from three -- so a dismissed finding that grew a second occurrence on a page already in the set stayed suppressed while the docs drifted further. Now counts per page: still no line numbers, so unrelated docs edits still do not reopen anything. `_as_counts` reads the older list shape as one per page, so upgrading does not reopen every stored decision at once. 240 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nto code_monitoring_investigation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
scripts/uidrift, a detector that watcheswandb/corefor user-facing label changes that leave our docs stale, plus the weekday workflow that runs it and carries the result in one rolling draft PR.It is deliberately boring: stdlib only, no network, no model, no new dependency. Everything it reports is derived from a commit range and our own docs tree.
Reviewing 11.8k lines without reading 11.8k lines
The commits are a ladder — each one is independently reviewable and the messages carry the reasoning. If you read nothing else:
scripts/uidrift/ADAPTING.md— what it looks for, what it deliberately ignores, and how to point it at a different repo. Start here; it's the design doc..github/workflows/uidrift-scan.yml— the only thing that runs on a schedule, so the only thing that can surprise you.scripts/uidrift/config.py— every tunable in one place. If the detector is ever wrong in a boring way, the fix is usually here.Roughly half the diff is tests and fixtures. 222 tests, no network, no fixtures larger than a real diff.
The one decision worth arguing about
The
wandb/coreclone is full history,--single-branch,--no-checkout— 1.3 GB and ~3.5 minutes. That looks like an obvious thing to optimize, and it isn't:--shallow-since=7 months--shallow-since=18 months--filter=blob:noneiter_commitsreads--numstat, which needs blob content; a one-day window spent 82s lazy-fetching, then failed on the promisor remote.--no-checkoutThe table is in ADAPTING.md too, so the next person to look at this finds the reasoning before the stopwatch.
Needs a human before this can prove itself
wandb-docs-source-readeris installed ondocs-code-evalandweave-internal, not oncore. The workflow prefers an App token and falls back to aWANDB_CORE_TOKENsecret — that secret now exists, so the fallback path should work, and installing the App oncorelater needs no edit here.That is untested.
workflow_dispatchonly appears once the workflow is on the default branch, so the first real run happens after merge, not before. Everything else — mode selection, the PR body's empty-lane and reopened-decision branches, both credential paths — was simulated locally against a realwandb/coreclone.Merging is low-risk regardless: the workflow triggers only on
scheduleandworkflow_dispatch, so it cannot run on a PR, and a failed run opens nothing.How it behaves once it runs
--incremental, taking its base from the head SHA in the newest report filename, so a quiet day scans a few commits and opens no PR.scan decide <id> --status dismissed --agreement false_positive. That's the detector's only feedback channel. A dismissal reopens by itself if docs later start covering that surface.Known gaps, called out so they aren't findings
settings_panel.tsx→ "Settings panel".scripts/uidrift_watch.pyis orphaned. It was the step-1 entrypoint;python3 -m uidrift.scansuperseded it at step 6 and nothing references it now. Happy to delete it in this PR — left in only because removing it wasn't asked for.