Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
9b486a3
Add stage 1 of the UI label drift detector
mdlinville Aug 12, 2026
0eef333
Add the docs corpus index (step 2)
mdlinville Aug 12, 2026
550dca3
Match docs literals as whole terms; grade edge cases instead of class…
mdlinville Aug 12, 2026
0149e15
Fix a style error in the prose-vs-label examples
mdlinville Aug 12, 2026
eb38b1d
Add structural signals: rename pairing, gate scope, flag lifecycle (s…
mdlinville Aug 12, 2026
513088a
Add triage, ownership, and the markdown report (step 4)
mdlinville Aug 12, 2026
e2b8694
Resolve ownership once per run, not once per finding (step 5a)
mdlinville Aug 13, 2026
1b7ced6
Add the ledger: decisions persist, everything else is recomputed (ste…
mdlinville Aug 13, 2026
2cfeb46
Add the entrypoint: scan, incremental mode, and decide (step 6)
mdlinville Aug 13, 2026
9c3b693
Report the run's counts as JSON, and stop indexing our own reports (s…
mdlinville Aug 17, 2026
2c7f296
Add the sink: a daily scan carried in one rolling draft PR (step 7)
mdlinville Aug 17, 2026
aa19110
Clear the CodeQL quality findings from review
mdlinville Aug 17, 2026
3745c6b
Remove the step-1 entrypoint, superseded at step 6
mdlinville Aug 17, 2026
a9ef6e6
Address the contained findings from review
mdlinville Aug 17, 2026
a8cec21
Fix two lane-assignment bugs found in review
mdlinville Aug 17, 2026
8f21ae9
Merge branch 'main' into code_monitoring_investigation
mdlinville Aug 17, 2026
aeaa0c1
Fold in the three review items that are cheaper now than later
mdlinville Aug 17, 2026
54f2632
Merge remote-tracking branch 'origin/code_monitoring_investigation' i…
mdlinville Aug 17, 2026
7771c69
Merge branch 'main' into code_monitoring_investigation
mdlinville Aug 19, 2026
60323e0
Merge branch 'main' into code_monitoring_investigation
mdlinville Aug 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,47 @@ git commit -m "Sync code examples from docs-code-eval"
git push
```

## UI label drift

**Workflow**: `uidrift-scan.yml`

Watches `wandb/core` for user-facing label changes that leave this repo's docs stale, and carries the resulting report in one rolling draft PR. The detector is `scripts/uidrift`; see [`scripts/uidrift/ADAPTING.md`](../../scripts/uidrift/ADAPTING.md) for what it looks for and why. This workflow is only the sink.

### Setup required before the first run

`wandb-docs-source-reader` is installed on `wandb/docs-code-eval` and `wandb/weave-internal` only, so **it cannot read `wandb/core` yet**. Pick one:

- **Preferred**: install `wandb-docs-source-reader` on `wandb/core` with **Contents: read**. Needs a `wandb` org owner. No secret changes here; the workflow already asks for `repositories: core`.
- **Fallback**: add a repository secret `WANDB_CORE_TOKEN` holding a token that can read `wandb/core`. The workflow prefers the App and falls back to this, so adding the App install later needs no edit.

With neither in place the first step fails immediately and names both options, rather than burning four minutes on a clone that cannot authenticate.

### Triggers

- **Scheduled**: weekdays at 13:00 UTC (6am PT), so a report is waiting at standup
- **Manual**: `workflow_dispatch` with `since` (window start for a non-incremental run), `seed` (ignore existing reports and rescan the whole window), and `dry-run` (report to the job summary, open no PR)

### What it does

1. Clones `wandb/core` — full history, single branch, no working tree. The ADAPTING.md table records why shallow and blobless clones were both rejected; do not "optimize" this without reading it.
2. Runs the scan. `--incremental` by default, taking its base from the head SHA in the newest report filename under `uidrift/reports/`; falls back to `--since` when no report exists yet.
3. Writes the report to the job summary, so a run is readable even when it opens no PR.
4. If there are findings (or a reopened decision), opens or updates a **draft PR** on the rolling branch `uidrift/drift-report` with the funnel counts, lane breakdown, and how to record a decision.
5. Fails the run — after the PR exists — if any stored decision reopened. That means a writer's earlier dismissal no longer matches the docs, which only a human can settle.

Merging the PR advances the watermark. Closing it unmerged is also safe: the next run rescans the same range and supersedes the report.

### Reviewing a report

Each row lands in one of three lanes: **agent** (mechanical rename, safe to apply), **pair** (a writer scopes it, an agent applies it), **human** (prose has to be written). Rows that are wrong get recorded rather than deleted:

```bash
PYTHONPATH=scripts python3 -m uidrift.scan decide <id> \
--status dismissed --by <you> --agreement false_positive --note '<why>'
```

`--agreement` is the detector's only feedback channel and cannot be reconstructed later. A dismissal reopens by itself if docs later start covering that surface, so it suppresses a row without hiding it forever.

## Readability delta

**Workflow**: `readability-delta.yml`
Expand Down
370 changes: 370 additions & 0 deletions .github/workflows/uidrift-scan.yml

Large diffs are not rendered by default.

51 changes: 51 additions & 0 deletions .github/workflows/uidrift-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: UI label drift tests

# Runs the detector's own test suite on every PR that touches it.
#
# The scan workflow (uidrift-scan.yml) triggers only on schedule and dispatch,
# so before this existed nothing ran scripts/uidrift/tests on a pull request: a
# regression could merge and first surface days later, in a rolling PR that a
# writer would reasonably read as drift rather than as a broken detector.
#
# The suite is stdlib unittest, needs no network and no wandb/core checkout, and
# finishes in seconds -- so it is cheap enough to gate every relevant PR.

on:
pull_request:
paths:
- 'scripts/uidrift/**'
- '.github/workflows/uidrift-tests.yml'
push:
branches: [main]
paths:
- 'scripts/uidrift/**'
workflow_dispatch:

permissions:
contents: read

concurrency:
group: uidrift-tests-${{ github.ref }}
cancel-in-progress: true

jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7

- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
# Matches the scan workflow. No dependency step: scripts/uidrift is
# stdlib-only by design.
python-version: '3.11'

- name: Run the detector test suite
# Discovery runs from scripts/ so the tests' relative imports (`from ..
# import config`) resolve against the uidrift package, which is how they
# are written and how they run locally.
run: |
python3 -m unittest discover -s scripts/uidrift/tests -t scripts -p 'test_*.py' -v
5 changes: 5 additions & 0 deletions .mintignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ runbooks/
node_modules/
.github/

# UI label drift reports and the decision ledger. Committed as a record for the
# docs team, never published: only /*.md is ignored above, so a nested
# uidrift/reports/*.md would otherwise become a page.
uidrift/

# Top-level config and assets (not doc pages)
# Note: Do not exclude .js or .css. Mintlify loads
# these automatically if present in the content root.
Expand Down
Loading
Loading