feat(audit-feature-flags): rebuild as a doctor (wizard native) by ryanwaits · Pull Request #329 · PostHog/context-mill

ryanwaits · 2026-08-06T18:57:16Z

Problem

Following up on the flags-doctor thread:

audit feature-flags is static and read-only today: greps the source, reads the roster, hands back a report. It never checks whether flags reach the running app, and can't act on what it finds. Its own report template signs off with "runtime assumptions unproven by the static code."

So a typo'd flag key returns undefined forever with no error, and since the audit only compares tenant→code, it can't see it. On a test app with one planted, the current skill told me to "confirm that every active flag matches one of the four keys referenced in code: beta-serach, …" — i.e. validate the typo as truth.

Changes

Rebuilt as a doctor: verify (read-only) → one wizard_ask multi-select → apply only what's selected. Shaped after web-analytics's doctor per Edwin's suggestion. Skill id, cli: block, and command unchanged. All nine static checks kept.

New:

Live delivery — probes /flags with the project's key via the app's configured host, cross-checks the roster, surfaces per-flag reason codes.
Ghost keys — code→tenant direction.
Evaluation reporting — catches $feature_flag_called suppression, config-level and per-call.

The interlock, from Vincent's "evaluating ≠ sending events" point: staleness is measured from $feature_flag_called, so a heavily-used flag with suppressed events looks unused — archive it and you turn off a live feature. The doctor withholds all tenant-side cleanup until reporting is verified, and explains why.

Bot filtering of /flags is expected behavior, so it's a teaching note rather than a finding — and probes send a browser-like UA so we don't false-alarm on it.

Test plan

npm test (140) and npm run build green; zip builds, warlock clean, cliEntries unchanged.
End-to-end on a Next.js app with three planted defects (--local-mcp + the wizard PR): all three caught with the right severities, two selected fixes applied surgically, archive candidate withheld by the interlock, healthy flags verified delivered.
Same app on the published skill: all three silent.

ryanwaits · 2026-08-06T19:30:05Z

Specific areas, if you're offering: mostly trade-offs I made deliberately but am not totally certain about yet:

1. Speed vs. depth. The old audit was static-only and quick. This one also probes /flags, fetches the roster, and asks for consent, so it's meaningfully slower. Read-only is still the default outcome (tick nothing, nothing happens; CI mode skips the fix phase entirely), but "I just want the fast static pass" is now a worse experience than it was. I've got --report-only and per-area flags (--only delivery) scoped as follow-ups.

2. Rebuilding in place vs. a new skill. I kept the id, command, and cli: block so there's no orphan and no second thing to maintain, but it does mean anyone who ran this expecting a read-only report now meets a prompt. Reasonable, or would you rather the doctor lived alongside the old one for a release?

3. Check calibration. Severities are my judgment, and across runs I saw one borderline check flip between pass and suggestion. If any thresholds feel wrong to you, that's the most valuable thing you could call out.

4. Prompt weight. More checks means a longer skill. It's comfortable on a small app but I haven't run it against a large monorepo...

edwinyjlim

overall, this is a solid list. the checks.md file is a little unwieldy though. i'd imagine it's going to be hard to scale prompts inline like that, but i like the task <> subagent structure

edwinyjlim · 2026-08-06T19:44:56Z

+## Available tools
+
+{{> mcp-tool-calling}}
+
+**Verify (read-only):**
+- `Glob` / `Grep` / `Read` — static checks over the source tree.
+- `Bash` (plain `curl` only) — live `/flags` probes. Keep probe commands minimal and legible; see `references/checks.md` for the exact shapes.
+- `feature-flag-get-all` (or the equivalent MCP flag-listing tool; `execute-sql` fallback) — the project's flag roster.
+- `mcp__wizard-tools__check_env_keys` — which env keys exist (never reveals values).
+- `docs-search` — latest doc URLs for remediation links.


edwinyjlim · 2026-08-06T20:12:27Z

-references:
-  preamble: "**Read ONLY this file.** Do not read any other reference file until this one tells you to."


FYI this is a prompt guardrail for the agent to follow the references in a strict sequence. I know you've defined the flow in the description.md but sometimes agent ignores instructions

ok thats fair, good catch - ive restored as prose guardrails

edwinyjlim · 2026-08-06T20:15:34Z

+## Doc links per check
+
+- `ff-bootstrap-when-known-set`, `ff-bootstrap-distinct-id-mismatch` → https://posthog.com/docs/feature-flags/bootstrapping
+- `ff-await-readiness`, `ff-default-values`, `ff-identified-only-pre-auth-targeting`, `ff-eval-before-identify` → https://posthog.com/docs/feature-flags/best-practices
+- `ff-key-authenticates`, `ff-flags-endpoint`, `delivered-*`, `ghost-*` → https://posthog.com/docs/feature-flags/troubleshooting
+- `ff-evaluated-not-reported` → https://posthog.com/docs/experiments/exposures
+- `stale-*`, `ff-active-but-unreferenced` → https://posthog.com/docs/feature-flags/cleaning-up-stale-flags
+- `ff-local-eval-polling-interval`, `ff-local-eval-in-edge-handlers` → https://posthog.com/docs/feature-flags/local-evaluation
+- `ff-test-ci-gating` → https://posthog.com/docs/feature-flags/cutting-costs


ooh mapping tasks to docs is interesting

gewenyu99 · 2026-08-06T20:29:54Z

Some discussions based on what you raised. Feel free to discuss but defer execution if you have ideas. I again don't expect any of this to happen during this short superday:

Speed vs. depth. There are two things to know here: do you think the model and effort choices make sense right now? How might you approach optimizing from this perspective? We also have a DAG based way to create execution graphs of mini agents, how might this change your assumptions?

And if, say, we can't make it faster, what would you optimize for? speed vs. depth? how would you make this decision or run an experiment.

Do you think they should stay separate or should all audits become doctor skills? How might you think about architecting this.
Check calibration: How would you calibrate this in the wild once this is released?
Prompt weight: how do you think about realistic upper limits of this wizard program? What are the real constraints, and what are subjective ones we have to judge based on taste/feedback?

sarahxsanders

this is a big step up from what it replaced, well done!! some high level stuff:

going from grep the source to actually probing flags is great
some contradictions in the context that need some cleaning up, I'd be curious to see how many turns the agent wastes when it gets tripped up by one!

sarahxsanders · 2026-08-06T20:13:15Z

+
+## Part 1 — Static checks (parallel subagents)
+
+Emit `[STATUS] Auditing feature flag correctness`, then make **six `Agent` tool calls in a single message** for the correctness checks (Tasks A–E and Task I). When all five return, emit `[STATUS] Auditing feature flag cost optimization` and dispatch the cost checks (Tasks F–H) the same way — one `Agent` call per check that actually runs, in a single message. Two cost checks are gated on the pre-flight local-evaluation signal; for a gated check that is skipping, emit its `audit_resolve_checks` update directly (`status: "pass"`, `details: "skip: local evaluation not detected"`) instead of dispatching a subagent.


this contradicts a little, "six agent tool calls" - "When five return", looks like it's left over from the old version possibly, but I'd clean this up because I could see the agent tripping on this and wasting turns

yep, it was six after all - good catch again. i also also relettered to kill the task/check collision

sarahxsanders · 2026-08-06T20:14:58Z

+
+### Check N — `ff-stale-rolled-out` (sweep)
+
+What it verifies: flags at **100% rollout with no conditions** that are still gated in code — the check is equivalent to a hardcoded value: dead branches, needless evaluations, cleanup candidates. (The tenant-side twin, zero-reference active flags, is Task I in Part 1 — `ff-active-but-unreferenced`.)


I think there's some stuff here that needs cleaned up, is left over from the old version or maybe edits you did! you mention the wrong task and check

also side note: this is a superrrrrr long file, I wonder if the agent is a little slow based on the file side and sheer amount of context here?

definitely, good catch

ryanwaits · 2026-08-07T00:55:35Z

@gewenyu99 on speed vs depth: i think i'd optimize for speed, because depth is something you can ask for specifically later. the checks already group into areas, so --only delivery or --only observability gets you the deep version of the thing you actually came here about. people already have the primitives to dig into any one of these (mcp,api, etc) they don't need the wizard to bundle all of it every run. the quick pass telling you where to look is still good enough in many ways

on separate vs all doctors: separate commands, same content, mostly so we don't end up with two copies of the check wording quietly drifting apart

ryanwaits added 3 commits August 6, 2026 12:01

feat(audit-feature-flags): rebuild as a doctor (verify, diagnose, fix)

ce4fa4a

feat(audit-feature-flags): detect per-call send_event suppression

9598e84

fix(audit-feature-flags): tighten consent prompt and MCP degradation

a9f8395

ryanwaits requested a review from a team as a code owner August 6, 2026 18:57

ryanwaits mentioned this pull request Aug 6, 2026

feat(audit): make feature flags a wizard native doctor program PostHog/wizard#1064

Closed

edwinyjlim reviewed Aug 6, 2026

View reviewed changes

sarahxsanders reviewed Aug 6, 2026

View reviewed changes

ryanwaits added 4 commits August 6, 2026 16:19

refactor(audit-feature-flags): extract per-check subagent prompt files

6265038

docs(audit-feature-flags): add phase-scoped reference read discipline

feafc5e

fix(audit-feature-flags): scope ask option claim, emit end statuses

9faf1f7

fix(audit-feature-flags): map parent delivery rows to remediation docs

07679a9

ryanwaits mentioned this pull request Aug 6, 2026

docs: add feature flags audit and fix guide PostHog/posthog.com#19322

Draft

		references:
		preamble: "Read ONLY this file. Do not read any other reference file until this one tells you to."


		## Part 1 — Static checks (parallel subagents)

		Emit `[STATUS] Auditing feature flag correctness`, then make six `Agent` tool calls in a single message for the correctness checks (Tasks A–E and Task I). When all five return, emit `[STATUS] Auditing feature flag cost optimization` and dispatch the cost checks (Tasks F–H) the same way — one `Agent` call per check that actually runs, in a single message. Two cost checks are gated on the pre-flight local-evaluation signal; for a gated check that is skipping, emit its `audit_resolve_checks` update directly (`status: "pass"`, `details: "skip: local evaluation not detected"`) instead of dispatching a subagent.


		### Check N — `ff-stale-rolled-out` (sweep)

		What it verifies: flags at 100% rollout with no conditions that are still gated in code — the check is equivalent to a hardcoded value: dead branches, needless evaluations, cleanup candidates. (The tenant-side twin, zero-reference active flags, is Task I in Part 1 — `ff-active-but-unreferenced`.)

Uh oh!

Conversation

ryanwaits commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Test plan

Uh oh!

ryanwaits commented Aug 6, 2026

Uh oh!

edwinyjlim left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gewenyu99 commented Aug 6, 2026

Uh oh!

sarahxsanders left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ryanwaits commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ryanwaits commented Aug 6, 2026 •

edited

Loading