Repository navigation
Add quarto call axe, a hidden experimental site accessibility scanner - #14815
Merged
Merged
Conversation
cwickham
added a commit
that referenced
this pull request
Aug 27, 2026
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
cderv
reviewed
Aug 31, 2026
cderv
left a comment
cderv
left a comment
Member
There was a problem hiding this comment.
First round of review for our 📲
This was referenced Aug 31, 2026
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
The cap was applied to the sorted path list before any file was read, so redirect stubs spent it. On a site whose first *.html is a stub, --max-pages 1 discovered nothing and the scan exited 2 with real pages sitting right there. Walk and classify together, stopping once the cap holds that many real pages. A capped scan's stub ledger now only covers what was walked before the cap filled — consistent with the rest of a subset scan, whose counts already describe the subset rather than the site. Reported by cderv in review of #14815.
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
A flag typo produced the same signal as a real finding: axeScanConfig throws before axeScan's own handling, so it fell through to quarto's root handler, which exits 1 — the code this command reserves for 'new findings at the --fail-on threshold'. --themes matching nothing was wrong the other way, exiting 2 as if the scan hadn't finished. Both throw sites now raise the same named error and land on 3. A typo is not a result, so it doesn't borrow a code that means one. Reported by cderv in review of #14815.
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
Two entries for one signature merged: union the pages, take the most severe accepted impact. Acceptance records the impact it was written at and re-alerts on escalation past it, so 'most severe' is the lenient direction — a minor acceptance beside a critical one became 'critical, everywhere', and the stricter page inherited the looser page's tolerance in scope and impact both. Not a copy-paste slip worth absorbing, either: axe assigns impact per check rather than per rule, and a finding's impact is the worst across the occurrences in that scan, so two honest entries can disagree. parseBaseline now reports each repeat, reading the raw data so a duplicate and a typo surface in the same pass. acceptances() builds the map straight and treats a duplicate as an internal error. Suggested by cderv in review of #14815.
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
As a post-render script the command scanned whatever was in the output dir, including after 'quarto render index.qmd' — findings describing a mixture of this render's output and the last one's. Post-render scripts do run on incremental renders; the comment at project.ts:830 claims otherwise but the only guard there is 'if (!projResults.error)'. Skip with a note and exit 0, so an incremental render still succeeds. Preview reloads leave RENDER_ALL unset too, so they skip as well. The signal is an inference and says so in the comment: quarto has no 'am I a project script' flag, and QUARTO_PROJECT_OUTPUT_DIR is the closest thing — present for both script phases, absent otherwise. QUARTO_PROJECT_OUTPUT_FILES would be precise but goes missing under the QUARTO_USE_FILE_FOR_PROJECT_OUTPUT_FILES escape hatch. Suggested by cderv in review of #14815.
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
Two holes in the one property the scanner is sold on — that a cell always finishes. The recovery after a timeout sent Page.navigate with no deadline. If the transport itself was what wedged, that send never returned and the scan stopped finishing cells: a hang, not a timeout. Both recovery commands now go through sendWithin, which gives up after a second and never rejects. Only the load-event wait was capped before. And a timed-out cell was abandoned, not stopped: its in-flight round-trip still completed, and its next step ran against the page the *following* cell had navigated to — a stale Runtime.evaluate injecting axe and starting a second scan underneath it. Sends from the run body now go through a wrapper that refuses once the cell is abandoned. Both are pinned by unit tests over a stubbed transport; without the fix the wedged-transport one hangs until the harness's two-minute guard. Reported by cderv in review of #14815.
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
A bare repo path is no use to a reader of the changelog, so it is now a GitHub URL. The help text loses the pointer rather than carrying a long URL in a terminal: the quarto-web page that replaces dev-docs will get one, following the 'For details, see:' shape quarto run uses. Reported by cderv in review of #14815.
Member
Author
|
Thanks for this — a good first round. Every thread has a reply. Seven
All seven passed roborev, and the axe suite is green: 162 unit tests and the Four of your ideas I have kept as follow-ups rather than folding into this One question back, in the |
cwickham
marked this pull request as ready for review
September 3, 2026 17:00
Member
Author
|
@cderv round one is fully addressed — every thread has a reply, and six of them have a commit behind it. Since those replies I also merged Ready for another look. If it looks good to you, I think it is ready to merge. Two things that follow this PR rather than block it:
One question is still open, in the |
Registers `axe <site-dir>` under the already-hidden `dev-call` parent, with the six v1 flags (--pages, --max-pages, --viewports, --themes, --timeout, --settle) parsed into a typed AxeScanConfig and echoed. The scan, aggregate and report stages follow.
Ports the quarto-web harness's raw-CDP driver to typed TypeScript and replaces its render-time axe hook with scan-time injection: after load and settle, the vendored axe.min.js (4.10.3) is evaluated in the page and axe.run() awaited under the per-cell timeout. Any rendered site now scans as-is, offline, on one known axe version — so the CDN warm-up hack, the console-payload sniffing and the special render all come out. The orchestrator serves the site dir with httpFileRequestHandler, launches headless Chrome via getBrowserExecutablePath() on its own throwaway profile, walks the page x viewport x theme matrix, and writes raw per-cell JSON to _axe-checks/cells/. Cells fail closed: timeout, evaluation error or a missing payload is reported and exits 2, never a pass. Chrome rewrites its profile as it shuts down, so the temp profile dir is removed only after the process has really exited. On Ctrl-C an onCleanup handler kills Chrome and leaves the dir behind, since cleanup handlers can't await. Verified against a rendered website: all cells ok; identical results with Chrome's DNS blackholed (offline); prefers-color-scheme emulation catches a dark-only color-contrast failure and the mobile viewport a mobile-only one; a cell that blocks the main thread times out and the next cell still scans; SIGINT leaves no orphan Chrome; bogus, empty and non-directory site-dirs exit 2. Adds an optional onListen to handleHttpRequests so the scan's own progress output isn't preceded by Deno's "Listening on ..." line.
Neither exclusion did any work here. Tabster's `[data-tabster-dummy]` sentinels arrive with the preview client's Fluent UI bundle, and `.quarto-axe-report` is the in-page panel axe-check.js builds when a site is rendered with `axe:` metadata — so neither appears in the plain rendered sites this command scans. Both were carried over from axe-check.js's own axe.run() call, along with a "won't fix upstream" claim that tabster#288 does not support: that issue is still open with no maintainer response. If someone does scan an `axe:`-rendered site, the overlay's own violations are information rather than noise. Results on the test site are unchanged: 12 cells, 12 ok, same rule ids.
Ports the harness's aggregate and report stages, with the semantics the notes had already settled but the harness code had not caught up with. Signatures use the investigated "option 1" normalization: attribute values are kept with digit runs wildcarded, rather than stripped. Stripping turned `div[data-bs-target=".callout-4-contents"]` into the generic `div[data-bs-target]`, which every Bootstrap collapse, modal, tab and dropdown also matches — so one accepted callout defect silently suppressed unrelated conformance failures site-wide. The baseline is now the hand-written projection ledger: `signature` x `pages` x `impact`, empty `pages` meaning site-wide, a listed `pages` fail-closed at finding level, and escalation past the accepted impact re-alerting. `--update-baseline` is not ported; stale entries are reported and pruned by hand. Conformance labels come from `axe-check.js`'s own `axeConformanceLevel`, `impactRank` and `standardRank` rather than a second implementation, so a finding reads the same whether it came from a scan or the in-page report. That module guards its self-init on `typeof document`, which is what makes it importable here — the same property tests/unit/axe-*.test.ts rely on. `schemas.ts` holds Zod for both contracts. Baseline validation needed more than a schema: to Zod, a misspelled `impcat:` is an ignored unknown key plus a missing `impact`, so the report blamed `impact` while the culprit sat two characters away — and a `superRefine` never runs once a required field is missing. `parseBaseline` runs both checks and merges the issues, so the error names the typo and suggests the field it meant. `_axe-checks/` and `_axe-baseline.json` now anchor at the project root (nearest `_quarto.yml` at or above the site dir), falling back to the working directory for loose HTML. They sit beside the output dir, never inside it: a full render of a website or book deletes the output dir. Verified end-to-end on a rendered site: findings.json satisfies its own schema; all five baseline behaviours (site-wide accept, page-scoped accept, page-scoped re-alert on an unlisted page, impact-escalation re-alert, stale entry) land as expected; two hand-typo'd fields produce named errors with suggestions and exit 2. The report was driven in headless Chrome — row drill-down, column re-sort with row/detail pairing intact, the why-accepted column, the stale notice, and the copy-AI-briefing payloads.
The signature is the most consequential value the scanner computes: too broad and one accepted defect suppresses unrelated conformance failures site-wide, too narrow and a baseline entry stops matching when a counter changes. Two gaps around it. First, nothing recorded which normalizer produced a signature. `version` covers findings.json's field shape, so a normalizer change would leave every field intact while re-keying every signature — reported as "N new findings, M baseline entries not seen", which is indistinguishable from "you fixed everything and broke an equal amount". The ledger's notes, the record of why each finding was accepted, would quietly stop applying. `signatureScheme` makes that a named error instead, and points at occurrences[].target as the unchanged raw selector to re-annotate from. It is optional in the baseline, so the first hand-written ledger needs no ceremony. Second, the normalization had no tests. tests/unit/axe-signature.test.ts covers it as explicit should-collapse / must-stay-distinct pairs over selectors axe really emits on Quarto output, rather than as assertions about the regexes. The must-stay-distinct half is the important one: it pins that a callout, a modal, a carousel and a tabset link do not share a signature just because they all hang off data-bs-target, which is exactly what the pre-option-1 normalizer got wrong. A third group asserts exact output so a scheme change shows up as a diff rather than as a changed collapse count. 30 tests, all passing. Also covers a case with no fixture yet: a color-contrast payload missing its check data must fall back to the selector rather than key on "undefined on undefined" and collapse every contrast finding in the site into one.
The two-pass reader was built on a wrong premise. I had it that a schema can't report an unrecognized key and a missing required field together, because a `superRefine` never runs once a required field is missing. That is true of `superRefine`, but strict-key checking is part of the object parse itself, so `.strict()` reports both in one pass: findings.0.impact: Required findings.0: Unrecognized key(s) in object: 'impcat' Adjacent lines make the typo obvious, so the hand-rolled Levenshtein "did you mean" was polish sitting on top of ~60 lines that Zod already covers. Deleted, along with the tolerated-field set and the manual per-entry loop; schemas.ts drops from 341 to 275 lines. Keeping the paste-a-whole-finding affordance costs one expression rather than the machinery: declare every finding field as an ignorable optional and merge the entry schema over the top, so its required fields stay required. A finding pasted out of findings.json plus a note validates; an invented field is still rejected. `parseBaseline` survives only for the signature-scheme check, which needs to explain what a mismatch means rather than say "invalid literal". Behaviour change worth noting: the outer object is `.strict()` too, so an unknown top-level key is now an error. That catches a misspelled `findings` but also rejects a hand-added `"_comment"`, since JSON has nowhere else to put one. tests/unit/axe-baseline-parse.test.ts pins the deliberate choices: the paste shortcut validates, a typo names both halves, an invented field is rejected, `pages` and `note` are required so scope and rationale are never guessed, and a stale scheme is one named error rather than mass staleness. 42 axe unit tests passing.
A purpose-built inaccessible Quarto site under tests/docs/axe-scan/site, with every violation planted on purpose and a manifest README recording intent. Source only: a committed render would carry site_libs/ and go stale, so the smoke test renders it in setup and removes it after. Writing the manifest first paid off twice, because two planted cases did not behave as designed. The tier-2 links were meant to prove that three `#panel-N` components collapse to one signature while a `#settings-dialog` stays separate. They all collapsed. axe had keyed them on `href`, which the normalizer drops as volatile, so the attribute never reached the selector. Switching to `<button>` — no href to key on — makes axe select on the attribute and the case works as intended. That distinction is the whole justification for keeping attribute values, so it is now proven on real markup rather than on hand-written selectors. The dark-only contrast paragraph also produced a second, unplanned contrast failure: it contained inline code, and cosmo's code colour fails against the planted background. That would have made the fixture depend on a theme palette. The paragraphs are now kept free of any element carrying its own colour, and the README says why. One finding is left unplanned on purpose: Quarto's navbar-toggler, which is systemic and mobile-only and belongs to Quarto. It is documented, and no assertion depends on it — or on any total count — so a Quarto upgrade that fixes it cannot fail these tests. Tests, 77 passing: - axe-aggregate.test.ts (19) drives the aggregate stage from eight verbatim per-cell captures. Verbatim matters: keeping whole payloads means an axe upgrade that changes their shape surfaces in a fast unit test. - axe-baseline-reconcile.test.ts (15) uses hand-built cells, where control beats fidelity: site-wide and page-scoped accepts, the re-alert on an unlisted page, escalation, de-escalation staying accepted, stale detection in three variants, and duplicate-entry merging. - smoke/axe/axe-scan.test.ts runs the real thing — browser, static server, all three stages — in 15s, asserting projections looked up by signature. The fixture's committed _axe-baseline.json exercises all four reconcile outcomes in one file, including an entry accepted at a lower impact than planted (so escalation re-alerts) and one matching nothing (so staleness is reported). Its `.a11y-accepted-example` entry is a designed future failure: when exclude-in-source ships, that entry should turn stale.
`quarto-bld validate-bundle` runs eslint's no-undef over the bundled
quarto.js, and it caught two things this branch introduced.
The first was importing `axe-check.js` to reuse its three conformance
labellers. That looked like good reuse — the labels match the render-time
`axe:` report by construction — but `axe-check.js` is a browser module, so
esbuild inlined the entire overlay (reveal navigation, dashboard rescan,
report DOM) into the CLI bundle for the sake of three pure functions, and
its page globals surfaced as four no-undef errors:
'Reveal' is not defined
'getComputedStyle' is not defined
'bootstrap' is not defined
They are `typeof`-guarded and harmless at runtime, but whitelisting them in
the prelude would blunt a check whose whole job is catching real bundler
bugs. conformance.ts now mirrors the three functions in TypeScript, and
tests/unit/axe-conformance-parity.test.ts imports both and asserts they
agree across every tag shape axe emits — best-practice, each WCAG
version/level, multi-digit and multiple criteria, obsolete criteria, and no
conformance tags. Tests aren't bundled, so that comparison is free. The
bundle lost ~400 lines and all four errors.
A shared pure module would be better than either, and is verified viable —
`axe-check.js` is injected with `type="module"`, and a FormatDependency's
`resources` land in the same output dir as its `scripts`, so both sides
could import one `axe-conformance.js`. Deferred: it changes the render path
every `axe:` user hits, and adds a runtime 404 risk, for a hidden
prototype's benefit. Recorded in the design note's cut list for the public
command.
The second was `TextDecoderStream`, used to drain Chrome's stderr. It is a
genuine Deno global simply missing from the bug-finder prelude's list, so it
is added there rather than worked around in the scanner.
After both: 1 error, pre-existing, in puppeteer code — the
`@typescript-eslint/ban-ts-comment` rule-not-found that the prelude's own
comment already documents. Also confirmed axe.min.js is *not* inlined: it
stays a runtime resource, so the bundle doesn't carry 553KB of axe-core.
No changelog entry. `dev-call` has never appeared in a changelog across
eleven versions, and an entry would advertise a hidden command that carries
no API commitment while its semantics are in play.
Headers audited (twelve new files, current year), deno fmt and lint clean,
113 axe tests passing, full typechecked run green.
The theme axis was measuring nothing on a default Quarto site. Emulating `prefers-color-scheme: dark` leaves the dark stylesheet `rel="disabled-stylesheet"` and the body white, because `respect-user-color-scheme` defaults to false — so every dark cell was an exact duplicate of its light sibling, and one of the two axes justifying the 2x2 default earned nothing. The axis tests the two themes the *author* ships. The user's OS preference is only one possible route into a theme, and a page may offer either route: Quarto's `light:`/`dark:` themes are selected by its colour-scheme toggle, author CSS under `@media (prefers-color-scheme: dark)` by emulation. Both are now set, pointing at the same theme, and the result is read back rather than assumed — which makes `respect-user-color-scheme` invisible here: when it is true, emulation has already done the work and no click happens. Alignment runs per cell, not once per run. The toggle writes `localStorage` and every cell shares one origin, so a click leaks into later page loads. This is not hypothetical: on the fixture, the final light cell now records `colorScheme: "toggled"` because it *loaded in dark* and had to be clicked back. Aligning once would have scanned it dark while labelling it light. Each cell records how it got there — `emulated`, `toggled`, `assumed-identical`, or `unreachable`. The last matters most: a page that ships a dark theme but renders no toggle cannot be switched, so its dark cell is really a light cell and is reported rather than counted as coverage. Cells on pages with no Quarto dark theme reuse their sibling's payload instead of running axe twice on the same DOM. That is an assumption, not a proof, and it is wrong for a custom `.scss` rolling its own dark mode or for bslib's web components, whose `prefers-color-scheme` rules live in shadow roots. A sound version — enumerate every stylesheet including shadow roots, fail closed on anything unreadable — was built and removed as unjustified complexity for v1; both design notes record the gap as an M5 question. Reuse never omits a cell, so the accounting stays honest, and `--themes dark` alone scans such a page properly. Evidence this works: the new `theme.qmd` plants a contrast failure in darkly's own palette (`#6c757d on #222222`, 3.39:1). It appears in dark cells only, and it was **invisible before this change** — proof distinct from `media.qmd`, whose dark-only failure comes from an author media query and was already reachable by emulation. The fixture now covers both routes separately. 81 axe tests passing. Results are unchanged by the reuse optimisation: 24/24 cells, 12 findings, both before and after.
Found by scanning a real site with a `_brand.yml` that specifies no dark values: the "reused a sibling result" note never appeared, though the site has no dark mode. A light-only `_brand.yml` makes Quarto emit the colour-scheme link machinery with `data-mode="light"` links but no dark build and no toggle. The probe deciding whether a theme cell can reuse its sibling asked for `link.quarto-color-scheme[data-mode]`, which matches those light links, so it concluded there was a dark theme to select and scanned both themes for nothing. The alignment probe next to it already used `[data-mode="dark"]` correctly; the loose selector came in when the capability probe was simplified. Effect was wasted work rather than wrong results — the dark cells scanned the same DOM and produced the same findings — but on a brand.yml site that is half the matrix, which is exactly the cost this reuse exists to avoid. The fixture had no such case, which is why nothing caught it. `static/brand-light-only.html` reproduces the markup by hand rather than adding a second fixture project: the probe runs per page, not per site, so one static page pins it. Verified by reintroducing the bug and watching the new assertion fail, then restoring the fix. Its planted violation is a bare `<button>`, not an alt-less image. An image normalizes to `img` and would join the systemic `image-alt` finding from `_systemic.qmd`, changing that finding's page list — which is how the first attempt failed, coupling a colour-scheme regression fixture to an unrelated grouping assertion. Fixture is now 7 pages, 28 cells, 13 findings. 81 axe tests passing.
One site and one smoke test shared a page list, so adding a page moved
assertions in unrelated verifiers — twice. Now sites/{findings,matrix,
brand-light-only} each carry one concern with their own smoke test, the
shared plumbing lives in tests/smoke/axe/shared.ts, and the captured
unit-test cells are grouped per site so each aggregates as a scan that
really happened.
Page discovery now reads each rendered page once: the inline before-body script marker (quartoToggleColorScheme) means a light/dark pair; its absence means one cell, labelled 'default'. The matrix is known and printed before the browser launches. Each cell selects its mode by seeding localStorage["quarto-color-scheme"] before navigation and verifying from the body class after load — which deletes AxeColorSchemeMechanism and all four states, sibling-payload reuse, alignColorScheme, the toggle click, the post-click settle and both console warnings, and no longer misses a light-coloured theme in the dark: slot the way the [data-mode="dark"] probe did. A one-mode page whose sole bootstrap link measures dark (theme: darkly) is annotated dark-coloured — in the cell payload and on the console, never in the cell's name. --themes becomes a filter over the discovered modes: it prunes the light/dark pair, 'default' cells are always included, and a filter matching zero cells is an error. findings.json gains a per-page modes field; report.html is unchanged. New fixture site sites/darkly covers the annotation; the research behind the design is in llm-docs/html-dark-mode-architecture.md, with its discovery pointer in .claude/rules/formats/html-dark-mode.md.
_axe-checks/ is the scanner's report from a previous run — when the anchor is the site dir (output-dir: "." projects, or loose HTML scanned from inside it) a rerun would scan its own report. site_libs/ is vendored library code: reveal ships plugin/notes/speaker-view.html with every deck, and its findings belong upstream, not to the site's author. Both are skipped at any path depth. Found playtesting (M5): braid br-b2-self-scan, br-skip-site-libs.
scanCell's run body had no try/catch, so a transport failure — a crashed tab, a dropped WebSocket, a CDP protocol error — propagated out of runAxeScan uncaught: process dead, no findings.json, no report, even with hundreds of good cells on disk. Now it returns an error cell; if the connection is really dead, later cells fail fast the same way and the exit code says incomplete, with everything scanned so far still reported. The orchestrator also catches what scanCell can't (shared setup, a stage bug) and reports incomplete instead of dying on an uncaught error. Braid: br-b1-transport-fail-closed.
Redirect stubs (a zero-delay meta refresh generated from _redirects) load locally, then Chrome follows the refresh during the settle window — axe was auditing the external destination and attributing its violations to the local page path (found on positron-website: opensource.posit.co findings under blog/index.html). Also broke the offline story silently. After settle, the cell probes window.location.href and fails closed with a new 'redirected' status — origin and path decide; search and hash are ignored so reveal's history-API fragment rewrites still count as the same page. Belt and braces: axe's own result.url is checked after the run, so a redirect between probe and run is caught too. Braid: br-redirect-escape.
The how-to claimed a shared-include defect is one finding 'however many pages repeat it'. That holds when axe emits selectors that normalize identically (fixture-verified: the shared include yields bare img on both pages) and can break for anonymous elements whose surrounding DOM differs — axe then picks different minimal selectors and one cause splits. Say so.
Verified live: 'quarto call axe _site/' as a project post-render script scans after outputs are written, from the project dir (trailing slash fine); without --fail-on findings never fail the render, with it a threshold trip or incomplete scan fails quarto render with the scan's error line in the output. CI recipe now uses marocchino/sticky-pull-request-comment so each push updates one comment instead of stacking new ones.
The --help output is the first thing anyone who discovers the hidden command reads; align it with the docs' no-stability wording and point at dev-docs/axe-scan.md.
quarto preview of the report fails inside a project (_-prefixed dirs are not project inputs; verified: 'No output created by quarto render report.md'), while a plain quarto render works standalone. Document the hierarchy: editor/GitHub preview first, render-then-open for HTML, --report into the site source for the previewable-with-site flow. Deliberately no _quarto.yml inside _axe-checks: it would make renders there pick up the README and adds magic to a disposable directory.
320 CSS px is the one viewport width WCAG names (SC 1.4.10 Reflow — the 400%-zoom equivalent of a 1280 window). axe has no automated reflow rule, but every rule runs against the reflowed layout, and 320 sits in the same Bootstrap breakpoint regime as any phone width, so the mobile chrome still renders. 390x844 was the iPhone 12-14 logical size inherited, undocumented, from the quarto-web harness. Signatures never embed viewports, so committed baselines keep matching; only viewport-gated findings can churn.
'The scanner needs six CDP methods' conflated protocol commands (scan logic, always ours) with CdpClient's capabilities (generic transport: send/once/close). Spell out the three layers — launcher, transport, scan logic — and which part of cri.ts was declined (its mermaid-shaped facade) vs what a future unification would actually share (the deno-cri transport underneath, and sooner, the launcher).
cderv read 'posting it as a PR comment is workflow configuration, not tooling' as possibly announcing a gap Quarto should fill, and two sentences ending in colons back to back did not help.
The cap was applied to the sorted path list before any file was read, so redirect stubs spent it. On a site whose first *.html is a stub, --max-pages 1 discovered nothing and the scan exited 2 with real pages sitting right there. Walk and classify together, stopping once the cap holds that many real pages. A capped scan's stub ledger now only covers what was walked before the cap filled — consistent with the rest of a subset scan, whose counts already describe the subset rather than the site. Reported by cderv in review of #14815.
A flag typo produced the same signal as a real finding: axeScanConfig throws before axeScan's own handling, so it fell through to quarto's root handler, which exits 1 — the code this command reserves for 'new findings at the --fail-on threshold'. --themes matching nothing was wrong the other way, exiting 2 as if the scan hadn't finished. Both throw sites now raise the same named error and land on 3. A typo is not a result, so it doesn't borrow a code that means one. Reported by cderv in review of #14815.
Two entries for one signature merged: union the pages, take the most severe accepted impact. Acceptance records the impact it was written at and re-alerts on escalation past it, so 'most severe' is the lenient direction — a minor acceptance beside a critical one became 'critical, everywhere', and the stricter page inherited the looser page's tolerance in scope and impact both. Not a copy-paste slip worth absorbing, either: axe assigns impact per check rather than per rule, and a finding's impact is the worst across the occurrences in that scan, so two honest entries can disagree. parseBaseline now reports each repeat, reading the raw data so a duplicate and a typo surface in the same pass. acceptances() builds the map straight and treats a duplicate as an internal error. Suggested by cderv in review of #14815.
As a post-render script the command scanned whatever was in the output dir, including after 'quarto render index.qmd' — findings describing a mixture of this render's output and the last one's. Post-render scripts do run on incremental renders; the comment at project.ts:830 claims otherwise but the only guard there is 'if (!projResults.error)'. Skip with a note and exit 0, so an incremental render still succeeds. Preview reloads leave RENDER_ALL unset too, so they skip as well. The signal is an inference and says so in the comment: quarto has no 'am I a project script' flag, and QUARTO_PROJECT_OUTPUT_DIR is the closest thing — present for both script phases, absent otherwise. QUARTO_PROJECT_OUTPUT_FILES would be precise but goes missing under the QUARTO_USE_FILE_FOR_PROJECT_OUTPUT_FILES escape hatch. Suggested by cderv in review of #14815.
Two holes in the one property the scanner is sold on — that a cell always finishes. The recovery after a timeout sent Page.navigate with no deadline. If the transport itself was what wedged, that send never returned and the scan stopped finishing cells: a hang, not a timeout. Both recovery commands now go through sendWithin, which gives up after a second and never rejects. Only the load-event wait was capped before. And a timed-out cell was abandoned, not stopped: its in-flight round-trip still completed, and its next step ran against the page the *following* cell had navigated to — a stale Runtime.evaluate injecting axe and starting a second scan underneath it. Sends from the run body now go through a wrapper that refuses once the cell is abandoned. Both are pinned by unit tests over a stubbed transport; without the fix the wedged-transport one hangs until the harness's two-minute guard. Reported by cderv in review of #14815.
A bare repo path is no use to a reader of the changelog, so it is now a GitHub URL. The help text loses the pointer rather than carrying a long URL in a terminal: the quarto-web page that replaces dev-docs will get one, following the 'For details, see:' shape quarto run uses. Reported by cderv in review of #14815.
cderv
pushed a commit
that referenced
this pull request
Sep 8, 2026
A bare repo path is no use to a reader of the changelog, so it is now a GitHub URL. The help text loses the pointer rather than carrying a long URL in a terminal: the quarto-web page that replaces dev-docs will get one, following the 'For details, see:' shape quarto run uses. Reported by cderv in review of #14815.
cderv
force-pushed
the
feat/axe-scan-command
branch
from
September 8, 2026 09:22
85c93b8 to
08742c1
Compare
report.md called a stale entry resolved "if this was a full-site scan", which only accounts for --pages. A --themes light scan never runs a dark cell and a narrowed --viewports never runs the 320px one, so either can report an accepted finding as unseen without anything having looked for it, and a cell that failed closed inside an otherwise full run is the same hole. The baseline is hand-written and every entry exists because someone wrote it and said why, so inviting a prune on that evidence is the expensive direction to be wrong in. staleIsConclusive answers the one question a stale entry raises, and report.md and the console summary now say which case they are in. Axis narrowing is compared by membership rather than by count, so a repeated --themes light,light cannot stand in for the value it is missing. Also folded in, from the same pass over the flag surface: --max-pages accepted a fractional value, which is then compared with === against a running page count, so the cap silently never applied. --report is recorded in findings.json so the README's regenerate command reproduces the destination instead of reverting to the default. The field is nullish and additive, so the scheme does not move. The redirect-stub sniff matches its script markers inside <script> elements rather than anywhere in the document.
--report _axe-baseline.json ran a clean scan and then destroyed the ledger it had just read: the baseline is read before the scan and the report is written after it, with no check on where it lands. findings.json, README.md and .gitignore are overwritable the same way. A hand-written acceptance file is not something to lose to a flag typo, so a destination the scan owns is now a usage error before any work starts. The comparison folds case on Windows and macOS, where the filesystem is case-insensitive but case-preserving and an unfolded check would miss the spelling that matters. Deno.realPathSync was the obvious canonicalizer and was dropped: it corrected _AXE-BASELINE.JSON in one temp directory and left it alone in another, in the same run on the same machine, which is not something to back a destructive check with. The destination is only ever overwritten, never deleted. Clearing it up front would have closed a narrow window, an aborted run leaving a previous report in place, at the price of deleting a path the user named outside the directory the command owns.
Member
|
Thanks, this is in good shape. I did a second pass over the branch before merging and found two things worth fixing, so rather than open more threads I pushed the commits straight to the branch. Revert either if you disagree.
|
cderv
pushed a commit
that referenced
this pull request
Sep 9, 2026
A bare repo path is no use to a reader of the changelog, so it is now a GitHub URL. The help text loses the pointer rather than carrying a long URL in a terminal: the quarto-web page that replaces dev-docs will get one, following the 'For details, see:' shape quarto run uses. Reported by cderv in review of #14815.
cderv
pushed a commit
that referenced
this pull request
Sep 9, 2026
A bare repo path is no use to a reader of the changelog, so it is now a GitHub URL. The help text loses the pointer rather than carrying a long URL in a terminal: the quarto-web page that replaces dev-docs will get one, following the 'For details, see:' shape quarto run uses. Reported by cderv in review of #14815.
cderv
pushed a commit
that referenced
this pull request
Sep 9, 2026
A Chrome driver has three layers: launch the process, transport CDP commands over the socket, and the task logic on top. The launcher and transport were each duplicated between mermaid's cri.ts and the axe scanner; this unifies both, leaving only the task logic (cri.ts's facade vs. scanCell) separate since their commands differ. - New shared launchChrome() in src/core/cri/launch.ts replaces both ad hoc launchers, fixing three failure-path bugs along the way: an unread stdout pipe that could block Chrome, a cri.ts error path that could hang awaiting cmd.status, and onCleanup handlers that leaked one per mermaid render since they can't be unregistered - axe's CdpClient now runs its CDP transport on the vendored deno-cri, the same client cri.ts uses, removing the third hand-rolled WebSocket implementation in the tree - CdpClient adds its own in-flight-send tracking on top of deno-cri, since deno-cri leaves in-flight commands unsettled forever on a dropped socket rather than rejecting them - registerForExitCleanup() now installs its own exit handler, closing a gap where a command that spawns a browser without shelling out (quarto call axe) could orphan the process on Ctrl-C - New tests/smoke/axe/axe-transport-failclosed.test.ts exercises fail-closed behavior against a real browser instead of a manual check - llm-docs/axe-scan-architecture.md updated to describe the shared launcher and transport No user-visible change. Verified via full unit/smoke suites, a byte-identical mermaid PNG render before/after, and a full quarto-web scan producing an identical finding set before/after. Follow-up to a review thread on #14815.
cderv
pushed a commit
to quarto-dev/quarto-web
that referenced
this pull request
Sep 10, 2026
…er (#2185) Add docs/websites/website-accessibility.qmd for `quarto call axe` (quarto-dev/quarto-cli#14815), the experimental site-wide accessibility scanner. Covers prerequisites, running a scan, reading generated artifacts, the baseline workflow, exit codes, a CI recipe, post-render usage, and a worked example against Quarto 1.11.4. Also updates docs/output-formats/html-accessibility.qmd: its "Site-wide accessibility checks" section pointed at a future mode that now exists — that paragraph now links to the new page instead. - Page complements the generated `_axe-checks/README.md` (workflow here, field-by-field reference there) rather than duplicating it - Deliberately left out of the sidebar while the command is experimental; wiring it into Websites nav is a two-line `_quarto.yml` change later - No guidance yet on excluding content in source (vs. baselining) — that pathway doesn't exist yet - Worked example omits one `region` finding tied to a separate Quarto bug (quarto-dev/quarto-cli#14875) so regenerating the excerpts later reproduces unchanged text
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds
quarto call axe <site-dir>— a hidden, experimentalcommand that scans an already-rendered Quarto site for accessibility
violations with quarto-cli's vendored axe-core.
What it does:
as other Quarto features:
quarto install chrome-headless-shell, or asystem Chrome/Edge) over a page × viewport × color-mode matrix. Each
page's modes are discovered from its rendered HTML, so a light-only site
scans once per page and a
light:/dark:site scans both slots.as-is, offline, on the same axe version the render-time
axe:option uses.the color pair for
color-contrast), so one template defect on 40 pages isone finding with a count, and its id is stable across runs.
_axe-baseline.json):new findings alert, accepted ones are counted but out of the way,
escalations and out-of-scope recurrences re-alert.
_axe-checks/findings.json(machine contract),report.md(GitHub-flavored markdown), and a generated
README.mdthat documents theartifacts for whoever — or whatever agent — finds them in a repo.
infrastructure failure in the output and the exit code, never a pass.
Exit codes:
0complete,1new findings at/above--fail-on <impact>(opt-in, for CI),
2incomplete — and 2 takes precedence over 1.Why hidden under
call: the semantics (signature scheme, baselinegrain, flag surface) should be validated by real use before becoming public command. Nothing here is a public API commitment.
Docs in this PR:
dev-docs/axe-scan.md— how to use it (flags, baseline workflow, CIrecipe). Will eventually seed the
quarto-webdocs.llm-docs/axe-scan-architecture.md— how it works and why (pipeline, modediscovery, signatures, baseline semantics, the raw-CDP decision).
Deliberately not in scope (deferred to the public command):
_axe.ymlconfig, interaction states, ruleset scoping, source mapping (
.qmd→output), a rich report extension. The render-time
axe:option isuntouched; the scanner mirrors its conformance labellers, pinned by a parity
test.
Testing: 9 unit test files (115 tests) run the aggregate/baseline/discovery/
config logic browser-free against captured real axe payloads; 5 smoke test
files run the full command over fixture sites in
tests/docs/axe-scan/,including subprocess exit-code tests covering
--fail-on, fail-closedtimeouts, and a browser that cannot start. CI already installs
chrome-headless-shellunconditionally; a manualtest-smokes-paralleldispatch on this branch is green:
https://github.com/quarto-dev/quarto-cli/actions/runs/33106781443.
Checklist
I have (if applicable):
(not applicable yet: the command is hidden; docs live in
dev-docs/and graduate to quarto-web when it goes public)
AI-assisted PR