Skip to content

Commit 255f913

Browse files
lann-akamailann-akamai
committed
Engine-shell canary lanes: SpiderMonkey nightly + JSC trunk + Deno-canary probe (#22)
Non-gating findings lanes running trunk-fresh engines against the full conformance corpus, weeks ahead of playwright's browser rolls. Deno-lane shaped: the driver spawns the shell, the in-shell entry reads the corpus straight off disk and emits @deltic: JSON protocol lines on stdout, and classification reuses the same xfail.ts + overlay + stale-delta machinery as the browser lanes (classify/totalsOf/diffTotals extracted verbatim into tools/browser/classify.ts; browser lanes verified byte-equivalent). - tools/shell/{entry,polyfill,bundle,run-lane,fetch,deno-canary}.ts + capability probes (wat+wasm: multi-memory, wasm-GC, exception-handling, memory64, tail-calls, relaxed-simd, JSPI round trip). Capability matrix reported per run BEFORE the corpus, so a proposal landing or default flipping is visible even where the corpus doesn't reach it. - Shell platform gaps discovered and polyfilled narrowly (lane-side, not runtime): TextEncoder/TextDecoder (USVString replacement semantics - proven by the string/values corpus), crypto.subtle.digest(SHA-256), atob/btoa. Browsers have all four natively, which is why the browser lanes never noticed. - harness/shell/expectations/: spidermonkey-nightly recorded from a real run - EXACT Deno parity (1250/0/99), all capabilities true, zero overlay entries; JSPI unflagged on nightly (the default flip Firefox 153 still prefs). jsc-trunk seeded pending the first CI run (the x86_64-only channel cannot execute on the arm64 dev box; the zip-layout discovery in fetch.ts is the flagged first-run risk). - .github/workflows/canary.yml: weekly cron + workflow_dispatch, ci.yml conventions; x64 job (SM + JSC + deno-canary) and arm64 job (SM + deno-canary; no JSC arm64 channel). Drivers exit 0 on recorded deviations, 2 on infrastructure failure; nothing feeds the release gate. - deno-canary probe: dl.deno.land canary fetch, deno/V8 version delta report, capability matrix + full conformance under the canary binary (today: V8 15.0.245.2-rusty on both, table identical). Shell quirks recorded in-code: SM needs --module=<path> (bare form misparses positionals); jsc -m tolerates trailing arguments; both resolve relative paths against CWD; fixed repo-relative paths replace argv plumbing entirely.
1 parent 2a236f0 commit 255f913

28 files changed

Lines changed: 1934 additions & 110 deletions

‎.github/workflows/canary.yml‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
name: Canary
2+
3+
# Weekly + on-demand only: SpiderMonkey nightly, JSC trunk, and a Deno-canary
4+
# probe run trunk-fresh engine builds against the conformance corpus (issue
5+
# #22). These are FINDINGS lanes, never gates — every driver here exits 0
6+
# even with recorded deviations, per each lane's `required: false` overlay
7+
# and the drivers' own exit policy (`tools/shell/run-lane.ts`,
8+
# `tools/shell/deno-canary.ts`: exit 2 is reserved for infrastructure
9+
# failure, e.g. a missing shell binary or a channel unreachable). Nothing
10+
# here feeds `ci.yml`'s release gate.
11+
12+
on:
13+
schedule:
14+
- cron: "0 6 * * 1" # Monday 06:00 UTC
15+
workflow_dispatch:
16+
17+
env:
18+
DENO_VERSION: "2.9.5"
19+
WASM_TOOLS_VERSION: "1.247.0"
20+
21+
jobs:
22+
shell-canaries:
23+
runs-on: ubuntu-24.04
24+
timeout-minutes: 45
25+
steps:
26+
- uses: actions/checkout@v4
27+
with:
28+
submodules: true # third_party/component-model: testgen + conformance corpus
29+
- uses: dtolnay/rust-toolchain@stable
30+
with:
31+
targets: wasm32-unknown-unknown
32+
- uses: Swatinem/rust-cache@v2
33+
- uses: taiki-e/install-action@v2
34+
with:
35+
tool: wasm-tools@${{ env.WASM_TOOLS_VERSION }}
36+
- uses: denoland/setup-deno@v2
37+
with:
38+
deno-version: ${{ env.DENO_VERSION }}
39+
40+
- name: build the translator shim (wasm32)
41+
run: cargo build -p translator-shim --target wasm32-unknown-unknown --release
42+
- name: generate the conformance corpus
43+
working-directory: harness
44+
run: deno task gen
45+
46+
- name: cache shell downloads
47+
uses: actions/cache@v4
48+
with:
49+
path: .shell-cache
50+
# No version pin in the key: nightly/trunk builds change daily by
51+
# design, so this cache is a same-day rerun optimization only —
52+
# fetch.ts/deno-canary.ts each check for an existing binary but a
53+
# stale one just gets reused, matching a findings lane's tolerance
54+
# for slight staleness between the cron tick and the job running.
55+
key: shell-cache-${{ runner.os }}-${{ runner.arch }}-${{ github.run_id }}
56+
restore-keys: |
57+
shell-cache-${{ runner.os }}-${{ runner.arch }}-
58+
59+
- name: fetch SpiderMonkey nightly
60+
run: deno run -A tools/shell/fetch.ts spidermonkey
61+
- name: SpiderMonkey nightly lane
62+
run: deno run -A tools/shell/run-lane.ts spidermonkey --json sm-results.json
63+
64+
- name: fetch JSC trunk (x86_64)
65+
run: deno run -A tools/shell/fetch.ts jsc
66+
- name: JSC trunk lane
67+
run: deno run -A tools/shell/run-lane.ts jsc --json jsc-results.json
68+
69+
- name: Deno canary probe (V8-trailing-edge substitute for a d8 lane)
70+
run: deno run -A tools/shell/deno-canary.ts --json deno-canary-results.json
71+
72+
- name: upload results
73+
if: always()
74+
uses: actions/upload-artifact@v4
75+
with:
76+
name: shell-canaries-x64
77+
path: |
78+
sm-results.json
79+
jsc-results.json
80+
deno-canary-results.json
81+
if-no-files-found: warn
82+
83+
- name: append summary
84+
if: always()
85+
run: |
86+
{
87+
echo "## Engine-shell canaries (x64)"
88+
echo
89+
echo "SpiderMonkey nightly, JSC trunk, Deno canary — findings lanes, non-gating."
90+
for f in sm-results.json jsc-results.json deno-canary-results.json; do
91+
if [ -f "$f" ]; then
92+
echo
93+
echo "### $f"
94+
echo '```json'
95+
python3 -c "import json,sys; d=json.load(open('$f')); print(json.dumps(d.get('header', d), indent=2)[:4000])" || cat "$f"
96+
echo '```'
97+
fi
98+
done
99+
} >> "$GITHUB_STEP_SUMMARY"
100+
101+
# JSC has no arm64 build channel (jsc-built-products is x86_64-only) — this
102+
# leg runs the two lanes that DO have arm64 coverage (issue #22: "SM
103+
# nightlies publish aarch64 shells; JSC trunk and d8 do not").
104+
shell-canaries-arm:
105+
runs-on: ubuntu-24.04-arm
106+
timeout-minutes: 45
107+
steps:
108+
- uses: actions/checkout@v4
109+
with:
110+
submodules: true
111+
- uses: dtolnay/rust-toolchain@stable
112+
with:
113+
targets: wasm32-unknown-unknown
114+
- uses: Swatinem/rust-cache@v2
115+
- uses: taiki-e/install-action@v2
116+
with:
117+
tool: wasm-tools@${{ env.WASM_TOOLS_VERSION }}
118+
- uses: denoland/setup-deno@v2
119+
with:
120+
deno-version: ${{ env.DENO_VERSION }}
121+
122+
- name: build the translator shim (wasm32)
123+
run: cargo build -p translator-shim --target wasm32-unknown-unknown --release
124+
- name: generate the conformance corpus
125+
working-directory: harness
126+
run: deno task gen
127+
128+
- name: cache shell downloads
129+
uses: actions/cache@v4
130+
with:
131+
path: .shell-cache
132+
key: shell-cache-${{ runner.os }}-${{ runner.arch }}-${{ github.run_id }}
133+
restore-keys: |
134+
shell-cache-${{ runner.os }}-${{ runner.arch }}-
135+
136+
- name: fetch SpiderMonkey nightly
137+
run: deno run -A tools/shell/fetch.ts spidermonkey
138+
- name: SpiderMonkey nightly lane
139+
run: deno run -A tools/shell/run-lane.ts spidermonkey --json sm-results.json
140+
141+
- name: Deno canary probe
142+
run: deno run -A tools/shell/deno-canary.ts --json deno-canary-results.json
143+
144+
- name: upload results
145+
if: always()
146+
uses: actions/upload-artifact@v4
147+
with:
148+
name: shell-canaries-arm
149+
path: |
150+
sm-results.json
151+
deno-canary-results.json
152+
if-no-files-found: warn
153+
154+
- name: append summary
155+
if: always()
156+
run: |
157+
{
158+
echo "## Engine-shell canaries (arm64)"
159+
echo
160+
echo "SpiderMonkey nightly + Deno canary (JSC trunk has no arm64 channel — skipped)."
161+
for f in sm-results.json deno-canary-results.json; do
162+
if [ -f "$f" ]; then
163+
echo
164+
echo "### $f"
165+
echo '```json'
166+
python3 -c "import json,sys; d=json.load(open('$f')); print(json.dumps(d.get('header', d), indent=2)[:4000])" || cat "$f"
167+
echo '```'
168+
fi
169+
done
170+
} >> "$GITHUB_STEP_SUMMARY"

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,7 @@ node_modules/
55
dist/
66
# M3 browser lane: playwright browser downloads (PLAYWRIGHT_BROWSERS_PATH)
77
.browser-cache/
8+
# Engine-shell canary lanes (issue #22): fetched SpiderMonkey nightly / JSC
9+
# trunk / Deno canary binaries.
10+
.shell-cache/
811
*.tmp

‎harness/shell/expectations/jsc-trunk.ts‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
// JSC trunk lane expectation — a findings lane (best-effort, non-gating;
2+
// issue #22). SEEDED, pending the first CI run: the `jsc-built-products`
3+
// channel is x86_64-only, and this track was implemented on an aarch64 host
4+
// (per issue #22: "first CI run should sanity-probe the jsc-built-products
5+
// binary … but never executed here; wrong local arch"). `totals: null` means
6+
// the driver (`tools/shell/run-lane.ts` -> `tools/browser/classify.ts`
7+
// `diffTotals`) skips the TOTAL-row check entirely until a real run seeds it
8+
// — verified: `diffTotals` only runs `if (exp.totals)`.
9+
//
10+
// MACHINERY VALIDATION (not a parity target — do NOT read this as JSC
11+
// trunk's expected shape): the driver, entry, bundler, and protocol were all
12+
// exercised end-to-end locally against a STABLE jsc 2.52 (GTK, extracted
13+
// from the Debian/Ubuntu `libjavascriptcoregtk-bin` .deb — recipe in
14+
// `tools/shell/run-lane.ts`'s header) via:
15+
//
16+
// deno run -A tools/shell/run-lane.ts jsc \
17+
// --shell-bin /path/to/jsc --lib-path /path/to/libdir
18+
//
19+
// Expected (and observed) on that stable build per issue #22's empirical
20+
// table: capability matrix shows jspi=false, multiMemory=false (both land
21+
// only in trunk); a large deviation report follows because JSPI-needing
22+
// commands fail outright rather than classifying pending. That is a stable-
23+
// build artifact of running the machinery early, not a jsc-trunk finding —
24+
// no overlay entries are seeded from it. The first real CI run against
25+
// jsc-built-products trunk is what populates `deltas`/`totals` here for
26+
// real, the same way `spidermonkey-nightly.ts` was seeded from an actual
27+
// nightly run.
28+
//
29+
// SHELL-SURFACE FACTS specific to jsc (see `tools/shell/entry.ts`,
30+
// `tools/shell/polyfill.ts` for where these matter):
31+
// - `readFile(path, "binary")` (top-level global, not namespaced) for
32+
// binary reads.
33+
// - `-m <path>` (module mode) with positional args AFTER the module path
34+
// working reliably as `arguments` in the shell global — unlike
35+
// SpiderMonkey's `--module=`, this shell tolerated
36+
// `jsc -m file.mjs -- foo bar` cleanly, but the driver still passes no
37+
// positional args (parity with the SpiderMonkey invocation; the entry
38+
// needs none either way).
39+
// - `JSC_*` env vars for feature flags — an unknown one makes jsc exit
40+
// with an error, so nothing speculative is ever passed; this driver
41+
// passes none.
42+
// - unreachable trap wording: `"Unreachable code should not be executed"`
43+
// — already a `TRAP_MESSAGE_EQUIVALENTS` row in `harness/src/runner.ts`,
44+
// no matcher work needed here.
45+
46+
import type { ShellLaneExpectation } from "./types.ts";
47+
48+
export const jscTrunk: ShellLaneExpectation = {
49+
lane: "jsc",
50+
required: false,
51+
notes:
52+
"JSC trunk (jsc-built-products, x86_64 CI only). SEEDED — no CI run yet " +
53+
"(implemented on an aarch64 host; see this file's header). totals: null " +
54+
"so the driver skips the TOTAL-row check until the first real run.",
55+
deltas: [],
56+
totals: null,
57+
};
58+
59+
export default jscTrunk;

‎harness/shell/expectations/spidermonkey-nightly.ts‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
// SpiderMonkey nightly lane expectation — a findings lane (best-effort,
2+
// non-gating; issue #22).
3+
//
4+
// RESULT (2026-08-09, SpiderMonkey nightly `js` shell, linux-aarch64,
5+
// jsshell-linux-aarch64.zip fetched from
6+
// archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/): **the
7+
// lane runs the full corpus, Deno-identical.** All 59 files, 1395 commands,
8+
// zero deltas, zero stale xfails, zero unexpected failures.
9+
//
10+
// CAPABILITY MATRIX (this run): jspi = {suspending: true, promising: true,
11+
// roundTrip: true}, multiMemory = true, wasmGc = true, exceptionHandling =
12+
// true, memory64 = true, tailCalls = true, relaxedSimd = true. Every
13+
// proposal this lane's compile-probes (`tools/shell/probes/*.wasm`) check is
14+
// already implemented in this nightly build — consistent with SpiderMonkey
15+
// nightly being the trailing-edge-but-still-ahead-of-stable-Firefox
16+
// reference issue #22 was written to watch (JSPI unflagged here vs. Firefox
17+
// 153's `javascript.options.wasm_js_promise_integration` pref — see
18+
// `harness/browser/expectations/firefox.ts`).
19+
//
20+
// SHELL-SURFACE FINDINGS (feed the polyfill scope in `tools/shell/polyfill.ts`
21+
// and the shell detection in `tools/shell/entry.ts`):
22+
// - `os.file.readFile(path, "binary")` for binary reads; both relative
23+
// (resolved against CWD) and absolute paths work.
24+
// - `TextEncoder`/`TextDecoder`, `crypto.subtle.digest`, and `atob`/`btoa`
25+
// are ALL absent — none of the four is a browser/Deno-only convenience;
26+
// each is polyfilled in `tools/shell/polyfill.ts` (see its header for
27+
// the runtime call sites that drove the scope of each).
28+
// - `--module=<path>` (with `=`; a bare `--module <path> <positional>`
29+
// misparses the first positional as a second script to run — see
30+
// `tools/shell/run-lane.ts`'s `runShell` comment) with no positional
31+
// args at all is the reliable invocation; the entry needs none because
32+
// every path it reads is a fixed repo-relative location and the driver
33+
// sets the shell's CWD to the repo root.
34+
// - `print()` is the stdout sink; unprefixed shell diagnostics (module
35+
// load warnings etc.) show up on stdout too, hence the `@deltic:`
36+
// sentinel prefix in the protocol instead of assuming every line is
37+
// one of ours.
38+
// - Job-queue draining (`drainJobQueue()`) is present but not needed for
39+
// the corpus run itself: top-level `await` in module mode drives the
40+
// queue automatically (verified), same as Deno/browsers.
41+
//
42+
// Track this file the same way `harness/browser/expectations/firefox.ts`
43+
// tracks Firefox: any future delta gets a named, dated entry here, never a
44+
// blanket overlay.
45+
46+
import type { ShellLaneExpectation } from "./types.ts";
47+
48+
export const spidermonkeyNightly: ShellLaneExpectation = {
49+
lane: "spidermonkey",
50+
required: false,
51+
notes:
52+
"SpiderMonkey nightly (linux-aarch64 jsshell). Full Deno parity: " +
53+
"zero deltas, all compile-probes true (multi-memory/wasm-GC/EH/memory64/" +
54+
"tail-calls/relaxed-simd), JSPI round trip verified end to end.",
55+
deltas: [],
56+
totals: {
57+
commands: 1395,
58+
executed: 1349,
59+
passed: 1250,
60+
failed: 0,
61+
xfail: 99,
62+
pendingRuntime: 41,
63+
pendingCapability: 0,
64+
unsupportedDirective: 5,
65+
},
66+
};
67+
68+
export default spidermonkeyNightly;

‎harness/shell/expectations/types.ts‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
// Shell-lane expectation overlay types (issue #22).
2+
//
3+
// Same shape and role as `harness/browser/expectations/types.ts` (imports
4+
// `LaneDelta`/`LaneTotals` from there rather than redefining them — see this
5+
// track's dispatch); only the `lane` tag differs, since these are engine
6+
// shells, not browsers.
7+
8+
import type { LaneDelta, LaneTotals } from "../../browser/expectations/types.ts";
9+
export type { LaneDelta, LaneTotals };
10+
export { deltaKey } from "../../browser/expectations/types.ts";
11+
12+
export interface ShellLaneExpectation {
13+
lane: "spidermonkey" | "jsc";
14+
/** Human summary printed with the table. */
15+
notes: string;
16+
/** `true` = a red lane is a gate failure; `false` = findings-only lane.
17+
* Every canary lane in this track is `false` (issue #22: "findings lanes,
18+
* never gating"). */
19+
required: boolean;
20+
/** Expected per-file/line deltas against the Deno lane. */
21+
deltas: LaneDelta[];
22+
/** Expected TOTAL row. `null` = do not assert totals (e.g. jsc-trunk,
23+
* seeded pending its first CI run). */
24+
totals: LaneTotals | null;
25+
/**
26+
* Whole-file deltas: files this engine is not expected to complete at all
27+
* (e.g. a missing feature makes the executor unusable). Every command in
28+
* the file is exempted, with the reason recorded once.
29+
*/
30+
fileDeltas?: { file: string; reason: string }[];
31+
}

0 commit comments

Comments
 (0)