Synthetic benchmark: all() handle mint/lift cost per fresh instance (wasmtime + deltic) - #82
Merged
Conversation
…cale Synthetic benchmark isolating the per-fresh-instance cost of the tests.all() protocol from corpus/generator work, quantifying the instance-per-case overhead discussed in #22/#25: components/bench-suite registers N trivial cases (BENCH_CASES env, default 10000), and two phase-timing drivers measure instantiate / all#1 (registry build + mint + lift) / all#2 (mint + lift, registry cached) / name / run / teardown per fresh instance — bench-mint (wasmtime, byte-identical engine config and untyped-Val calls as the production Runner) and js/runner-deltic/bench-mint.mjs (pinned deltic embedder, plain node). Findings (docs/findings.md 19-21), medians on one 17-core dev box: - wasmtime 47: all() at 10k cases = 3.2ms per instance, splitting ~3:1 into guest registry build (~240ns/case) vs mint+lift (75-80ns/handle, linear through 30k); instantiate 21us, store drop 83us. So the guest-side registry build, not the handle lift, is the larger half of #22's measured K=1 tax — SDK static-table work and #25 outrank a direct-access interface until stacked. - deltic pre-83fff30/Node 24: same shape, bigger constants — instantiate ~650us (30x), lift ~340-370ns/handle (4.3x), boundary calls ~25us (10x). - harness.mjs freshCases relocation (linear name() scan) is the dominant JS-leg cost at scale: ~N/2 x 25us =~ 130ms/case at 10k, ~20x the all() it sits on top of; positional relocation is sound (all() order is contractually deterministic) and erases it. The bench artifacts stay out of every gate: bench-suite is a non-default workspace member like the other fixtures, not built by just build, and not a lockfile citizen.
…ld vanishes The mint benchmark's follow-up (issue #25): with the registry-build half measured as ~3x the lift at 10k cases, try wizening the suite so fresh instances are born with the case table built. It works, but only by driving wasmtime-wizer 47 as a library (Wizer::run_component takes a caller-supplied instantiate closure, so our linker satisfies test-context — a host resource init never calls — plus full WASI). The CLI path is blocked three ways, recorded as finding 22: the invoke grammar rejects versioned interface qualifiers, unknown-import stubbing cannot synthesize resource types, and composed bundles hit 'nested components with modules not currently supported'. The init entry is therefore a bare-named wizer-initialize export: bench-suite's new wizer-init feature adds it as a second inline-WIT world, merged by wasm-component-ld. keep_init_func(false) — the default — emits an invalid component (dangling core-instance export reference); the driver keeps it. Custom sections survive the rewrite: scheduling and drift checks work on the wizened artifact. Measured (findings 23-24), 10k-case suite, medians: - wasmtime: all#1 3.15ms -> 663us (= all#2: born initialized); instantiate unchanged at ~19us (CoW absorbs the 122KB -> 1.29MB snapshot); store drop 80us -> 12us. End-to-end K=1 full-isolation run: 30.8s -> 7.1s sequential, 1.14s at jobs=8 — per-case isolation on a wizened suite now undercuts #22's shared-instance numbers. - deltic: net ~1.5x only — all#1 6.9ms -> 2.9ms but instantiate 0.78ms -> 2.17ms (no CoW; the active data segment is copied per instantiation). K>1 remains the JS-leg lever. Also corrects finding 21's scan-cost constant: hot-loop name() is ~3.4us (a cold single call measures ~26us), so the freshCases scan averaged ~17ms/case at 10k (33ms worst, measured), not 130ms; fixed by positional relocation in PR #83 (33.4ms -> 0.0ms measured). New surfaces, all out of the gates: bench-suite feature wizer-init (default build unchanged — a pure suite world), runner feature wizer with the required-features bin wizer-preinit (optional dep wasmtime-wizer, absent from default builds; clippy clean under the feature).
Collaborator
Author
|
Scope addition (second commit): the wizer experiment the mint numbers motivated. Component-level pre-init works via wasmtime-wizer 47 as a library ( |
…e extra export Correction prompted by reading the actual grammars (wasm-wave's FuncNameToken lexer, wit-parser's ItemName): the invoke syntax supports versioned interfaces and always has — the version goes LAST (pkg:ns/iface.func@1.2.3), resolving the dot ambiguity; the export-name-order form I tried is rejected by design. 'The invoke grammar cannot name versioned exports' in the previous commit was wrong; the unhelpful 'invalid token' error obscured a syntax error. Consequences, all verified: - wizer-preinit now inits via polymorph:test/tests.all@0.1.0() — the parenthesized wave-call form (the bare item-name path requires [] -> []). Works on any UNMODIFIED suite artifact; the minted handles are per-call state and cost nothing measurable in the snapshot (1,286,599 vs 1,289,515 bytes with a dedicated no-op init export). Same numbers: all#1 700us = all#2 648us at 10k. - bench-suite's wizer-init feature (second inline-WIT world, optional wit-bindgen dep, force_registry_init hook) is deleted — dead weight. - The CLI's real remaining blockers, re-verified with correct syntax: resource imports cannot be stubbed (suite worlds), nested components (bundles), and wizer defaults WASI off (-S cli needed for env-reading inits). keep_init_func(false)'s dangling-export rewrite is a known open bug (bytecodealliance/wasmtime#13168) — moot for the all()-as- init route, where keeping is semantically mandatory anyway. - One genuine small gap remains upstream: the wave func-name lexer's semver subpattern is bare X.Y.Z, so prerelease-versioned interfaces (@0.3.0-rc-...) cannot be named in call form. Finding 22 rewritten accordingly.
Collaborator
Author
|
Third commit corrects the second: the invoke grammar was never broken — version-last syntax ( |
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measures the actual overhead of
test-casehandle minting at suite scale — the "all()per fresh instance" tax behind #22's instance-per-case numbers and #25's motivation — isolated from corpus preprocessing and generator work.What's here
components/bench-suite: N trivially-passing cases via one#[case_row](default 10000, host-set through theBENCH_CASESwasi env import). No corpus, no per-case data: registering, minting, and lifting handles is all it does. Non-default workspace member; not in any gate; not a lockfile citizen.bench-mintbin (component-test-runner): wasmtime driver with the productionRunner's exact engine config (pooling, CoW, epoch instrumentation, untypedValcalls), timing per fresh instance: instantiate / all#1 (registry build + mint + lift) / all#2 (mint + lift only — registry cached) / name[0] / run[0] / store drop.js/runner-deltic/bench-mint.mjs: the same phases under the pinned deltic embedder on plain node.Results (medians, 17-core x86_64 Linux dev box)
wasmtime 47 (
bench-mint --cases 100,1000,10000,30000 --instances 20):(Cross-checked against the production path:
COMPONENT_TEST_PROFILE=1 ct-runner … --only bench/mint/c00000reports all#1=3.19ms / all#2=0.93ms on the same artifact.)deltic pre-83fff30 / Node 24 (
--instances 10; translator init ~16ms + translate ~25ms one-off):What the numbers force
all()tax (~3:1 at 10k: 2.4ms build vs 0.79ms lift, both linear). A direct-access suite interface (get(name)/take(indexes), the Runner execution-policy guidance #22-thread sketch) caps at ~25% of the win unless the SDK also makes the registry lazy/static — SDK static-table work and Pre-initialized suites (build-time wizer snapshot of the case table) #25 come first, or stacked.freshCasesrelocation: a linearname()scan ≈ N/2 × 25µs ≈ 130ms per case at 10k — ~20× theall()it sits on top of.all()order is contractually deterministic across instances, so positional relocation (+ onename()verify) is sound and erases it. Not changed here; needs its own issue/PR.Verification
just checkgreen (fmt, clippy, host tests, WIT).ct-runner … --enumerateon the artifact: 10000 names, grammar-valid, prefix record covers them (drift check passes).