jco: scheduler stops delivering waitable events once a detached task holds in-flight imports across export calls

[lann]

Performance cost while this stands: the portable workaround (bounded clock polling instead of cross-task wakers, 5 ms method quantum / 10 ms pump tick) puts the composed endpoint echo handshake at ~21 ms wasmtime↔wasmtime versus ~4 ms for the spike's event-driven pump on the same wire and relay — roughly a 5× handshake latency cost paid on every host to keep behavior identical, reversible when this and the inter-task-wakeup gap below are fixed.

Found implementing the endpoint surface (#3); this is the defect that blocks the lann:iroh/endpoint jco target. Diagnosed on jco-transpile 0.5.2 (composed-call symptom also reproduced on 0.6.1).

The architecture that triggers it. endpoint.bind spawns a detached pump task (wit_bindgen::spawn_local) that permanently holds in-flight async imports (a websocket receive plus a clock tick), then returns. Any later export call — an async resource method, or even a second bind — never executes its first wasm slice: with JCO_DEBUG the new task's enter() runs and its bookkeeping completes, but no waitable events are delivered afterward; every task ends suspended and the instance is dead. Wasmtime interleaves the same component correctly (just matrix's endpoint-relay-wasmtime-wasmtime row).

What was ruled out along the way, each reproducing the hang independently until removed:

• Cross-task wakers: wit-bindgen's inter-task-wakeup feature signals through a guest-internal unit stream; wasmtime delivers the write, jco never does (parked tasks sleep forever). The endpoint now uses bounded clock polling instead of cross-task wakers, so this is dodged — but it is its own jco gap worth reporting.
• Racing a clock future against a condition as a fallback would cancel an in-flight import subtask, which jco traps on (jco-transpile 0.5.2 traps on subtask.cancel: guests must resolve in-flight imports before returning #6) — so polling loops must await each wait-for to completion, which the implementation does.
• After both of those, the residual hang is the scheduler itself, as described above.

Also hit, separately: the wac-composed endpoint+demo component fails on jco with unexpected callee param count [1], _asyncStartCall invocation expected [5] at the demo→endpoint async cross-component call, on 0.5.2 and 0.6.1 both. So neither route (composed, or JS driving the surface directly) currently runs on jco.

State in-tree: the endpoint's jco leg is recorded as blocked in scripts/matrix.sh (comment) and the README; host-jco/src/run-endpoint.mjs is a complete JS consumer of the surface, ready to become the jco matrix row when this works. The spike demo's single-export architecture (everything inside one run call) is unaffected and stays green on jco — which localizes the defect to multi-export concurrency with live background tasks.

To do:

• Reported to the jco triage tracker, one issue per defect: Execution-slot queue serializes task lifetimes: a long-lived task deadlocks every later export call on the instance lann/jco#11 (scheduler stall), Guest-internal stream read completions are never delivered (wit-bindgen inter-task-wakeup parks tasks forever) lann/jco#13 (inter-task-wakeup guest-internal stream), Intra-composition async call fails at _asyncStartCall: 'unexpected callee param count [1], expected [5]' lann/jco#14 (_asyncStartCall on intra-composition async calls). Minimal reproductions (exec-model-style probes) remain to be extracted; the in-tree reproductions are named in each report.
• When fixed: wire run-endpoint.mjs into the matrix, and consider restoring waker-based wakeups in endpoint/src/endpoint_impl.rs (the bounded-polling ruling is recorded in its module doc).

Related: #6, #7, #9 — the async/streaming corner of jco now accounts for every workaround in this repository.

[lann]

Toolchain update in 447f6fb: host-jco now runs a pinned build of lann/jco's all-fixes branch (+ the first two commits of lann/jco#27) instead of npm jco-transpile 0.5.2. The string-lowering post-transpile patch is gone (fixed upstream), and stream batching removes the pathological per-element stream costs (import-stream probe ~6.5 s → ~30 ms). The endpoint jco leg stays blocked, but the blocker is now diagnosed end to end — jco's execution slot serializes task lifetimes per component, and relaxing it exposes deeper interleave races (JS and guest-side current-task registers, subtask bookkeeping) — written up on lann/jco#11, with fix attempts in lann/jco#27. run-endpoint-demo.mjs (driver for the composed artifact) joined run-endpoint.mjs as the ready consumer; with the full #27 stack the composed component got as far as QUIC Initials on the wire before the next race, so the remaining work is genuinely upstream scheduler hardening, not endpoint-side workarounds.

[lann]

The polling tax this issue records by anecdote is now a gated pair in just bench (9f917cf): spike-relay (event-driven pump) handshake median 4 ms vs endpoint-relay (bounded polling) 16 ms on the same wire, relay, and crypto. The recovery item — restoring event-driven wakeups when the jco scheduler gaps close — keeps this issue open; the bench pair is the before/after instrument for it.

[lann]

Evaluation against lann/jco all-fixes @ dbad4d7d (this repo @ bcaed0f, setup.sh with the jco pin moved to that rev; everything else per scripts/setup.sh). Headline: the scheduler stall this issue reports is fixed — the leg now runs QUIC on the wire — and the remaining blockers are two new, precisely-scoped guest-visible races, filed as lann/jco#47 and lann/jco#48.

What's fixed since the issue's state

• The headline defect (Execution-slot queue serializes task lifetimes: a long-lived task deadlocks every later export call on the instance lann/jco#11) is gone. jco's per-component execution slot — the thing that serialized whole task lifetimes and deadlocked every export call after bind's detached pump — was removed in fix(bindgen): allow concurrent task lifetimes within a component instance lann/jco#42, together with single-flight task resumption covering the first layer of interleave races that relaxing it exposes (the "deeper races" this issue's first comment predicted). The in-tree regression test for it is exactly this issue's architecture: a lingering pump task holding in-flight work while later exports enter the same component.
• Intra-composition async call fails at _asyncStartCall: 'unexpected callee param count [1], expected [5]' lann/jco#14 is fixed (spilled-params _asyncStartCall, fix(bindgen): allow spilled caller params in guest->guest async calls lann/jco#22): the composed demo's demo→endpoint async calls work — the composed leg's server binds and prints its endpoint-id.
• Both "ready consumers" now run: run-endpoint.mjs and run-endpoint-demo.mjs bind, dial through the local relay, and reach the QUIC handshake under Node/JSPI. just probes passes on jco at this pin, including blockon-in-detached-pump.
• Guest-internal stream read completions are never delivered (wit-bindgen inter-task-wakeup parks tasks forever) lann/jco#13's covered shape passes on all-fixes (the inter-task-wakeup coverage from test(transpile): cover wit-bindgen inter-task wakeup across CM tasks lann/jco#24, wit-bindgen 0.60 guests, green in the suite) — relevant to this issue's recovery item, with the 0.59 caveat below.
• Context: the tls sibling's conformance suite went fully green under jco on this same build (Conformance: jco targets blocked on upstream jco defects in composed p3 stream/future paths polymorph-tls#28), via fix(bindgen): emit future.transfer trampoline before instantiation code lann/jco#36/bench: reinstate the webcrypto boundary call-count assertion on a surviving host #38/endpoint: revert the bounded-polling pump to event-driven wakeups (the jco-era workaround is now pure cost) #42/endpoint: event-driven wakeups replace the bounded-polling pump #44/Converge .deps pins on the JSR-consuming siblings; gate module identity on the resolved graph #46.

What still blocks the leg — two fresh signatures, one family

Both legs get through bind/dial and die mid-handshake in guest asserts of the current-task/event-routing family (the register discipline that lann/jco#31's residual and the lann/jco#30 evaluation flagged as the remaining structural gap):

1. Direct-surface leg (Guest waker woken across CM task boundaries under post-#42 concurrency: wit-bindgen inter_task_wakeup_disabled stub panics mid-QUIC-handshake (iroh endpoint, direct-surface leg) lann/jco#47): the server guest panics in wit-bindgen 0.59's inter_task_wakeup_disabled.rs:36 — a waker woken across CM task boundaries. The bounded-polling architecture never creates that situation under wasmtime (same binary passes composed wasmtime↔wasmtime, roundtrip_ms=26, on the same relay); post-endpoint: revert the bounded-polling pump to event-driven wakeups (the jco-era workaround is now pure cost) #42 jco scheduling manufactures it.
2. Composed leg (Guest callback dispatched under a null/foreign current-task register during nested sync p2 import handling (waitable.rs !task.is_null panic; iroh endpoint, composed leg) lann/jco#48): the client guest panics waitable.rs:244 assertion failed: !task.is_null() — traced to a guest re-entry during nested sync p2 output-stream.write handling under a wrong/cleared register (the unconditional-null finally in the sync register window, plus resume paths outside the single-flight tick loop).

So the remaining work is still "genuinely upstream scheduler hardening" as this issue put it — but the surface has shrunk from "the scheduler stops delivering events" to two named register-routing defects with guest-side asserts, traces, and this repo pinned as the repro.

Notes toward the to-dos here

• Matrix wiring: premature until per-path datagram ceiling: relay/WebRTC have no physical 1200 B MTU, and max-datagram-size changes when connection.path moves #47/endpoint WIT: no way to read the peer's application close (code, reason) — protocol refusals race their own connection close #48 land — but when they do, the only repo-side change needed is the pin bump + the matrix rows; both drivers worked unmodified.
• Waker restoration (the 4 ms vs 16 ms recovery): the guest-internal wakeup path now works on all-fixes for wit-bindgen 0.60 guests. The endpoint guests build with 0.59; worth folding a 0.60 bump into the recovery experiment when the time comes. The polling-tax instrument (just bench pair) is untouched by any of this and stays the right before/after gate. Note the host-side polling cadence changed in fix(bindgen): allow concurrent task lifetimes within a component instance lann/jco#42 (idle 30 ms → 10 ms, immediate re-tick after a resumption), so the endpoint's own 5 ms/10 ms quanta now sit on a slightly faster host loop — the bench medians may already have moved a little even before waker restoration.
• Toolchain pin: all-fixes now auto-publishes install-ready tarballs on every merge (https://github.com/lann/jco/releases/tag/all-fixes-<sha7>, e.g. all-fixes-dbad4d7), so the .deps/jco source build in setup.sh could become a release-tarball pin like the websocket sibling's — optional, the source pin works fine.