Settlement pump: between-calls guest liveness (embedder-api A11)

contracts/embedder-api.md

@@ -37,10 +37,17 @@ exception naming) while it is still cheap — semantics unchanged
 untouched**: the brand key stays `deltic.witError/1` (an opaque constant,
 CEWD-style, so pre-A10 copies and hand-rolled brands keep interoperating)
 and plan-format op discriminants (a different contract) keep `tag`;
+and plan-format op discriminants (a different contract) keep `tag`;
 A10 release note (2026-08-12): the rename changes `@deltic/protocol`'s
 export surface, so the JSR package moves to **0.2.0** — immutable `0.1.0`
 keeps the pre-A10 names for pre-A10 runtime prereleases (`^0.1.0` never
-resolves across), and post-A10 workspace publishes depend on `^0.2.0`.**
+resolves across), and post-A10 workspace publishes depend on `^0.2.0`;
+amendment A11 (2026-08-12) makes between-calls guest liveness normative:
+host-import settlements are serviced by a settlement pump while no export
+call is in flight, so background tasks parked on host-call wakeups (clocks,
+fetches) progress without embedder traffic — embedder-never-acts operations
+still hang (never trap) and failures still surface on the next driving
+call.**
 This document supersedes `descriptor-ir.md`'s interim
 "host value mapping" table as the destination for host-facing value shapes.
 The runtime's *raw* boundary (`instance.exports`, `HostImports`) keeps the
@@ -323,6 +330,18 @@ class PeerTrappedError extends Error {  // A7: a stream/future op whose peer ins
   containing object and are called unbound.
 - Params are positional; param names appear only in types/docs (they are
   excluded from the world digest — `contracts/digest.md`).
+- **Between-calls liveness** (amendment A11, 2026-08-12): guest progress
+  does not require an in-flight export call. A host import that settles
+  while no call is being driven is serviced then — a background task parked
+  on a waitable set whose pending host call resolves (a clock subscription,
+  a fetch) resumes at settlement time, not at the embedder's next call.
+  This is the JS-host analogue of dwelling in wasmtime's `run_concurrent`,
+  and what makes guest-encapsulated keep-alive tickers (componentize-go's
+  goroutine bridge over `wasi:clocks.wait-for`) self-driving under deltic.
+  Two prior bounds are unchanged: an operation waiting on the *embedder's*
+  half of a host stream/future still hangs until the embedder acts (never
+  a trap — see Streams and futures), and a settlement-time failure
+  surfaces on the next call into the instance, as before.
 
 ## Resources
 

docs/architecture.md

@@ -343,10 +343,18 @@ FIFO ready-queue by default; a seeded-shuffle mode (`DELTIC_SCHED_SEED` env
 var) exercises the spec-allowed nondeterminism in tests, verified across
 seeds. Documented at `runtime/src/task/scheduler.ts`. A load-bearing
 architectural rule discovered post-M2: **one driver per store** — concurrent
-`driveAsync` loops can double-resume threads; between export calls the host
-pump stands down whenever an export-call driver is live (the invariant and
-its benignity argument are documented at the site in
-`runtime/src/exec/boundary.ts`).
+`driveAsync` loops can double-resume threads; between export calls the two
+fallback drivers stand down whenever an export-call driver is live (the
+invariant and its benignity argument are documented at the site in
+`runtime/src/exec/boundary.ts`). There are exactly three drivers: export
+calls, the host-activity pump (embedder stream/future operations landing
+between calls), and — since embedder-api amendment A11 — the settlement
+pump, which services host-import settlements that land while the store is
+driver-idle. The settlement pump is what gives background tasks host-driven
+liveness between export calls (a task parked on a waitable set whose pending
+host call is a clock resumes at settlement time); wasmtime only delivers
+such wakeups while the embedder dwells in `run_concurrent`, but a JS host's
+event loop is always dwelling, so deltic makes it unconditional.
 
 Named divergence (2026-08-10, [#92](https://github.com/lann/deltic/issues/92)):
 **the async form of `subtask.cancel` is not atomic under jspi.** The

runtime/src/exec/boundary.ts

@@ -33,6 +33,7 @@
   clearResumingThread,
   EventCode,
   withActivation,
+  hasRealHostCall,
   hasResumingThread,
   type EventTuple,
   NeedsJspi,
@@ -41,7 +42,9 @@
   packSubtaskResult,
   PendingCapability,
   notifyInstancePoisoned,
+  realHostCalls,
   Store,
+  storeQuiescent,
   Subtask,
   WaitableSet,
   SubtaskState,
@@ -214,7 +217,7 @@
     stringEncoding: opts.stringEncoding,
     memory: opts.memory,
     realloc: opts.realloc === null ? null : (o, os, a, n) => {
       const realloc = require(opts.realloc, "realloc")!;
       const p = callCore(realloc, [o, os, a, n]);
       trapIf(p.length !== 1 || typeof p[0] !== "number", "realloc result");
       return (p[0] as number) >>> 0;
@@ -459,6 +462,10 @@
     if (store.hostFailure !== undefined) throw takeHostFailure(store);
     if (done()) {
       traceDrive("drive", store, done, "EXIT-done");
+      // Fully-synchronous completion: no `driveAsync` ran, so its exit hook
+      // will not fire — arm the settlement pump here for any host calls the
+      // guest registered fire-and-forget during this drive.
+      ensureSettlementPump(store);
       return;
     }
     // A thread parked on a Promise (jspi) can only progress after a microtask
@@ -586,12 +593,14 @@
  * the same store interleave their `serviceSettled`/`tick` phases, and the
  * host-stream pump was observed to trip `Trap: table entry empty` out of
  * `runCallbackLoop` when it drove unconditionally alongside an export call's
- * loop. Export calls own their loops and cannot yield to anyone; the pump is
- * a *fallback* driver — it exists only for host operations that land BETWEEN
- * export calls — so it is the side that stands down, using the two accessors
- * below, narrowing the window to the cooperative residue described above.
- * When an export call's loop is live it already races `pendingHostCalls` and
- * `store.awaiting`, i.e. it pumps host activity on the embedder's behalf.
+ * loop. Export calls own their loops and cannot yield to anyone; the pumps
+ * are *fallback* drivers — the host-activity pump for embedder operations
+ * that land BETWEEN export calls, the settlement pump (below) for host-call
+ * settlements that land between them — so they are the side that stands
+ * down, using the two accessors below, narrowing the window to the
+ * cooperative residue described above. When an export call's loop is live it
+ * already races `pendingHostCalls` and `store.awaiting`, i.e. it pumps host
+ * activity on the embedder's behalf.
  */
 const driverDepth = new WeakMap<Store, number>();
 const driverIdle = new WeakMap<Store, { p: Promise<void>; r: () => void }>();
@@ -613,6 +622,155 @@
   return w.p;
 }
 
+// ---------------------------------------------------------------------------
+// The settlement pump: liveness between export calls
+// ---------------------------------------------------------------------------
+//
+// A host-import promise that settles while a driver is live is serviced by
+// that driver (`driveAsync` races `store.pendingHostCalls`). One that settles
+// while NO driver is live only mutates scheduler state — the registration
+// site's continuation delivers results and readies threads, but nothing calls
+// `serviceSettled`/`tick`, so the work sits queued until the next export call
+// or host stream/future operation happens to drive the store. For a guest
+// with genuinely background work — the canonical shape is a task parked WAIT
+// on a waitable set whose pending host call is a clock (a componentize-go
+// keep-alive ticker, a wasi:clocks `wait-for`) — that turned "the host will
+// wake me" into "the embedder's next unrelated call will wake me": a liveness
+// gap, not a policy (wasmtime's event loop delivers such wakeups whenever the
+// embedder dwells in `run_concurrent`; on a JS host the event loop is always
+// dwelling).
+//
+// The settlement pump closes the gap: whenever a driver exits leaving real
+// host calls outstanding (`hasRealHostCall` — activity arms excluded, they
+// mean "the embedder may still act", not "the host owes an event"), a
+// detached keeper parks on `Promise.race` of those calls and, when one
+// settles, drives the store to quiescence with the same loop and the same
+// cooperative discipline as the host-activity pump above it in the driver
+// hierarchy:
+//
+//   * it stands down whenever an export call's loop is live
+//     (`storeDriverDepth` / `whenStoreDriverIdle`, plus the `> 1` clause in
+//     its `done`, exactly as `HostActivity.#pumpAsync`);
+//   * its `done` returns true whenever `pendingHostCalls` is empty, which is
+//     the precondition of BOTH deadlock traps in `driveAsync` — the pump can
+//     therefore never convert the documented embedder-never-acts hang into a
+//     trap (see the `driveStoreAsync` note above);
+//   * failures park on `store.hostFailure` for the next embedder call to
+//     surface, the channel every between-calls driver already uses.
+//
+// Every real `pendingHostCalls` entry is born during guest execution, i.e.
+// inside some driver, so arming at driver exit (`driveAsync`'s finally and
+// `drive`'s synchronous completion) observes every registration. One known
+// exception is documented rather than wired: a HOST-initiated async resource
+// dtor (embedder `drop()` between calls, cabi/handles.ts `callDtorGated`)
+// registers outside any driver; its settlement surfaces at the next drive
+// exactly as before this pump existed.
+//
+// STALE SNAPSHOTS: the keeper races the real host calls it saw when it
+// parked. A drive it performs can register NEW calls (the keep-alive ticker
+// re-arming is the routine case), and `ensureSettlementPump` may be called
+// while the keeper is already parked. Both are handled by a nudge promise
+// raced alongside the snapshot: arming an already-live pump fires the nudge,
+// the keeper wakes, re-snapshots, and re-parks.
+
+const settlementPumps = new WeakSet<Store>();
+const settlementNudges = new WeakMap<Store, { p: Promise<void>; r: () => void }>();
+
+function armSettlementNudge(store: Store): Promise<void> {
+  let n = settlementNudges.get(store);
+  if (n === undefined) {
+    let r!: () => void;
+    const p = new Promise<void>((res) => (r = res));
+    n = { p, r };
+    settlementNudges.set(store, n);
+  }
+  return n.p;
+}
+
+function fireSettlementNudge(store: Store): void {
+  const n = settlementNudges.get(store);
+  if (n !== undefined) {
+    settlementNudges.delete(store);
+    n.r();
+  }
+}
+
+/**
+ * Ensure a settlement pump is watching `store`'s real outstanding host calls.
+ * Idempotent and cheap; called at every driver exit. Never throws.
+ */
+export function ensureSettlementPump(store: Store): void {
+  if (settlementPumps.has(store)) {
+    // Already parked (or driving): wake it so it re-snapshots the race —
+    // this call may be reporting host calls registered after it parked.
+    fireSettlementNudge(store);
+    return;
+  }
+  if (store.hostFailure !== undefined) return;
+  if (!hasRealHostCall(store)) return;
+  settlementPumps.add(store);
+  void settlementPumpLoop(store);
+}
+
+async function settlementPumpLoop(store: Store): Promise<void> {
+  let failed = false;
+  try {
+    for (;;) {
+      // Stand down while any driver is live: it races `pendingHostCalls`
+      // itself and services settlements on the guest's behalf.
+      while (storeDriverDepth(store) > 0) {
+        await whenStoreDriverIdle(store);
+      }
+      // A parked failure belongs to the next embedder call (the only place
+      // it can surface); driving into it here would just consume and re-park
+      // it in a loop.
+      if (store.hostFailure !== undefined) return;
+      const real = realHostCalls(store);
+      if (real.length === 0) return;
+      const nudge = armSettlementNudge(store);
+      // Rejections are not this pump's to report: the registration site's
+      // own continuation parks them on `store.hostFailure`.
+      await Promise.race([
+        ...real.map((p) => p.then(() => {}, () => {})),
+        nudge,
+      ]);
+      if (storeDriverDepth(store) > 0) continue;
+      // Drive unconditionally after a wake: `storeQuiescent` cannot see a
+      // READY waiting thread (the usual product of a settlement — the
+      // continuation readied the guest and deleted its own host call), so
+      // gating the drive on it skips exactly the work this pump exists to
+      // do. `driveAsync` drains ready threads before consulting `done`, and
+      // a vacuous round exits on its first `done` evaluation.
+      await driveStoreAsync(
+        store,
+        // Quiescence, not completion — and the same three exit clauses as
+        // the host-activity pump: nothing only an event-loop turn could
+        // advance; `pendingHostCalls` empty (the deadlock traps'
+        // precondition, so this pump provably never traps); another driver
+        // appeared (ours is the 1).
+        () =>
+          store.pendingHostCalls.size === 0 ||
+          storeQuiescent(store) ||
+          storeDriverDepth(store) > 1,
+        "settlement pump",
+      );
+    }
+  } catch (e) {
+    failed = true;
+    store.hostFailure ??= e;
+  } finally {
+    settlementPumps.delete(store);
+    // Close the exit race: an `ensureSettlementPump` that saw us live and
+    // fired the nudge after our last snapshot check must not be lost.
+    if (
+      !failed && store.hostFailure === undefined &&
+      storeDriverDepth(store) === 0 && hasRealHostCall(store)
+    ) {
+      ensureSettlementPump(store);
+    }
+  }
+}
+
 async function driveAsync(
   store: Store,
   done: () => boolean,
@@ -862,6 +1020,10 @@
       const w = driverIdle.get(store);
       driverIdle.delete(store);
       w?.r();
+      // The store just went driver-idle; if real host calls remain, hand
+      // liveness to the settlement pump (which stands down again the moment
+      // any driver starts).
+      ensureSettlementPump(store);
     }
   }
 }
@@ -1466,7 +1628,7 @@
     task.return_(results);
     // Post-return runs after the results were read out of guest memory,
     // with may_leave cleared (reference canon_lift).
     const postReturn = require(opts.postReturn, `${name} post-return`);
     if (postReturn !== null) {
       assert_(inst.mayLeave, "post-return with may_leave already false");
       inst.mayLeave = false;
@@ -1512,7 +1674,7 @@
   // *mixed* activation, which pin (c) punishes: the first Suspending import
   // it reached would trap.
   const callback = enterWasm(
     require(opts.callback, `${name} callback`)!,
     input.mode,
   );
   const [packed] = normalizeCoreValues(

runtime/src/exec/host_streams.ts

@@ -66,11 +66,13 @@ import {
   BUFFER_MAX_LENGTH,
   type ComponentInstanceState,
   CopyResult,
+  markHostActivityArm,
   type PayloadChunk,
   sameElemType,
   SharedFutureImpl,
   SharedStreamImpl,
   type Store,
+  storeQuiescent as quiescent,
 } from "../task/mod.ts";
 
 /**
@@ -202,32 +204,13 @@ export class HostBuffer {
  * their presence as a reason to keep looping (that is the "activity keeps
  * pendingHostCalls non-empty forever" hazard: a pump whose exit condition is
  * `pendingHostCalls.size === 0` would never exit).
- */
-const activityArms = new WeakSet<Promise<unknown>>();
-
-/**
- * Is there anything left that only a turn of the event loop could advance?
- * Activity arms do not count: they say "the embedder may still act", which is
- * precisely the state in which the pump should stop and let the operation's
- * promise stay pending (the documented hang).
  *
- * `store.settled` (an array of settled-but-unserviced activation tails) DOES
- * count: it gates `tick`, so exiting with a tail queued is a lost wakeup —
- * the store is wedged until some other driver appears, and between export
- * calls there is none.
+ * The registry and the two predicates over it (`hasRealHostCall`,
+ * `storeQuiescent`, imported above as `quiescent`) moved to
+ * task/scheduler.ts so that boundary.ts's settlement pump — the OTHER
+ * between-calls driver — shares the same classification without an import
+ * cycle. Arms are minted here and marked via `markHostActivityArm`.
  */
-function quiescent(store: Store): boolean {
-  return store.settled.length === 0 && store.awaiting.size === 0 &&
-    !hasRealHostCall(store);
-}
-
-/** Is there host-call work outstanding that is not just an activity arm? */
-function hasRealHostCall(store: Store): boolean {
-  for (const p of store.pendingHostCalls) {
-    if (!activityArms.has(p)) return true;
-  }
-  return false;
-}
 
 /**
  * Keeps `store.pendingHostCalls` non-empty while a host end is live, so the
@@ -250,7 +233,7 @@ class HostActivity {
   #arm(): void {
     if (this.#store === null || this.#promise !== null || this.#closed) return;
     this.#promise = new Promise<void>((r) => (this.#resolve = r));
-    activityArms.add(this.#promise);
+    markHostActivityArm(this.#promise);
     this.#store.pendingHostCalls.add(this.#promise);
   }
 

runtime/src/task/scheduler.ts

@@ -937,6 +937,60 @@ export class Store {
   }
 }
 
+// ---------------------------------------------------------------------------
+// Host-call classification (shared by the drivers in exec/)
+// ---------------------------------------------------------------------------
+
+/**
+ * Host-activity "arm" promises, by identity: entries a driver parks in
+ * `Store.pendingHostCalls` purely to say "the embedder may still act". They
+ * are NOT outstanding work — treating them as such is the "activity keeps
+ * `pendingHostCalls` non-empty forever" hazard documented in
+ * exec/host_streams.ts — so the between-calls drivers filter them out via
+ * `hasRealHostCall`/`realHostCalls`. The registry lives here (rather than in
+ * exec/host_streams.ts, which mints the arms) so exec/boundary.ts's
+ * settlement pump can share the classification without an import cycle.
+ */
+const hostActivityArms = new WeakSet<Promise<unknown>>();
+
+/** Mark `p` as an activity arm (exec/host_streams.ts `HostActivity`). */
+export function markHostActivityArm(p: Promise<unknown>): void {
+  hostActivityArms.add(p);
+}
+
+/** Is there host-call work outstanding that is not just an activity arm? */
+export function hasRealHostCall(store: Store): boolean {
+  for (const p of store.pendingHostCalls) {
+    if (!hostActivityArms.has(p)) return true;
+  }
+  return false;
+}
+
+/** Every outstanding host call that is real work (not an activity arm). */
+export function realHostCalls(store: Store): Promise<unknown>[] {
+  const out: Promise<unknown>[] = [];
+  for (const p of store.pendingHostCalls) {
+    if (!hostActivityArms.has(p)) out.push(p);
+  }
+  return out;
+}
+
+/**
+ * Is there anything left that only a turn of the event loop could advance?
+ * Activity arms do not count: they say "the embedder may still act", which is
+ * precisely the state in which a between-calls driver should stop and let the
+ * operation's promise stay pending (the documented hang, exec/host_streams.ts
+ * module header).
+ *
+ * `store.settled` (settled-but-unserviced activation tails) DOES count: it
+ * gates `tick`, so exiting with a tail queued is a lost wakeup — the store is
+ * wedged until some other driver appears.
+ */
+export function storeQuiescent(store: Store): boolean {
+  return store.settled.length === 0 && store.awaiting.size === 0 &&
+    !hasRealHostCall(store);
+}
+
 /**
  * The reference's `canon_lift` sync driving loop (line 2213):
  *