|
| 1 | +# Handoff: execnet on a Trio core |
| 2 | + |
| 3 | +Branch `feat/trio-host-thread-io`, draft PR **pytest-dev/execnet#422**. |
| 4 | +This is the doc to read first. Two companions: |
| 5 | + |
| 6 | +- **`ROADMAP-3.0.md`** — what this branch ships as, and what is still open. |
| 7 | + The work list lives there, not here. |
| 8 | +- **`handoff-history.md`** — the compressed record of what landed, with |
| 9 | + commit ranges and the lessons that cost a debugging round each. |
| 10 | + |
| 11 | +## How to work here |
| 12 | + |
| 13 | +``` |
| 14 | +uv run pytest testing/ # 552 passed, 66 skipped |
| 15 | +uv run pytest testing/ -n 12 # must stay green (~7s) |
| 16 | +uv run pre-commit run -a # never grep-filter its output |
| 17 | +uv run tox -e docs # builds with -W and doctests doc/example/ |
| 18 | +``` |
| 19 | + |
| 20 | +ssh paths have a real local harness in `testing/test_ssh_local.py` (an |
| 21 | +asyncssh server; needs a system ssh client). Hypothesis stress coverage |
| 22 | +is `testing/test_channel_stress.py` behind `--stress=N`. |
| 23 | + |
| 24 | +Known flakes, all timing: |
| 25 | + |
| 26 | +- `test_socket_installvia` EOFs rarely under load. |
| 27 | +- `test_gateway_status_busy` (numexecuting race) and |
| 28 | + `test_popen_stderr_tracing` (capfd race) keep their `flakytest` marks |
| 29 | + and XPASS when idle — see `handoff-history.md`. |
| 30 | +- `test_info_reports_what_a_coordinator_needs` failed once under `-n 12` |
| 31 | + (2026-07-31), green in isolation and on rerun; **not diagnosed**. |
| 32 | + |
| 33 | +CI runs pytest-xdist's own suite against this execnet — see |
| 34 | +"The xdist contract" in `ROADMAP-3.0.md`. That job is the one that |
| 35 | +catches what our suite structurally cannot. |
| 36 | + |
| 37 | +## Where the repo stands |
| 38 | + |
| 39 | +There is **one protocol engine**, the async-native `AsyncGateway` |
| 40 | +(`_trio_gateway.py`), and everything else is a surface over it. The wire |
| 41 | +protocol (`Message` framing) is unchanged from 2.1. |
| 42 | + |
| 43 | +**No source is shipped over the wire, ever.** Workers are launched as |
| 44 | +`execnet worker <transport> <config>`; foreign and remote interpreters are |
| 45 | +uv-provisioned; a dev coordinator builds and ships a wheel. Version skew |
| 46 | +gets a rough major/minor check (`_trio_worker._check_version`). |
| 47 | + |
| 48 | +### Four namespaces, one per concurrency library you drive execnet from |
| 49 | + |
| 50 | +| namespace | what it is | |
| 51 | +|---|---| |
| 52 | +| `execnet` / `execnet.sync` | the blocking API, a facade over the engine; top level aliases into `sync` | |
| 53 | +| `execnet.trio` | trio-native `AsyncGroup`/`AsyncGateway`/`AsyncChannel`, awaited in your own `trio.run` | |
| 54 | +| `execnet.aio` | the same surface for asyncio, bridged per call onto the host loop | |
| 55 | +| `execnet.gevent` | the sync surface with gevent-parking waits | |
| 56 | + |
| 57 | +`trio`/`aio`/`gevent` load lazily via module `__getattr__`, so |
| 58 | +`import execnet` does not import an event loop (pinned by |
| 59 | +`testing/test_namespaces.py`). |
| 60 | + |
| 61 | +A `Host` is one thread running one Trio loop; there is **one shared host |
| 62 | +per process**, `Group(host=...)` to override. Blocking calls made from |
| 63 | +inside a running event loop raise and name `execnet.aio` / `execnet.trio` |
| 64 | +— worker-side channels are exempt, since exec'd code may run its own loop. |
| 65 | + |
| 66 | +### The CLI is the launch contract |
| 67 | + |
| 68 | +``` |
| 69 | +execnet worker --protocol-stdio | --protocol-fd FD[,FD] |
| 70 | + | --protocol-connect ADDR | --protocol-listen ADDR |
| 71 | + | --protocol-share |
| 72 | + --config JSON | --config-fd FD | --config-file PATH |
| 73 | + --stdin/--stdout/--stderr DISPOSITION |
| 74 | +execnet server [HOST:PORT] [--once] |
| 75 | +execnet info |
| 76 | +``` |
| 77 | + |
| 78 | +`ADDR` is `unix:/path` or `host:port`. Everything that starts a worker |
| 79 | +emits these tokens; there is no second launch path. `execnet info` |
| 80 | +answers JSON (version, trio, executable, platform, protocols) so |
| 81 | +provisioning learns a remote's version *before* connecting. |
| 82 | + |
| 83 | +`transport=socket|stdio` is a spec key; **`socket` is the default for |
| 84 | +every worker execnet spawns**, which is why a worker's stdio is free for |
| 85 | +the code it runs. |
| 86 | + |
| 87 | +| gateway | handoff | |
| 88 | +|---|---| |
| 89 | +| popen, POSIX | `pass_fds` + `--protocol-fd` (socketpair) | |
| 90 | +| popen, Windows | `socket.share(pid)` + `--protocol-share`, blob in the config | |
| 91 | +| `socket=` / `installvia=` | the same two, server-side | |
| 92 | +| `ssh=` / `vagrant_ssh=` | `ssh -R` unix socket, worker dials back (`--protocol-connect`) | |
| 93 | +| `via=` | the sub's stdio, relayed over the coordinator's protocol | |
| 94 | + |
| 95 | +`--protocol-listen` has no user today; it is what a trampoline or a |
| 96 | +port-forwarded worker would use (see the Kubernetes section of the |
| 97 | +roadmap). ssh on Windows stays on stdio and cannot do otherwise: CPython |
| 98 | +has never exposed `AF_UNIX` there (cpython#77589) and Win32-OpenSSH has no |
| 99 | +`StreamLocal` forwarding. `resolve_transport` raises for an impossible |
| 100 | +request rather than letting a gateway hang. |
| 101 | + |
| 102 | +### Worker profiles (`profile=`, spelled `execmodel=` before 3.0) |
| 103 | + |
| 104 | +| profile | loop thread | exec'd code runs | channel | extra deps | |
| 105 | +|---|---|---|---|---| |
| 106 | +| `thread` (default) | side thread | hybrid: the first `remote_exec` claims the worker's main thread, further ones overflow to pool threads | sync | — | |
| 107 | +| `trio` | **main thread** | async sources as tasks, one thread total; sync sources rejected | `AsyncChannel` | — | |
| 108 | +| `gevent` | side thread | a greenlet per `remote_exec` on a main-thread hub | sync | `execnet[gevent]`, auto-added by uv provisioning | |
| 109 | +| ~~`main_thread_only`~~ | deprecated alias for `thread` | | | | |
| 110 | + |
| 111 | +`TrioWorkerExec` is a pure FIFO admission pump delegating to strategy |
| 112 | +objects (`WORKER_EXEC_STRATEGIES`); subinterpreters are a future strategy |
| 113 | +slot, not built. `AsyncGroup.makegateway` defaults workers to `thread` — |
| 114 | +the coordinator's shape does not dictate the worker's. |
| 115 | + |
| 116 | +### File map (src/execnet/) |
| 117 | + |
| 118 | +| file | role | |
| 119 | +|---|---| |
| 120 | +| `_message.py` / `_serialize.py` | wire protocol + sans-IO `FrameDecoder`; serializer (CHANNEL opcode incl. duck-typed `save_AsyncChannel`) | |
| 121 | +| `_channel.py` / `_gateway_base.py` / `_errors.py` | sync `Channel`/`ChannelFactory`; `BaseGateway`/`WorkerGateway`; error types | |
| 122 | +| `_trio_gateway.py` | **the engine**: `ByteStream` Protocol, `RawChannel`/`AsyncChannel`, `AsyncGateway` (outbound queue of `(frame, on_written)`, `_finalize` hook), `AsyncGroup` (all transports, reapers, bounded terminate), `ThreadedFdStream`, stream/argv helpers | |
| 123 | +| `_trio_host.py` | `Host`'s loop thread, `SyncBridgeGateway`, `FacadeAsyncGroup`, `SyncIOHandle`, `RawTunnelStream`, the `GATEWAY_START_*` handlers | |
| 124 | +| `_trio_worker.py` | worker entry, `TrioWorkerExec` + exec strategies, `_prepare_protocol_fds`, `_check_version` | |
| 125 | +| `_boundary.py` / `_portal.py` | the (private) boundary kit: `Wakener`/`Mailbox`/`OneShot`/`Flag`, `LoopPortal` | |
| 126 | +| `_host.py` / `_gateway.py` / `_multi.py` | shared `Host`; sync `Gateway`; sync `Group` + `MultiChannel` | |
| 127 | +| `sync.py` / `trio.py` / `aio.py` / `gevent.py` | the four public namespaces | |
| 128 | +| `_cli.py` / `_socketserver.py` / `_provision.py` | the CLI, `execnet server`, uv provisioning + argv builders | |
| 129 | +| `_execmodel.py` | `WORKER_PROFILES`, `resolve_profile`, and the deprecated `ExecModel` xdist shim | |
| 130 | +| `_rsync.py` / `_rsync_remote.py` / `_xspec.py` / `_exec_source.py` | rsync, spec parsing, remote_exec source normalization | |
| 131 | +| `_shim.py` + `gateway*.py`, `multi.py`, `rsync*.py`, `xspec.py` | the deprecated pre-Trio module names, warning and forwarding | |
| 132 | + |
| 133 | +## Invariants — do not regress |
| 134 | + |
| 135 | +**Protocol and lifecycle** |
| 136 | + |
| 137 | +- Sends from non-loop threads block until the frame hit the OS write (120s |
| 138 | + → `OSError`), so an abrupt `os._exit` cannot drop "sent" data; loop-thread |
| 139 | + sends only enqueue. All sends go through one portal-posted FIFO. |
| 140 | +- After close: `OSError("cannot send (already closed?)")`; |
| 141 | + `trio.RunFinishedError` maps to the same. |
| 142 | +- exec admission order == message arrival order (`TrioWorkerExec._pump`). |
| 143 | + Trio shuffles its run batch, so never rely on task-spawn order. |
| 144 | +- Channel callbacks run in a threadpool thread driven by a per-channel |
| 145 | + consumer *task*: per-channel order is strict, a slow callback does not |
| 146 | + block the reader, and `waitclose()` still returns only after every |
| 147 | + callback including the endmarker has run. |
| 148 | +- `Group.terminate(timeout)` never hangs (~2×timeout bound, issues |
| 149 | + #43/#221). |
| 150 | +- Sync blocking waits (send-ack, receive, waitclose, join) stay on |
| 151 | + `threading.Event`/queue so KeyboardInterrupt can interrupt them; |
| 152 | + `portal.run` (KI-deferred) is only for management ops. |
| 153 | +- A killed worker is `EOFError` on every transport — a dead peer *resets* |
| 154 | + a socket where a pipe reaches EOF, and the reader maps that. |
| 155 | + |
| 156 | +**Launch and provisioning** |
| 157 | + |
| 158 | +- No source shipping. Workers import an installed execnet + trio. |
| 159 | +- The worker config never travels in a remote argv — it carries `env:` |
| 160 | + values, and `ps` is world-readable. ssh uses `--config-fd 0`. |
| 161 | +- **Hand a socket over as a socket, never as an fd.** Rebuilding one with |
| 162 | + `socket.socket(fileno=fd)` re-derives family/type/proto by querying the |
| 163 | + handle, which PyPy on Windows fails with `WinError 10014`. |
| 164 | +- Filling in a *missing* spec value is idempotent and fine; rewriting one |
| 165 | + the caller set is not. xdist reuses one spec object and re-reads it. |
| 166 | +- Worker teardown ends in `os._exit(0)` because trio's `to_thread` cache |
| 167 | + uses non-daemon threads. |
| 168 | +- `import execnet` must not import the trio event loop. |
| 169 | +- Keep engine idioms portable — neutral `ByteStream`, sans-IO |
| 170 | + `FrameDecoder`. See "What pins us to Trio" in `ROADMAP-3.0.md`. |
| 171 | + |
| 172 | +**Failure modes that each cost a debugging round** |
| 173 | + |
| 174 | +- A socket worker that cannot be spawned must not hang the coordinator. |
| 175 | + It is spawned by the *server*, so the exception dies there while the |
| 176 | + coordinator waits for a handshake byte. A host that cannot hand a |
| 177 | + socket over refuses *before replying with an address* — the last moment |
| 178 | + a reason can reach the coordinator — and a spawn that fails anyway |
| 179 | + closes the connection so the wait ends. |
| 180 | +- A failed socket gateway must not kill the gateway it was requested |
| 181 | + through. It runs as a task on that coordinator's host; letting it |
| 182 | + propagate cost the coordinator too, which is how one unsupported |
| 183 | + gateway became 51 errors. |
| 184 | +- `_check_event_loop` runs *before* the channel-state check in |
| 185 | + `send`/`receive`. Both are caller bugs, but which one you were told |
| 186 | + about used to depend on whether the peer had closed yet. |
| 187 | +- Anything that warns in a *worker* can livelock a pytest run: a warning |
| 188 | + raised inside pytest's warning-recording hook records a warning. The |
| 189 | + `execnet.dumps` shim warns once per process for exactly this reason. |
0 commit comments