[QEC] Decoding server: serve HOST_CALL requests inline; retire the worker threads, queues, and CqrTransceiver - #769
Merged
cketcham2333 merged 4 commits intoAug 7, 2026
Conversation
…rker threads, queues, and CqrTransceiver The host request path used to be: shim -> CqrTransceiver::inject (frame copy, process-unique token, pending_ registration) -> RpcDispatcher re-parse/re-route -> WorkItem copy -> bounded queue (depth 64) -> per-decoder worker thread -> decoder -> transport.send -> pending_ lookup -> memcpy tx slot -> promise wakeup of the parked dispatcher. Two thread crossings and ~5 buffers per get_corrections. Now dispatch_rpc resolves the DecodingSession by the payload's decoder_id (the HOST_CALL ABI carries no context pointer, and a shared ring may serve several decoders) and calls its handle_* method, which parses the rx slot in place, runs the decoder on the calling CUDAQ dispatcher thread, and writes the RPCResponse straight into the tx slot (magic release-stored last). Zero thread crossings, zero copies, zero steady-state allocations. Same shape as the GPU device-graph dispatch path and the legacy direct path (host::enqueue_syndromes), whose validation ladder, capture hook, and pin helper handle_enqueue now shares line for line. New/changed: - RpcSlot.h: single home for in-place slot parsing and in-place response writing (EnqueueView/parse_enqueue moved from CqrTransceiver.h; GetCorrections/Reset views, peek_decoder_id, write_response, ResultWriter). Deliberately device-portable (no allocation/exceptions/STL) so the GPU dispatch kernel can later compile the same source — one point of truth for the wire layout. - DecodingSession: payload-level cores (enqueue_core, get_corrections_core, reset_core) plus the inline handle_* entry points; get_corrections packs corrections directly into the tx slot, and a result larger than the slot fails with INTERNAL_ERROR (same no-truncation rule as the old CqrTransceiver::send). - The enqueue response moves to the handler tail (single write point). Client-visible timing is unchanged: the CUDAQ dispatcher publishes the tx flag only when the handler returns. A volume-completing enqueue runs its decode inline and blocks only this session's ring — decoder parallelism comes from per-decoder rings (one dispatcher thread each); a shared ring still routes correctly but serializes. - Deferred-error contract unchanged: enqueue failures latch shot_state=failed, respond OK, and surface as INTERNAL_ERROR at the next get_corrections. - Busy accounting moves into a BusyScope RAII in the handlers; cudaqx_qec_decoding_server_max_concurrent keeps its name and now measures sessions concurrently executing on dispatcher threads. - Single-caller tripwire: a debug-only scoped in-flight guard asserts that no two handlers execute on one session concurrently — catching the invalid two-rings-one-decoder wiring loudly while tolerating legal serial dispatcher-thread replacement. It compiles out entirely in release — invalid-config guards must not add hot-path latency. Enforced (always-on) protection is deliberately NOT added: the HOST_CALL ABI carries no ring identity to validate against, and two senders feeding one decoder is semantically broken (interleaved rounds decode wrong) regardless of memory safety — real multi-sender serialization is the multi-ring decoding-unit design (fix C). - pin_decode_device_cached (hardware_guards.h): thread_local-cached device pin (one compare per call steady-state; debug re-verify), and CUDA pin failures surface at bring-up via a CudaDeviceGuard probe in DecodingSession::create (was: the worker pin handshake). - The shim's duplicated --save_syndrome capture is deleted; capture now uses the same hooks as the legacy path, byte-identical output. Deleted (~1,500 LOC) — everything the inline path made unreachable; every layer kept "just in case" is a layer customers must read past: - CqrTransceiver.h: the singleton transceiver (inject, tokens, pending_ map, per-call promises, inbox deque, frame builders, write_ack). Correlation state is unnecessary when each call writes its own tx slot synchronously. - RpcDispatcher.h/.cpp + ResponseWriter: the second-stage re-parse/ re-route of a call the CUDAQ function table already routed. - DecodingSession worker machinery: WorkItem, bounded queue, worker thread, try_enqueue, worker_loop, send_response, and the syndromes_dropped latch — there is no queue to overflow; backpressure is the transport ring. SYNDROMES_DROPPED/BUSY stay reserved in the wire enum but are no longer emitted. - RoundAccumulator + SyndromeMappingTable: unreachable fragment-merge machinery (only the hardcoded identity table ever existed; vp_id was always 0). The syndrome_mapping_id wire field stays: id 0 accepted, anything else rejected with BAD_REQUEST instead of silently decoding as identity; non-identity mappings re-enter via the identity-aware decoder API (fix B) when they become configurable. - SpinPolicy.h + QEC_DECODING_SERVER_SPIN_US: both spin-then-block sites died with the cross-thread waits they optimized. - LoopbackTransceiver, the DecodingServer recv loop, and install_dispatch_sink: no implementors/users remain. DecodingServer shrinks to the device_graph lifecycle shell; run() parks until stop(). Instrumentation: HopStats.h drops the cross-thread hop machinery and becomes a stack-resident StageScope recording parse/decode/respond stages per request. QEC_DECODING_SERVER_HOP_STATS and the `total` metric keep their semantics (handler entry -> exit) so reports remain comparable; the volume-completing enqueue now shows its decode in stage_decode (previously hidden between the enqueue ACK and the get_corrections wait). QEC_PIN_RECV/QEC_PIN_WORKER die with their threads. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
cketcham2333
marked this pull request as ready for review
August 6, 2026 14:33
Collaborator
|
Thanks Chuck Overall, this is a strong simplification of the host dispatch path. I left one small question about documenting the current one-ring-per-decoder ownership assumption as multi-ring support evolves. |
melody-ren
reviewed
Aug 7, 2026
melody-ren
left a comment
Collaborator
There was a problem hiding this comment.
Thanks Chuck! Looks good to me. Just some minor nits
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
melody-ren
approved these changes
Aug 7, 2026
cketcham2333
enabled auto-merge (squash)
August 7, 2026 21:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Retires the cross-thread request path in the QEC decoding server ("fix A" of the
decoding-server redesign). Every host RPC previously crossed two server threads:
shim → CqrTransceiver::inject (frame copy, token, pending map) → RpcDispatcher
re-parse/re-route → WorkItem copy → bounded queue → per-decoder worker thread →
decoder → transport.send → pending lookup → tx-slot memcpy → promise wakeup of
the parked dispatcher — two thread crossings and ~5 buffers per get_corrections.
Now the shim resolves the
DecodingSessionby the payload'sdecoder_idandcalls its
handle_*method inline on the CUDAQ dispatcher thread: the rxslot is parsed in place, the decoder runs on the calling thread, and the
response is written straight into the tx slot (magic release-stored last).
Zero thread crossings, zero copies, zero steady-state allocations — the same
shape as the GPU device-graph dispatch path and the legacy direct path
(
host::enqueue_syndromes), whose validation ladder, capture hook, and pinhelper
handle_enqueuenow shares.Net: +960 / −2,702 lines.
Measured latency (udp two-process rig, d3 surface code, pymatching, 500 shots)
(
get_correctionsno longer contains the decode wait — the decode runs inlinein the volume-completing enqueue, whose tail carries the actual decode time.)
What's new
writing; deliberately device-portable (no allocation/exceptions/STL) so the
GPU dispatch kernel can later compile the same source.
handle_*entrypoints; corrections pack directly into the tx slot (no truncation: an
oversized result fails with INTERNAL_ERROR).
invalid two-rings-one-decoder topology loudly in debug builds and compiles
out entirely in release (invalid-config guards must not add hot-path
latency). Always-on enforcement is deliberately omitted: the HOST_CALL ABI
carries no ring identity to validate against, and multi-sender serialization
belongs to the multi-ring decoding-unit design.
surface at server bring-up via a create-time probe.
QEC_DECODING_SERVER_HOP_STATSand thetotalmetric keep their semantics.What's deleted (~1,500 LOC)
CqrTransceiver.h(singleton transceiver: inject, tokens, pending map,promises, inbox, frame builders)
RpcDispatcher+ResponseWriter(second-stage re-parse/re-route)DecodingSessionworker machinery (WorkItem, bounded queue, worker thread,syndromes_droppedlatch — backpressure is the transport ring; theSYNDROMES_DROPPED/BUSY codes stay reserved on the wire but are no longer
emitted)
RoundAccumulator+SyndromeMappingTable(unreachable fragment-mergemachinery; the
syndrome_mapping_idwire field stays — id 0 accepted,others rejected with BAD_REQUEST)
SpinPolicy.h+QEC_DECODING_SERVER_SPIN_US(the cross-thread waits theyoptimized no longer exist)
LoopbackTransceiver, the recv loop,install_dispatch_sinkBehavior notes
unchanged (the tx flag publishes on handler return). The deferred-error
contract is unchanged.
ring: decoder parallelism comes from per-decoder rings (one dispatcher
thread each). A shared ring still routes multiple decoders correctly but
serializes their decodes.
surface_code-4-yaml-test.shexports theper-decoder ring ports (all tokens required, fail-fast), and
hard-patch-pymatchingruns 500 shots so its opportunistic overlap evidenceis statistically certain (the deterministic proof is the three-way-overlap
barrier test).
run_realtime_decoding.sh: fixes a hololink→hsb rename leftover(
--hololink→--hsb-ip) needed for--source fpgaruns.