Reference code for both sides of a DoubleZero edge multicast feed: libraries for publishing one, reference implementations for subscribing to one.
DoubleZero delivers data as GRE-encapsulated UDP multicast. The kernel handles GRE de-encapsulation, so applications see clean UDP.
| Where | ||
|---|---|---|
| Publishing a feed | Rust crates a venue publisher is built from | rust/codec, rust/publisher |
| Subscribing to market data | Parsers, book-builders, demo stack | go/, demo/ |
| Subscribing to Solana shreds | Receive-only receivers, kernel sockets and XDP | rust/, go/ |
| Recording a feed | Keeps the bytes that arrived, with the recorder's own losses inside the archive | rust/recorder |
Wire formats are specified in edge-feed-spec, which is also the authority for vocabulary here.
The feed arrives on a GRE tunnel interface (e.g. doublezero1) as UDP multicast. The DoubleZero client handles tunnel setup and heartbeat responses; the subscribing designs here are receive-only.
Physical NIC: Eth → Outer IP → GRE → Inner IP → UDP → payload
GRE interface: Inner IP (148.51.x.x → 233.84.178.1) → UDP → payload
$ ip a s doublezero1
26: doublezero1@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1476 qdisc noqueue state UNKNOWN
link/gre 64.130.37.175 peer 4.42.212.122
inet 169.254.10.233/31 scope link doublezero1
gre-decap is an optional XDP program that strips GRE inline on the physical NIC, so socket-based applications can receive the feed without a tunnel interface.
BRINGING-UP-A-FEED.md is the path in order: what a venue implements, how its publisher is built and configured, how the recorder is pointed at it, and what the infrastructure repositories own.
Crates a publisher depends on from its own repository — by git tag, which RELEASING.md explains. See rust/README.md.
| Crate | Owns |
|---|---|
dz-edge-core |
Datagram and message headers, sequencing, receive-side walk, decimal conversion |
dz-edge-tob |
Top-of-Book: Quote, Trade |
dz-edge-refdata |
Reference data: InstrumentDefinition, ManifestSummary |
dz-edge-mbp |
Market-by-Price, the first depth feed: LevelUpdate, BookClear, and the three snapshot messages |
dz-publisher-metrics |
The normative dz_publisher_* Prometheus set |
Each feed has a parser that subscribes, decodes and republishes records on a Unix socket, and a book-builder that maintains book state and persists to ClickHouse.
| Feed | Spec | Parser | Book-builder |
|---|---|---|---|
| Top-of-Book & Trades | spec | topofbook-parser | topofbook-bot |
| Market-by-Order | spec | marketbyorder-parser | marketbyorder-bot |
| Market-by-Price | spec | marketbyprice-parser | marketbyprice-bot |
| Component | |
|---|---|
| topofbook-parser | Decodes datagrams, writes JSON/CSV to a file or Unix socket |
| topofbook-bot | Per-symbol top-of-book state as Prometheus metrics, optional ClickHouse writes |
| marketbyorder-parser | Three-port subscriber and wire decoder, fans decoded JSONL out on a Unix socket |
| marketbyorder-bot | Order-keyed books; per-event rows and top-N snapshots to ClickHouse |
| marketbyprice-parser | Three-port subscriber and wire decoder, fans decoded JSONL out on a Unix socket |
| marketbyprice-bot | Price-keyed L2 books across channels; per-event rows, snapshot levels and top-N snapshots to ClickHouse |
demo/ runs all of it — parsers, book-builders, ClickHouse, Grafana — with one command and pre-provisioned dashboards for all three feeds.
Two packet types on the shred feed: shred packets on port 7733 (~1247–1272 bytes), and 4-byte DoubleZero liveness probes on port 5765.
| Language | Kernel sockets | XDP |
|---|---|---|
| Rust | rust/kernel-receiver | rust/xdp-receiver |
| Go | go/kernel-receiver | go/xdp-receiver |
| C | planned | planned |
Kernel sockets are the simple path, XDP the fast one.
A recorder keeps the bytes of a feed and its own losses, so that did the publisher send what the spec says it must, and did it arrive? can be answered after the fact — hours later, or against a rule that did not exist when the traffic passed. Without the bytes there is nothing left to ask; without the recorder's own losses, every gap it caused is charged to the publisher.
It decodes nothing on the record path. A decoder rejects the datagram most worth keeping, so a datagram whose bytes we cannot explain is archived anyway. See rust/recorder/README.md.
| Crate | Owns |
|---|---|
dz-recorder-core |
The types every other crate speaks, the Source/Sink/Observer traits, and the configuration |
dz-recorder-capture |
Live capture: membership, kernel receive timestamps, drop accounting, rejoin |
dz-recorder-archive |
The pcapng writer: rotation, compression, hashing, manifest, staging watermark |
dz-recorder-replay |
An archive read back as a Source, plus the synthetic publisher the tests use |
dz-recorder-health |
The header-only observer and the normative dz_recorder_* Prometheus set |
dz-recorder-loss |
Sequence loss over any Source, live or replayed, under one rule set |
dz-recorder-e2e |
The chain end to end, and the specification's own rule set applied to it |
dz-recorder |
The binary that wires capture, archive and health together |
Two capture modes sit behind one Source. AF_PACKET is the default: it
records what the network delivered, so a datagram the recorder's own socket
would have lost to receive-queue overflow is still in the archive and correctly
attributed. Socket mode is the fallback where CAP_NET_RAW is unavailable; it
synthesises the IP and UDP headers and records that it did, so no reader
mistakes a synthesised field for a captured one.
| Path | |
|---|---|
rust/codec, rust/publisher, rust/recorder |
Publisher, codec and recorder crates, one Cargo workspace (README) |
rust/kernel-receiver, rust/xdp-receiver |
Shred receivers, excluded from that workspace |
go/ |
Parsers, book-builders, Go shred receivers |
gre-decap/ |
XDP GRE decapsulator |
demo/ |
Docker Compose stack and Grafana dashboards |
testdata/golden/ |
Wire vectors the Go and Rust decoders both assert against |
docs/ |
Design documents and plans (index) |
pcaps/ |
Sample captures; read with Wireshark's solana.shreds dissector on UDP 7733 |
Prose follows GLOSSARY.md: a datagram is one UDP payload, a header plus N messages; a channel is a shard named by Channel ID; a parser decodes and republishes; a book-builder consumes parser output.
The -bot paths and the dz_bot_*, dz_mbo_bot_* and dz_mbp_bot_* metric names predate that glossary and are kept because CI jobs, binaries and dashboards reference them. Frame Length is the spec's own field name at offset 22, so parse_errors_total{reason="frame_length"} matches it deliberately.
A match arm is a branch here, and that is not what Rust calls it. The language reference says arm, so the word a contributor reaches for is the one the glossary bans outright — in every sense, with only ARM64 and the ARM vendor surviving. The glossary names this exact case in its replacement list: "a match or select! branch is a branch". It is repeated here because the next person to touch a match reads this file and not that one, and a rule nobody finds is a rule that drifts back one site at a time.
Apache 2.0. See LICENSE.