POC: Fluentd exporter for observability of the raft tracing

[cjen1-msft]

This is a proof of concept for the fluentd observability sink.
I've implemented it using the raft tracing as a benchmark (as getting that data out is a major outcome of this work).

Architecture

sequenceDiagram
    participant H as Host thread
    participant E as Enclave thread (each)
    participant Q as Per-thread SPSC queue
    participant L as Logger consumer thread
    participant F as Fluentd

    H->>H: Configure exporter
    H->>Q: Preallocate one queue per producer
    H->>L: Start consumer
    activate L
    L->>F: Attempt TCP connection
    H->>H: Bind host's queue
    H->>E: Start enclave threads
    activate E
    E->>E: Bind assigned queue to thread-local pointer

    Note over E,Q: Queues are statically allocated by host

    loop Emit events
        E->>E: Serialize event to MessagePack
        E->>Q: Enqueue owned record
        activate Q
        alt Slot available
            Q-->>E: Accepted
        else Queue full
            Q-->>E: Rejected - drop
        end
        deactivate Q
        Note over E: Continue without waiting
    end

    loop Drain queues independently
        L->>Q: Read next record
        activate Q
        Q-->>L: Record bytes, retained until send finishes
        L->>F: Send bytes
        L->>Q: Destroy record and release slot
        deactivate Q
        L->>L: Report drop thresholds
        opt No work available
            L->>L: Sleep 1 ms
        end
    end

    H->>E: Stop and join producers
    deactivate E
    H->>L: Request shutdown
    L->>Q: Drain for up to 2 seconds
    L->>L: Count remaining records as dropped
    L->>F: Close TCP connection
    L-->>H: Consumer joined
    deactivate L

Loading

Benchmarking

This is using the blocking basic benchmark, two nodes, 1k clients, 4k slots in the ringbuffer, and a RAM disk as this was running on an ACI machine.
The specific results are the median throughput run for each.

100ms signatures

┌──────────┬───────────────────┬──────────────┬────────┬──────────────────────┬───────────────┬─────────────┬───────────────┐
│ Export   │ Throughput (tx/s) │ Mean latency │    p99 │ Mean CPU, both nodes │ Trace records │ Trace bytes │ Total elapsed │
├──────────┼───────────────────┼──────────────┼────────┼──────────────────────┼───────────────┼─────────────┼───────────────┤
│ Disabled │             7,454 │    114.00 ms │ 310 ms │              149.85% │             0 │           0 │       37.17 s │
├──────────┼───────────────────┼──────────────┼────────┼──────────────────────┼───────────────┼─────────────┼───────────────┤
│ Enabled  │             7,312 │    114.57 ms │ 190 ms │              153.18% │       470,018 │ 181,254,399 │       37.03 s │
└──────────┴───────────────────┴──────────────┴────────┴──────────────────────┴───────────────┴─────────────┴───────────────┘

10ms signatures

┌──────────┬───────────────────┬──────────────┬─────────────┬──────────────────────┬───────────────┬────────────┬───────────────┐
│ Export   │ Throughput (tx/s) │ Mean latency │   p50 / p99 │ Mean CPU, both nodes │ Trace records │ Trace data │ Total elapsed │
├──────────┼───────────────────┼──────────────┼─────────────┼──────────────────────┼───────────────┼────────────┼───────────────┤
│ Disabled │            10,858 │     75.69 ms │ 63 / 100 ms │              222.44% │             0 │          0 │       37.07 s │
├──────────┼───────────────────┼──────────────┼─────────────┼──────────────────────┼───────────────┼────────────┼───────────────┤
│ Enabled  │            10,270 │     79.20 ms │ 67 / 170 ms │              225.01% │       669,537 │ 250.29 MiB │       37.17 s │
└──────────┴───────────────────┴──────────────┴─────────────┴──────────────────────┴───────────────┴────────────┴───────────────┘

Open questions

• How to report drops
  • Current implementation logs on power of two of drops.
• Static or dynamic observability queues
  • POC is static, as this simplifies it greatly
  • Dynamic is somewhat complex to avoid producer side logging issues.
• How much to trace
  • We currently write a decent portion of the raft state on each emission, but can probably reduce that quite a bit safely.