[Feature] Store BanyanDB's own logs in BanyanDB (self-stored logs)

[tristaZero]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

1. Feature introduction

BanyanDB self-stores its metrics, but not its logs.

	Metrics	Logs
Destinations	2 — Prometheus + self-storage	1 — console only
Abstraction	meter.Provider (pluggable)	a raw io.Writer
Fan-out	✅ factory → every provider	❌
Lifecycle service	✅ metricService	❌
Stored in BanyanDB	✅ _monitoring group	❌
Survives pod restart	✅	only if something else collected it

The entire log output story is one line, fixed at logger.Init() time:

// pkg/logger/setting.go
if development { w = zerolog.ConsoleWriter{Out: os.Stderr, …} } else { w = os.Stderr }
ctx := zerolog.New(w).Level(lvl).With().Timestamp()

Proposal: give logs the same two-destination model metrics already have — console and self-storage — reusing the native-metrics machinery wherever it already solves the problem.

Non-goals: replacing console output; ingesting non-BanyanDB logs; changing the module tree, level filtering, or existing flags.

2. Deployment architecture

flowchart TB
    C["clients (OAP, bydbctl)"] --> LB["gRPC load balancer :17912"]

    subgraph LT["Liaison tier — stateless, NO disk"]
        L0["liaison-0<br/>:17912 client · :18912 peer<br/>+ FODC agent"]
        L1["liaison-1<br/>+ FODC agent"]
    end

    subgraph DT["Data tier — OWNS THE DISK"]
        DH["data-hot-0 :17912<br/>+ FODC agent<br/>+ lifecycle sidecar<br/>+ backup sidecar<br/>+ restore init container<br/>PVC: measure/stream/trace/property"]
        DW["data-warm-0"]
        DC["data-cold-0"]
    end

    FP["fodc-proxy · 1 per cluster<br/>:17913 /metrics · /cluster/topology"]

    LB --> L0 & L1
    L0 & L1 -- "write / query :17912" --> DH
    DH -- "hot→warm→cold" --> DW --> DC
    L0 & L1 -. "gRPC register" .-> FP
    DH & DW & DC -. "gRPC register" .-> FP
    FP --> P["Prometheus → Grafana"]

Loading

Pairing rules

Component	Cardinality	Paired with	Form
liaison	N (≥2)	—	behind a gRPC LB, peers via :18912
data	M (≥2), hot/warm/cold	—	discovered by liaisons
FODC agent	1 : 1 per node	one liaison or one data node	sidecar
FODC proxy	1 per cluster	all agents	standalone pod
lifecycle	1 per data node	co-located data node	sidecar, 127.0.0.1:17912
backup	1 per data node	co-located data node	sidecar, shares PVC
restore	1 per data node	nothing	init container
migration	1 per cluster	all PVCs, zero live nodes	standalone pod, data tier at replicas=0

liaison and data are cluster-scoped peers. Everything else is a per-node companion glued to one node by 127.0.0.1 or a shared volume.

3. Role → function → log destination

Self-storage needs a process owning the storage engine — the component holding queryable groups on disk. Only data and standalone do. Everything else must reach one, and the hop count differs.

Role	Function	Storage engine	Log destination
standalone	all-in-one	✅	in-process — queue.Local(), own shard
data	stores & serves shards	✅	in-process — queue.Local(), own shard
liaison	routes writes/queries	❌	two tiers — liaison wqueue → part-sync → data node (ref)
lifecycle	hot→warm→cold migration	❌	one hop — pub → co-located 127.0.0.1:17912
backup	snapshots → S3/GCS/Azure	❌	one hop — pub → --grpc-addr; needs a schema-bootstrap decision
restore	remote backup → local dirs	❌	console only — init container, runs before its data node starts
migration	re-grid measure/stream data	❌	console only — runs with the data tier at replicas=0

Two corrections to a common mental model:

• A liaison is not diskless. It has a write queue at --measure-data-path / --stream-data-path (banyand/measure/wqueue.go) where it buffers parts before syncing them out. It lacks the storage engine, so it can hold a log batch in flight but can never be where logs are read back from.
• lifecycle and backup never go through a liaison. pub.NewWithoutMetadata(nil) defaults to ROLE_DATA, so they publish straight to their co-located data node — a strictly shorter path than the liaison's.

restore and migration are not gaps to fill later. A tool that runs while the database is down cannot log into that database — and for migration, a self-storing sink would violate its own precondition that nothing else writes to the target paths.

4. Proposed approach

4.1 The seam — one event, two writers

zerolog always encodes an event to JSON internally, and ConsoleWriter is itself an io.Writer that re-formats that JSON. So MultiLevelWriter hands the sink the fully-encoded line — module, level, timestamp, message, all structured fields — one source, one format, one set of parameters.

flowchart LR
    A["call site<br/>logger.GetLogger(measure).Info()<br/>.Str(group, g).Msg(flushed part)"] --> B["zerolog encodes ONE JSON event"]
    B --> CW["console writer<br/>stderr · ALWAYS ON"]
    B --> SW["switchableWriter<br/>atomic.Pointer"]
    SW --> S2["logSink — ONE ring buffer<br/>alive from Init()"]
    SW --> S3["io.Discard — after GracefulStop"]

Loading

❌ zerolog.Hook rejected — hooks see level + message but not the accumulated structured fields.

The writer contract

Three constraints come from zerolog's own implementation (v1.34.0), not from BanyanDB:

zerolog internal	Constraint on the sink
Event.write() calls putEvent(e), returning e.buf to a sync.Pool	p is reused. Buffering it without append([]byte(nil), p...) yields a slice the next log line overwrites.
multiLevelWriter.Write maps _n != len(p) → io.ErrShortWrite	Always return (len(p), nil). A dropped line must still report a full write; drops surface via the counter, never the return value.
MultiLevelWriter type-switches on zerolog.LevelWriter before wrapping in LevelWriterAdapter	Implement WriteLevel(l zerolog.Level, p []byte) and zerolog hands over the level as an enum — so --logging-native-level filtering is an int compare with no JSON parsing on the hot path, and the level entity tag needs no extraction.

type switchableWriter struct{ target atomic.Pointer[zerolog.LevelWriter] }

func (s *switchableWriter) WriteLevel(l zerolog.Level, p []byte) (int, error) {
    if w := s.target.Load(); w != nil {
        (*w).WriteLevel(l, p)   // errors deliberately ignored
    }
    return len(p), nil          // always
}

atomic.Pointer rather than a mutex: WriteLevel runs on every log line from every goroutine, while the pointer is swapped once per process (at GracefulStop). A lock-free load is the right trade at that ratio.

4.2 Lifecycle — deferred activation

Same deferral pendingMeasures / native.InitSchema already use for metric schemas — but applied to the consumer, not the buffer.

Init()        install switchableWriter → logSink   (buffer live, NO consumer, no I/O)
PreRun()      register drop counters                            (no metadata)
Serve()       1. create group + stream schema (idempotent)
              2. START the consumer goroutine (§4.3)
GracefulStop  closer.CloseNotify() → consumer drains once and exits (bounded)
              swap → io.Discard · close publisher

One buffer, two phases. The buffer is allocated at Init() and never replaced; "activation" only starts draining it. There is no second buffer and no hand-off, so a goroutine that loaded the writer just before activation cannot strand its line in an abandoned buffer. Producers see one unchanging target for the whole process lifetime.

4.3 Write workflow

The consumer is a dedicated goroutine, following accesslog.startConsumer — not a timestamp.Scheduler job as FlushMetrics uses. A scheduler job is a periodic callback: it fires on the tick and can do nothing between ticks, so it supports a time trigger and nothing else. The size trigger needs something that observes every push, which only a goroutine selecting on the buffer can do.

for {
    select {
    case <-s.closer.CloseNotify():
        s.flush(batch)                      // final drain, bounded
        return
    case <-flushTicker.C:                   // TIME trigger  (--logging-native-flush-interval, 5s)
        if len(batch) > 0 { s.flush(batch); batch = batch[:0] }
    case entry := <-s.buffered():
        batch = append(batch, entry)
        if len(batch) >= s.flushSize {      // SIZE trigger  (--logging-native-flush-size, 1024)
            s.flush(batch); batch = batch[:0]
        }
    }
}

flowchart LR
    R["buffer<br/>bounded · drop on full"] --> T{"consumer goroutine<br/>flush trigger"}
    T -- "interval 5s" --> B["build InternalWriteRequest"]
    T -- "size ≥ 1024" --> B
    T -- "CloseNotify" --> B
    B --> N{"nodeSelector"}
    N -- "nil (data/standalone)" --> LOC["queue.Local() → own shard"]
    N -- "set (liaison/lifecycle)" --> LO["Locate()"]
    LO -- ok --> PUB["pub → TopicStreamWrite → data node"]
    LO -- "fail" --> DROP["count no_node, drop<br/>(do NOT publish empty nodeID)"]

Loading

4.4 Schema

Derived field by field from the native-metrics schema in pkg/meter/native/provider.go. Each row names the proto field, so the diff against the existing implementation is explicit.

Running example — this line, emitted on node data-hot-0:

{"level":"warn","module":"MEASURE","group":"sw_metric","time":"2026-09-11T10:23:45.123Z","message":"flush took longer than expected"}

Group — common.v1.Group

Schema field	Metrics (_monitoring)	Logs (_monitoring_log)	Example value
metadata.name	_monitoring	_monitoring_log	"_monitoring_log"
catalog	CATALOG_MEASURE	CATALOG_STREAM	Catalog_CATALOG_STREAM (= 1)
resource_opts.shard_num	1	1 (configurable)	1
resource_opts.segment_interval	{UNIT_DAY, 1}	{UNIT_DAY, 1}	&IntervalRule{Unit: UNIT_DAY, Num: 1}
resource_opts.ttl	{UNIT_DAY, 1}	{UNIT_DAY, 7} (configurable)	&IntervalRule{Unit: UNIT_DAY, Num: 7}

Resource — metrics use database.v1.Measure, logs use database.v1.Stream

Not a rename. Measure and Stream are two distinct proto messages that coexist; nothing is renamed or modified. Logs simply instantiate a different existing type. "Resource" is BanyanDB's own umbrella term for what a group holds — docs/concept/data-model.md: "A group's catalog fixes which one kind of resource it holds (MEASURE, STREAM, TRACE, or PROPERTY)." There is no Resource type in code.

Schema field	Metrics (Measure)	Logs (Stream)	Example value
metadata.name	one Measure per metric name	one Stream, log	metrics: "total_written" · logs: "log"
tag_families[].name	single default	searchable + data	"searchable", "data"
tag_families[].tags[]	node/scope/metric labels	7 searchable + 1 binary	{Name:"level", Type:TAG_TYPE_STRING}, {Name:"body", Type:TAG_TYPE_DATA_BINARY}
fields[] (FieldSpec)	value FLOAT/GORILLA/ZSTD	field does not exist	— (no such field on Stream)
entity.tag_names	all 4 node tags + every label	[node_id, module, level]	[]string{"node_id","module","level"}

Write payload

Schema field	Metrics	Logs	Example value (for the line above)
request / value type	measure.v1.InternalWriteRequest / DataPointValue	stream.v1.InternalWriteRequest / ElementValue	—
…element_id	no counterpart	<node_id>-<process_epoch>-<seq>	"data-hot-0-1757585021-42"
…timestamp	time.Now().Truncate(time.Second) at flush	the event's own time	2026-09-11T10:23:45.123Z
…tag_families[0] (searchable)	{Tags: labelValues}	7 tag values	["data","data-hot-0","MEASURE","warn","10.1.2.3:17912","10.1.2.3:17913","flush took longer than expected"]
…tag_families[1] (data)	—	the raw line	[]byte("{\"level\":\"warn\",\"module\":\"MEASURE\",…}")
…fields	one FieldValue_Float	absent	—
entity_values	label values	must match entity.tag_names order	["data-hot-0","MEASURE","warn"]
topic	data.TopicMeasureWrite	data.TopicStreamWrite	data.TopicStreamWrite

tag_families:
  searchable: node_type, node_id, module, level, grpc_address, http_address, message
  data:       body  TAG_TYPE_DATA_BINARY   ← the complete original JSON line

• grpc_address / http_address — searchable, not in the entity. Not cardinality (both follow from node_id) but address churn: a restarted pod keeps its node_id and gets a new IP, which would open a new series and fragment that node's history.
• body holds the whole original line — no field lost to schema drift, byte-identical to the console.
• element_id = <node_id>-<process_epoch>-<seq>. timestamp = the event's time, not flush time, so lines buffered before activation land correctly on the time axis.

4.5 Two things that differ from metrics

	Metrics	Logs
Semantics	sampled state — a missed flush costs nothing, next flush carries the current value	events — a dropped line is gone. Needs bounded ring + explicit drop-oldest + drop counters
Recursion	gauge.Set() doesn't log	the write path logs. pub, sub, cluster-node-registry-* all call logger.GetLogger(...) → a naive sink feeds itself

Recursion guard, three layers: module denylist for write-path modules (primary) → sink never logs through pkg/logger, only rate-limited stderr → bounded ring caps amplification (backstop).

⚠️ The denylist is load-bearing for the liaison specifically: its two-tier path traverses the write queue and part-sync, both of which log.

Invariant across every failure mode: console output is never degraded by the sink. Self-storage is best-effort; the console is not.

4.6 Implementation phases

See this comment — delivery order, and why each phase adds exactly one new failure domain.

5. Parameters and configuration

Derived from the two existing surfaces — prefix from console logging, vocabulary from metrics:

pkg/logger/setting.go RegisterFlags       observability/services/service.go FlagSet
  --logging-env      prod                   --observability-listener-addr         :2121
  --logging-level    info                   --observability-modes          [prometheus]
  --logging-modules  nil                    --observability-metrics-interval        15s
  --logging-levels   nil                    --observability-native-flush-interval    5s
        └──── "logging-" prefix ──┐   ┌──── "modes" + "-native-" infix ────┘
                                  ▼   ▼
                           --logging-native-*

Flag	Default	Meaning	From (where the idea comes from)
--logging-modes	console	console, native, or both	metric — the modes idea, from --observability-modes ([prometheus])
--logging-native-level	warn	minimum level reaching storage	console log — --logging-level, but as an independent threshold. Metrics have no level concept
--logging-native-flush-interval	5s	time trigger	metric — --observability-native-flush-interval, same default
--logging-native-flush-size	1024	size trigger, in entries	access log — accesslog.DefaultBatchSize (100), a constant today
--logging-native-buffer-size	8192	ring capacity, in entries	access log — the validRequests channel capacity (100 sampled / 1000 not), a constant today
--logging-native-group-ttl	7d	_monitoring_log TTL	metric — provider.go hardcodes ResourceOpts.Ttl = {UNIT_DAY, 1}
--logging-native-shard-num	1	_monitoring_log shards	metric — provider.go hardcodes ResourceOpts.ShardNum = 1. Load-bearing for logs, given the single-node funnel

Deliberately not copied from metrics: --observability-listener-addr (logs are push-only — no pull endpoint to scrape) and --observability-metrics-interval (that is the Prometheus collection tick; logs have no collection phase).

Unchanged: the four existing --logging-env / -level / -modules / -levels flags. They apply upstream of both writers; altering them is an explicit non-goal. Each new flag gets the standard BYDB_* env binding, as logger.RegisterFlags already provides. restore / migration accept --logging-modes but reject native at Validate().

6. Reading logs back

No new query surface — _monitoring_log is an ordinary stream group:

bydbctl stream query -f - <<EOF
name: "log"
groups: ["_monitoring_log"]
timeRange: { begin: 2026-09-10T00:00:00Z, end: 2026-09-11T00:00:00Z }
criteria:
  condition: { name: "level", op: "BINARY_OP_EQ", value: { str: { value: "error" } } }
projection:
  tagFamilies:
    - { name: "searchable", tags: ["node_id", "module", "level", "message"] }
    - { name: "data", tags: ["body"] }
EOF

7. Failure modes

Each row is one stage of the write path, in path order. Derived by walking the pipeline and asking, at each stage, "what can fail here, and what happens when it does?"

producer ──► BUFFER ──► consumer ──► schema ──► Locate ──► publish ──► data node
   │          │          │           │          │           │           │
   │          │          │           │          │           │           └─ slow
   │          │          │           │          │           └─ error / timeout
   │          │          │           │          └─ no node available
   │          │          │           └─ CreateGroup / CreateStream fails
   │          │          └─ not started yet (pre-Serve)
   │          └─ full
   └─ never blocks  ← the invariant, not a failure

Every stage that can fail gets exactly one row, one behaviour, one counter label. The review test is therefore not "are these cases interesting" but "does every arrow in that diagram have a row" — a stage with no row is a stage whose failure is unhandled.

Stage	Situation	Behaviour
producer	always	never blocks the caller; console unaffected — the invariant, not a failure
buffer	consumer goroutine not yet started (pre-Serve)	accumulates; drained once Serve() starts the consumer
buffer	fills before the consumer starts	drop, count not_ready; console still has everything
buffer	full during a burst	drop, count buffer_full
schema	CreateGroup / CreateStream fails	stderr once; the ring keeps buffering (eventually dropping as not_ready); retry on the flush tick
schema	group dropped at runtime (OnDelete)	writes begin failing downstream → surfaces as publish_error. After N consecutive publish errors, re-run the idempotent InitSchema rather than failing forever
schema	exists but incompatible (hand-edited _monitoring_log)	CreateStream returns AlreadyExists, so the sink proceeds and writes fail on tag mismatch. Count schema_mismatch, log expected-vs-actual once to stderr. Never auto-migrate
Locate	no data node (liaison partition / migration window)	count no_node, drop — don't publish an empty nodeID
publish	error / timeout	count publish_error, drop that batch, continue
publish	data node disk full	also publish_error — indistinguishable from a network fault at this layer. Diagnose via the existing banyandb_system_disk metric
data node	slow	drops (whichever bound fills first); never backpressure into the caller
shutdown	GracefulStop with a full buffer	closer.CloseNotify() makes the consumer goroutine drain once and exit; that drain is time-bounded, and whatever cannot ship is counted shutdown_drop so the loss is not silent
sink itself	any internal error	direct rate-limited stderr, never through pkg/logger

Drops are exported as banyandb_logging_dropped_total{reason} through the existing metrics factory — loss is visible in the system that is reliable. Labels: not_ready, buffer_full, schema_mismatch, no_node, publish_error, shutdown_drop.

8. References

See this comment — existing log implementation, the native-metrics template, per-role wiring, and the liaison's two-tier write path.

Related issues

None found.

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[tristaZero]

Reference — the liaison's two-tier write path

Background for the liaison row in §3. A liaison write reaches disk through two independent node selections, not one. This is existing behaviour for user data and for native metrics.

flowchart TB
    S["liaison metricSvc / logSvc<br/>Locate(group, name, shardID, replicaID)"]
    T1["① a LIAISON node :18912<br/>intermediate buffer, NOT the store<br/>per-(group, shardID) write queue on its own disk"]
    T2["② a DATA node :17912<br/>via part-sync — THE ACTUAL STORE"]
    S -- "selector fed by tire1Client<br/>allowedRoles = [ROLE_LIAISON]" --> T1
    T1 -- "NewClusterNodeRegistry(TopicMeasurePartSync, tire2Client, …)<br/>allowedRoles = [ROLE_DATA]" --> T2

Loading

Streams mirror this with TopicStreamWrite / TopicStreamPartSync (banyand/stream/svc_liaison.go:207). Each selector's candidate pool is whatever its pub admits — pub.OnAddOrUpdate gates on allowedRoles (banyand/queue/pub/client.go:133) before a node reaches sel.AddNode.

How a node is chosen

roundRobinSelector.Pick is deterministic, not per-call rotation:

// pkg/node/round_robin.go:190
func (r *roundRobinSelector) Pick(group, _ string, shardID, replicaID uint32) (string, error) {
    //                                     ↑ resource name IGNORED
    i := sort.Search(len(r.lookupTable), …)              // entry for (group, shardID)
    return r.nodes[(i + int(replicaID)) % len(r.nodes)], nil
}

• lookupTable — one entry per (group, shardID), sorted by group then shard, across all groups.
• r.nodes — the sorted list of admitted node names.
• The resource name is discarded, so every measure in _monitoring resolves identically.

Consequences for a single-shard group

Exactly one (group, shardID) key exists, so tier ① always picks the same liaison and tier ② always picks the same data node — and every liaison in the cluster computes the same answer.

Funnel	N liaisons' full log volume → one liaison's wqueue → one data node. On the tier that exists to scale horizontally.
Self-loop	There is no self-exclusion in the publisher — ConnManager.OnAddOrUpdate dials every role-matching node including itself, and selfNode only stamps SenderNode on the request. So the designated liaison's own logs publish over gRPC back to itself at :18912 → sub server → writeListener → wqueue, which logs.
Drift	If that liaison restarts, r.nodes changes, i % len(nodes) moves, and a different peer becomes the collector.

All of _monitoring lives on one data node today. Tolerable for a few hundred datapoints every 5s; a different proposition for every WARN line in the cluster.

Implication for the design

FlushMetrics hardcodes Locate(group, name, uint32(0), uint32(0)). The log sink must not inherit that — it needs a varying shardID (e.g. hash(node_id) % shard_num) so --logging-native-shard-num actually spreads load. Tier ② already part-syncs with the real per-queue shardID, so the machinery exists.

docs/operation/observability/providers.md describes this as the liaison using "nodeSelector to select a data node" — that collapses the two tiers and predates the write queue. Worth correcting separately.

---

References — existing implementations

Existing log implementation

File	Role
pkg/logger/setting.go	root logger, the single hardcoded io.Writer, RegisterFlags
pkg/logger/logger.go	module tree (Named), per-module levels, zap bridge
docs/operation/observability/logging.md	current logging docs

Existing self-stored metrics (the template)

File	Role
banyand/observability/services/service.go	metricService — flags, PreRun, Serve, scheduler
banyand/observability/services/factory.go	fans one metric out to every enabled provider
banyand/observability/services/meter_native.go	nativeProviderFactory, deferred schema init
pkg/meter/native/provider.go	_monitoring group + measure schema, registerOrDefer
pkg/meter/native/collection.go	FlushMetrics — batch publish, node routing
docs/operation/observability/providers.md	the two metrics providers

Per-role wiring

pkg/cmdsetup/standalone.go · data.go · liaison.go · banyand/backup/lifecycle/lifecycle.go

[tristaZero]

Implementation phases

Delivery order for §4. The organising principle: each phase adds exactly one new failure domain, so when something breaks it is never ambiguous which layer is at fault.

Phase	Scope	Adds	Why this order
1	seam + sink + buffer + schema; standalone + data	—	queue.Local() is in-process: no gRPC, no node selection, no Locate, no discovery. Proves the LevelWriter contract, switchableWriter, buffer + drop accounting, schema deferral and the ElementValue payload with the fewest moving parts. Recursion risk is at its weakest here
2a	lifecycle	network, 1 hop	The shortest possible remote path — pub + pick-first to a co-located data node. Introduces Locate failure → degrade-to-console and the no_node / not_ready drop paths. lifecycle already has this exact wiring for metrics, so it is a copy, not an invention
2b	liaison	two-tier path, write queue, recursion	The hard one: two independent node selections, the wqueue and part-sync (both of which log), denylist completeness, and the varying-shardID requirement. Deliberately last among online roles
3	backup	bootstrap ordering	Blocked on a decision, not on code — add a metadata client, or depend on the group already existing (created by whichever data node booted first). The latter is cheaper but makes backup's logging depend on boot order. Separated so it cannot gate 1 or 2
—	restore, migration	—	Out of scope by design (§3). A tool that runs while the database is down cannot log into it. Document, don't implement

Two properties worth keeping

• Phase 1 is independently shippable. Standalone and single-data-node deployments get durable logs with none of the hard parts done — a real chunk of the user base, satisfied first.
• Phase 2 was originally one step. It was split after the two-tier write path and the liaison self-loop surfaced: bundling the easiest remote role with the hardest stopped making sense once the difference was clear.

What could reorder this

• If the module denylist proves trivially complete in practice, 2a and 2b could merge.
• If the wqueue-vs-direct routing question is settled in favour of direct-to-data, 2b loses most of its risk and could move ahead of 2a.

[Sruhvx-jpg]

Quick question on 4.2 GracefulStop:

Should swap -> io.Discard happen before draining the consumer?

If the consumer drains and exits first, any peer services logging during teardown will strand lines in the buffer with no consumer left to flush them. Swapping to io.Discard first cuts ingestion so the final drain cleanly catches everything before the cutoff.

Also happy to help implement or test this (e.g. Phase 1) if you'd like an extra pair of hands!

[hanahmily]

@Sruhvx-jpg Thanks for catching this—your concern is valid. Native log ingestion should stop before the final drain so shutdown logs are not stranded in the buffer.

@tristaZero is assigned to this issue. You’re very welcome to help test once the implementation is ready!

[hanahmily]

Native logging: configuration and routing

Proposed configuration and routing for native logging.

1. Independent configuration

Use separate flag/env namespaces, levels, and module overrides:

Destination	Flag prefix	Environment prefix	Level	Format
Normal console (stderr)	--logging-	BYDB_LOGGING_	error in production	Existing presentation settings
Native stream	--logging-native-	BYDB_LOGGING_NATIVE_	Default info	Always JSON

For example:

# Flags
--logging-env=prod --logging-level=error \
--logging-native-enabled=true --logging-native-level=info

# Equivalent environment variables
BYDB_LOGGING_ENV=prod
BYDB_LOGGING_LEVEL=error
BYDB_LOGGING_NATIVE_ENABLED=true
BYDB_LOGGING_NATIVE_LEVEL=info

--logging-native-enabled controls collection of this process's logs and defaults to
false. Console remains enabled. Keep existing normal-log flag names. Receiving and
forwarding other processes' native logs is a separate capability.
Native module overrides, buffer limits, flushing, TTL, and shard count each get matching
native flag/env pairs. An explicit flag overrides its corresponding environment value;
neither destination inherits the other's configuration. Native byte/event-size flag
names and defaults remain to be selected.

Admit locally generated events needed by either enabled sink, then filter per sink,
including module overrides and startup logging. With console=error, native=info,
and no overrides or recursion exclusions: info/warn are eligible only for native;
error for both; debug for neither. A dropped native info event has no console copy.
Native failure must not lower the console threshold.

Receivers validate incoming native requests but do not apply their own local logging
levels/module filters or echo received events to their console. Source identity and
element IDs remain unchanged through forwarding.

2. Stream write-request ring and routing

Buffer stream WriteRequest objects in a bounded, in-memory ring: one request per log
event, with the original JSON in the element's body tag. Do not buffer write-queue parts
or reuse the disk-backed queue's storage format. The consumer batches requests and adds
InternalWriteRequest routing fields where the internal write path requires them.

Native pending data stays in memory through liaison forwarding: no queue files, WAL,
or disk fallback. Destination stream persistence is expected.

• Adaptive memory limit: use the memory protector's AvailableBytes() to calculate
  bufferBudget = min(configuredMaxBytes, fraction * max(0, availableBytes - safetyReserve)).
  Recalculate as memory availability changes; the fraction and reserve need defined
  defaults and native flag/env settings. If availability is unknown (-1), use the
  configured cap, not an unlimited budget. Account for queued and in-flight requests
  with synchronized byte reservations; the protector's sampled value is not a reservation.
  When over budget, evict oldest queued requests or reject new ones; never reclaim
  in-flight memory or wait for capacity. Do not call the protector's blocking
  AcquireResource() on the logging path. Count memory-pressure drops.

Producer	Route
Standalone / data	In-process write to its own stream storage.
Liaison	Existing shard-based routing, with a native write-request ring, then dispatch to data.
Backup / lifecycle / FODC agent beside data	Colocated data node's existing gRPC endpoint, normally 127.0.0.1:17912.
FODC proxy	Liaison client endpoint, then the native forwarding path.
FODC agent beside liaison	Proposed: colocated liaison client endpoint; there is no local data node.
Restore / migration	Console only; the data tier is unavailable.

Use configured endpoints, not a new logging port. Reuse shard/node selection, not the
ordinary disk-backed queue or part-sync pipeline. The request-forwarding path still needs to be specified;
existing routing logic alone does not provide a memory-only transport.
Shard selection is deterministic, not per-batch rotation: use actual stream shard IDs.
Multiple shards permit distribution; a single shard does not. Whether a liaison's own
logs need peer forwarding remains a shard-ownership decision.

Phase 1 covers standalone/data. Add lifecycle and data-side agents with one-hop publishing,
then liaison-side agents/proxies with liaison integration. Backup requires schema
bootstrap; restore and migration remain excluded.

[tristaZero]

Thanks for all your feedback @Sruhvx-jpg @hanahmily — after going through all these inputs, I'd like to take most of them.

Two items may need our discussion:

1. --logging-modes=console,native vs --observability-modes=prometheus,native — keep the two schemes the same, or go with an independent one for logging?
2. Should we include FODC? It sits outside BanyanDB itself and looks like an optional observability module.

Details on both below, after what I'm adopting.

Accepted

#	Point	Note
①	Console error / native info — native more verbose than console	Our defaults had native ⊂ console, so the feature stored less than it printed. This was the biggest flaw in the proposal.
③	Independent namespaces, per-sink module overrides, no cross-inheritance	Adopting as specified.
④	Adaptive memory budget via the protector	Clearly better than our fixed 8192 entries, which bounded count not bytes — an 8 KB stack trace and a 200 B info line cost the same under our scheme.
⑥	Deterministic shard selection, real stream shard IDs	Confirms what we found: roundRobinSelector.Pick ignores the resource name, so with ShardNum: 1 every liaison resolves to the same node. All of _monitoring lives on one data node today.
—	@Sruhvx-jpg's GracefulStop ordering	Correct — ours drained before cutting ingestion, leaving a window the length of one network publish where lines were admitted and then silently stranded. Swapping first makes the cutoff explicit.

Two consequences of ① worth recording, because they invalidate parts of our write-up:

• The root level must be the more verbose of the two sinks. zerolog filters once at the root, upstream of every writer, so with root=error and native=info the native sink never sees an info event at all.
• "Console always has everything" is no longer true. Under native ⊃ console, a dropped native info has no console copy — a total loss. Our failure-mode section treated console as a safety net; that assumption is gone, which makes the drop counters the only evidence the data existed.

---

Discussion 1 — align with the metrics config pattern?

Metrics pattern today: a validated list, both destinations optional, at least one required.

--observability-modes=prometheus,native      {prometheus} | {native} | {both}

What we proposed: the same shape for logs.

--logging-modes=console,native               {console} | {native} | {both}

Same list, same three states, same validator — one mental model across both observability surfaces.

Your proposal uses --logging-native-enabled (bool) instead, which assumes console can never be disabled — so logs would have two states where metrics has three.

What do you think: keep the two schemes aligned, or go independent for logging?

Discussion 2 — should FODC be in scope?

Two reasons to leave rows 4 and 5 out:

1. It contradicts §1. Both rows need a liaison to accept and forward another process's logs — which §1 defers: "receiving and forwarding other processes' native logs is a separate capability."

2. FODC is optional, and the dependency is one-way.

fodc/  ──97 imports──►  banyand/ , pkg/
fodc/  ◄──0 imports───  banyand/ , pkg/

A cluster runs fine without FODC. Building liaison-side receive-and-forward into banyand/ to serve it would invert a dependency the codebase currently keeps clean.

Proposal — FODC becomes a consumer of this feature once forwarding exists, tracked separately:

Milestone	Needs new transport?
1. standalone / data — in-process	❌
2. lifecycle, backup, FODC agent beside data — one hop	❌
3. liaison — ring + tier-2 routing	ring only
4. (separate issue) forwarding → FODC proxy / agent-beside-liaison	✅

Two smaller confirmations

1. Scope of ⑤. Agreed on memory-only, no WAL, no disk fallback. On "existing routing logic alone does not provide a memory-only transport" — the gap may be smaller than that reads. Data nodes already accept TopicStreamWrite; publishing straight to a data node already happens (banyand/backup/lifecycle/steps.go:415); the liaison already holds tire2Client; shard selection already exists. The genuinely new piece looks like the byte-bounded ring with protector integration. If that matches your intent, the liaison row is cheaper than it appears.

2. "Existing shard-based routing … then dispatch to data." The liaison's existing TopicStreamWrite routing points at tier-1 (StreamLiaisonNodeRegistry over tire1Client, pkg/cmdsetup/liaison.go:102), so reusing it literally would dispatch to a peer liaison, not to data. I read the intent as reusing the selector mechanism with a new registry over tire2Client — 1 hop rather than 2. Please confirm.

The remaining details — buffering encoded bytes vs WriteRequest protos, whether to keep a bounded window for shutdown logs, where the per-sink module filter sits, and defaults for the TBD buffer flags — I'll work through in the design doc and bring back rather than take up space here.

@Sruhvx-jpg — help testing Phase 1 would be very welcome once there's something to run.

[hanahmily]

Thanks for incorporating the feedback. On the two discussion points:

1. Keep --logging-native-enabled. Native self-storage is an additional capability, independent of normal logging. Zerolog supports file and other writer outputs—not just console—although BanyanDB currently uses stderr. Preserving the normal logging path matters more than matching the metrics configuration shape; native-only logging is not a requirement here.

2. Keep FODC in scope, with phased delivery. “Receiving and forwarding other processes’ native logs is a separate capability” distinguishes forwarding from collecting the current process’s logs; it does not defer forwarding to another issue. A generic liaison-side receiver/forwarder does not invert the dependency: FODC consumes that capability, and BanyanDB need not import FODC packages.

The proposed implementation order is fine, but the final forwarding phase should remain part of this feature. Please carry the remaining implementation details into the design doc under the previously stated routing and memory-only requirements.

[tristaZero]

Both open points are settled: --logging-native-enabled stays a bool defaulting to false, and FODC stays in scope with forwarding as the final phase of this feature. The design is complete; what follows is the whole solution.

1. Feature introduction and scope

BanyanDB already self-stores its own telemetry, but only half of it. Metrics have a native path into the database; logs do not.

	Metrics today	Logs today
Destination	_monitoring group, one Measure per metric	one io.Writer, fixed at logger.Init time
Turned on by	--observability-modes=native	always on, no choice of destination
Queryable, retained	yes, with a TTL	no, stderr is the log driver's problem

During an incident the metrics are in the database and the logs are not. When a pod recycles, they are gone.

This feature adds a second, optional destination for logs: native self-storage into a new _monitoring_log stream group, behind --logging-native-enabled (bool, default false). Normal logging keeps its current behaviour — one zerolog writer, stderr today, though zerolog also supports file writers — and is never disabled.

	Roles	Rationale
In scope	standalone, data, liaison	reach stream storage directly
In scope	lifecycle, backup	colocated producers, using the receiver on their host's data node
In scope	FODC — the on-demand diagnostics collector, agents plus a proxy — final phase	a generic receiver/forwarder serves it, so BanyanDB imports nothing from FODC
Out of scope	restore, migration	they run when the data tier is unavailable

The sections that follow go roles, approach, configuration, failure modes, phases.

2. Role → function → log destination

One binary, several roles. One property decides the destination: whether the process owns a storage engine or must reach one.

Role	Function	Log destination
standalone	full server	own storage, in-process (queue.Local()) — as native metrics already do
data	shards, writes, queries	own storage, in-process (TopicStreamWrite)
liaison	front door: validate, route, batch	one hop to a data node over its tier-2 client, the pool spelled tire2Client in the code
backup, lifecycle, FODC agent beside data	snapshot, migration, diagnostics	colocated data node's cluster server, 127.0.0.1:17912
FODC proxy, FODC agent beside a liaison	diagnostics, no colocated data node	liaison gRPC port (--grpc-port, default 17912)
restore, migration	offline tools	normal logging only

A liaison has local disk: it runs a two-tier write queue whose normal StreamService/Write route ends in part files — more involved than the name suggests. What it lacks is the storage engine, so it can hold a batch in flight but can never be where logs are read back from.

Three shapes: in-process; the liaison selecting a data node by shard; and every other producer publishing into the generic receiver, subscribed on data nodes' cluster server and on the liaison.

Restore and migration use normal logging only by construction, not as a gap to close later: a tool that runs while the database is down cannot log into it. The native flags are registered only by the server, backup, lifecycle and FODC binaries.

3. Proposed approach

3.1 The writer seam

One event, two level-gated writers under zerolog's MultiLevelWriter. The zerolog level becomes an admission floor — the more verbose of the enabled sinks — and each sink then applies its own threshold, letting native run at info while normal logging stays at error.

flowchart LR
  E["log event"] --> F{"admission floor: min over enabled sinks"}
  F -->|below| X["dropped"]
  F -->|at or above| M["MultiLevelWriter"]
  M --> C["gate: --logging-level"] --> S["stderr or file"]
  M --> N["gate: --logging-native-level"] --> R["in-memory ring"]
  R --> K["consumer"] --> P["shard and node selection"] --> D["stream log"]

Loading

Worked example, normal logging at error and native at info, so the floor is info.

event	admitted	normal sink	native sink
debug	no	–	–
info / warn	yes	dropped by its own gate	stored
error	yes	printed	stored

Per-module thresholds resolve once when a logger is built, with normal level, native level and exclusion cached on it, never by parsing JSON per line. Matching is by prefix, since an override truncates the module name it stamps: --logging-modules=liaison-grpc stamps LIAISON-GRPC.STREAM-T1 as LIAISON-GRPC.

3.2 Buffer and transport

property	value
unit	one stream WriteRequest per event
payload	original JSON line kept whole, in the data:body binary tag
bounds	--logging-native-max-bytes 32mb, --logging-native-max-event-bytes 64kb (proposed defaults, section 4)
triggers	--logging-native-flush-interval 1s and --logging-native-flush-size 100
durability	none — no queue files, no WAL, no disk fallback

The consumer is a dedicated goroutine selecting over the closer, the flush ticker and the ring, so both triggers fire.

When native is enabled the ring is live from logger initialisation; service start only begins draining it, so no startup line is stranded in a buffer nobody drains.

3.3 Routing and shard selection

Shard IDs are deterministic and real, computed as the normal write path computes them — entity locator from the fetched schema, then hash modulo shard count — not the hardcoded shard 0 of native metrics. At --logging-native-shard-num=1 every event keys to one shard, funnelling the cluster onto a single data node.

Amendment to a stated requirement — maintainer sign-off needed. The issue description requires the liaison to reuse existing shard-based routing, then dispatch to data nodes. That route terminates in the liaison's disk-backed write queue and writes part files, which cannot satisfy the memory-only rule. This design keeps the selection and replaces the transport with a direct tier-2 batch publish to the selected data node, bypassing the write queue. The new native log topic serves the receiver's foreign producers, not the liaison's own route.

The receiver is deliberately thin.

Rule	Behaviour
Validation	InvalidArgument on a nil or non-millisecond timestamp, empty element_id or node_id, oversized body, out-of-enum level, oversized batch
Back-pressure	ResourceExhausted when its own ring is over budget; never blocks
Filtering	no local level, no module override — what arrives is what is stored
Echo	received events never enter its own logging path
Identity	element_id and identity tags pass through unchanged; identity must live in the element

--logging-native-enabled governs only a process's own collection: a node with it false still accepts, routes and persists forwarded writes.

3.4 Schema

object	value
group	_monitoring_log, catalog STREAM, TTL --logging-native-ttl-days (7), Replicas 0 — matching _monitoring
stream	log, one shared by every role
entity	node_id only
searchable tags	node_id, node_type, module, level, grpc_address, http_address, message, log_id
data tag	body, binary, the whole original line
index rules	none in v1

module and level are ordinary searchable tags, not entity tags. Entity tags accept only equality and set membership, so level != debug would be a hard query error with level in the entity; and module strings splice in group, measure and task names, giving unbounded series cardinality. As ordinary tags both stay filterable by equality, evaluated after the scan.

4. Parameters and configuration

Two independent namespaces; neither inherits from the other.

namespace	flags	environment	notes
normal logging	--logging-*	BYDB_LOGGING_*	unchanged; typically error in production
native stream	--logging-native-*	BYDB_LOGGING_NATIVE_*	default level info, always JSON

Every dashed flag binds to BYDB_<UPPER_SNAKE>, and the env value applies only when the flag was not set on the command line, so an explicit flag wins.

flag	default	meaning
--logging-native-enabled	false	collect this process's own logs natively
--logging-native-level	info	root level of the native sink
--logging-native-modules / --logging-native-levels	nil	per-module overrides, prefix-matched, length-checked
--logging-native-exclude-modules	built-in prefix set	modules never sent natively, breaking the write-path feedback loop; replaces, not appends
--logging-native-flush-interval	1s	longest a buffered event waits
--logging-native-flush-size	100	events that trigger a batch
--logging-native-write-timeout	5s	per-batch publish timeout
--logging-native-drain-timeout	5s	bound on the final drain at shutdown
--logging-native-max-bytes *	32mb	configured ring cap
--logging-native-max-event-bytes *	64kb	oversize events dropped whole
--logging-native-memory-fraction *	0.02	fraction of available memory in the adaptive budget
--logging-native-memory-reserve *	64mb	reserve subtracted before the fraction
--logging-native-shard-num	1	shards of _monitoring_log (open question)
--logging-native-ttl-days	7	retention
--logging-native-receiver-enabled	false	accept forwarded writes, wherever hosted; independent of collection

* Names and defaults for the four starred flags are proposals, not settled. Level semantics are in section 3.1; the four existing --logging-* flags keep their names and defaults.

With --logging-native-enabled=false — the default — behaviour is exactly as today: the ring is never allocated, no schema and no counters are created, and the cost is one atomic pointer load per event.

5. Failure modes

event -> producer -> buffer(ring) -> consumer -> schema -> selection -> publish -> destination
                                                                                       |
                      GracefulStop: stop admission, then bounded drain ----------------+

Every stage that can fail has a row, carrying one behaviour and one reason from the closed nine-value reason set on native_dropped_total.

Stage	Situation	Behaviour	Counter reason
producer	event over --logging-native-max-event-bytes	dropped whole, never truncated — a truncated JSON body is unparseable	oversize_event
buffer	ring over budget	drop the newest event, the writer still returning (len(p), nil); queued events are never evicted, so a burst keeps its head	buffer_full / memory_pressure
consumer	stalled in a slow publish	the ring absorbs it; batches capped at 25% of budget, one in flight, so a stall cannot pin admission	– gauge native_buffer_bytes{state=in_flight}
schema	create fails for anything but AlreadyExists, or the group is dropped at runtime	retry next flush, batch in hand dropped	schema_unavailable
schema	existing log stream is incompatible	refuse, one error through normal logging, sink stays disabled	schema_incompatible
selection	Locate errors or returns an empty node ID	count and return; never publish to an empty node	locate_failed
local publish	listener unhealthy	the bus skips it and drops the payload while returning an error, so any error means full batch loss	publish_failed
remote publish	error, breaker open, admission timeout, auth rejection	batch dropped, never re-queued; classified from the per-node error map from Close	publish_failed
destination	local topic not yet subscribed	retain, N retries, then drop	destination_unready
read path	a data node has not yet caught up on the schema	its query processor returns stream-not-exist and the distributed planner discards the whole response, blanking the query. Pre-existing for any group created while the cluster is running, user groups included — not introduced here. Mitigation is a one-site change in the planner, out of scope for this feature	–
shutdown	buffer still full at the deadline	remainder dropped once --logging-native-drain-timeout expires; teardown never blocked	shutdown_deadline

No row blocks, retries forever, or falls back to disk. Loss is bounded and always counted.

5.1 Memory budget

budget = min(max-bytes, memory-fraction * max(0, availableBytes - memory-reserve))
availableBytes < 0  ->  budget = max-bytes          # unknown is not unlimited

All three terms are native flags whose defaults are proposals. The budget is recomputed every 5s, counting queued and in-flight bytes alike. The sink reads AvailableBytes() only — never the blocking AcquireResource on the logging path, never waiting, never reclaiming bytes in flight; over budget it drops and counts. Phase 1 runs on the cap alone, the adaptive term arriving in phase 2.

Process	Adaptive term	Why
standalone, data	binds	protector registered as a run unit
everything else	cap only	availability reports -1, or no protector is registered. We leave the liaison as it is today rather than registering one, since that would arm load shedding that has never run

That gap is pre-existing, and registering the liaison protector is an open question below; the sink needs no change either way, so the cap is sized to be safe alone.

Normal logging is never degraded by the sink. Native is the more verbose sink, so at normal error and native info a dropped native info event has no normal copy anywhere — the drop counters are the only record it existed.

With --observability-modes=native alone those counters travel the same transport as the logs, so the same outage loses both. The sink therefore also emits a periodic drop summary straight to stderr, and we recommend keeping prometheus enabled alongside.

6. Implementation phases

Each phase adds exactly one new failure domain, so a breakage is never ambiguous about which layer is at fault.

Phase	Scope	What it de-risks
1	Writer seam in pkg/logger, both gates, ring, consumer, _monitoring_log group and log stream; standalone and data in-process	Capture and schema, with no network hop or node selection
2	Liaison route — tier-2 registry and its own selector over tire2Client — adaptive budget, metrics	Cross-node routing, and the adaptive budget where a protector is registered
3	Generic native receiver and forwarder on liaison and data nodes. Also: a batch message for the new topic — no repeated WriteRequest exists in write.proto today — plus method-policy rows and their contract test, without which the liaison refuses to boot	Foreign-event admission: validation, identity passthrough, back-pressure
4	Colocated producers (backup, lifecycle, FODC agent beside data) and the FODC proxy, onto the phase-3 receiver. backup and lifecycle reuse their existing connection settings; the FODC binaries have none to reuse — every FODC dial is currently insecure and the proxy has no outbound BanyanDB client, so both need new address and TLS flags	Producer integration alone; the receiver is proven

The receiver precedes those producers by design, so they never need a metadata client or their own copy of the shard rule. Background: implementation phases.

Phase 1 is independently shippable: standalone and single-data-node deployments get queryable, retained logs without cross-node routing or the adaptive budget.

Still open

Five decisions, each with a proposal. Silence means we take the proposal; the two marked 🔴 amend requirements you wrote, so those are the ones I would rather have answered.

Question	What it means	Proposal
🔴 1. Liaison transport	"Reuse existing shard-based routing" and "never touch disk" collide — that route ends in the write queue	Keep the selection, replace the transport with a direct tier-2 publish. Cheaper than it sounds: nothing enforces shard ownership on ingest, and lifecycle already publishes straight into a colocated data node today. Reusing a transport that is in production, not inventing one
2. Shard count	1 sends every routed producer to one data node. Raise-only — UpdateGroup does not compare old to new, but lowering silently hides written shards. No effect on reads, since queries broadcast	2, raisable later. _monitoring uses 1; logs are higher volume
🔴 3. Ring accounting unit	You specified buffering WriteRequest objects. Each runs 5–7× its raw JSON line once tag structs, oneof wrappers, timestamp and metadata are counted, so a 32 MB cap permits 100 MB+ resident — on a liaison, where the cap is the only protection and nothing else is watching. No precedent either way: neither metrics nor normal logging buffers	Buffer the raw line plus a fixed header and build the request at flush, so accounted bytes ≈ resident bytes. level arrives as an enum, timestamp and sequence sit in the header. module comes free from a thin per-module writer on each named logger; message is then the single field lifted at flush, off the hot path
4. Overflow policy	Ring full, another event arrives: refuse the new one or evict the oldest	Drop newest, as accesslog does when sampling. Keeps the head of a burst — the first error in a cascade — and an admitted event is never evicted. We cannot take accesslog's blocking default: the sink sits on the write path, where blocking risks deadlock
5. Entity	Entity tags allow only EQ/IN, so anything in the entity can be matched but never excluded	[node_id] only, module and level as ordinary tags. level != "debug" needs level outside the entity, and module has unbounded cardinality. Both stay filterable by equality. MATCH works on neither as an ordinary tag, so it is not an argument here

Inherited rather than decided. Four earlier questions dropped out once we checked what the project already does, and we simply follow it:

	Behaviour	Precedent
A data node's own logs	stored locally, dying with the node	native metrics do exactly this today — queue.Local() with a nil selector on data
Replicas	0	_monitoring sets none
Liaison memory protector	left unregistered, so the cap is the only term there	that is the liaison's current state; registering it would arm load shedding that has never run, which is not a logging change
Schema-propagation window	accepted, see Failure modes	pre-existing for any group created while the cluster is running, including user groups

Phase 1 is gated by 2, 3, 4 and 5. 1 shapes phase 2.

The design doc carries the full detail and will land with the implementation PR.

[hanahmily]

Thanks for working through the design. On the five open questions:

1. Liaison transport: Direct tier-2 publishing is fine. Reuse shard/node selection and bypass the disk-backed write queue.

2. Shard count: Start with 2. Shard count can be changed through the group schema. I verified the existing TestGroupCRUD, which updates a stream group from 2 to 4 shards and reads back the updated value; the test passes.

3. Ring representation: Keep WriteRequest objects, using a pool to reuse requests and their nested allocations. The stream schema defines the keys—level, timestamp, content, etc.—and requests carry the corresponding values. Do not retain the original JSON as an additional body. The proposed 5–7× comparison therefore does not describe the intended representation and should be removed unless supported by measurements of this design.

   Pooling reduces allocation churn and GC pressure; byte limits must still account for queued and in-flight requests. Return requests to the pool only after publishing completes or they are dropped, clear retained references, and avoid retaining oversized buffers.

4. Overflow policy: Drop newest. Admission must never wait for buffer capacity.

5. Entity: Use [node_id, level]. Keep module as an ordinary tag. We accept the current entity-filter restriction to EQ/IN; queries selecting multiple levels should use IN.

Please incorporate these decisions into the design doc and make the schema, configuration tables, and memory-accounting description consistent with them.