RFC-026 §4.7 P1: streaming enablement authority (internal schema v10)

[aaltshuler]

Slice 1 of the experimental streaming profile (RFC-026 §4.7, selected 2026-07-27): the graph-wide enablement flag as durable manifest state, end to end.

What ships

• Internal schema v10 — the required graph-global stream_profile singleton (present from genesis, streaming disabled, profile_revision 1) plus the reserved fold-attribution dead_letter_object slot. The bump is forced by decode semantics: v9 decoders silently skip unknown row kinds, so only the stamp turns "older binary meets streaming-capable graph" into a refusal instead of a writer blind to the freeze; and the attribution summary is deny_unknown_fields + structurally equality-compared between recovery sidecar and lineage row, so the DLQ slot cannot be retrofitted through a serde default later.
• Omnigraph::set_streaming_enabled_as — Cedar-gated (reserved per-graph Admin action, first consumer), full write prelude (heal barrier → stream-admission capture → schema/branch/stream-token gates → sidecar re-check), one SetStreamProfile exact-entry CAS with strict revision advance, one ordinary lineage commit (auditable in commit list). No recovery sidecar: zero pre-manifest durable effects (repair-style registered sidecar-free writer). Disable refuses typed (StreamingDisablePending, naming the undrained tables) while any stream lifecycle is non-terminal.
• Cluster propagation — graphs.<id>.streaming: true|false in cluster.yaml; cluster apply is the only flip mechanism (first-class streaming.<id> resource — deliberately not folded into the graph composite digest, whose Derived classification would fake-converge the ledger without an engine call). Absent = unmanaged; removal never disables; disable converges pending-until-drained (streaming_drain_pending, threaded like pending_recovery). cluster refresh reads the live value back from the graph, so crashed applies and out-of-band flips surface as ordinary plan drift toward the declaration.
• Genuine v9↔v10 fence — OMNIGRAPH_V9_BIN seam + current_v10_refuses_and_rebuilds_genuine_v9_and_v9_refuses_v10, validated locally against a binary built from the pinned final-v9 commit; CI builds that binary (FINAL_INTERNAL_V9_COMMIT) before the workspace suite with the same 7-package cargo-clean discipline as the v8 step. New in-source migrations assertions pin the exact 0.9.x refusal strings locally so a release-map edit breaks fast, not only in the env-gated cell (the Align crossversion refusal assertions with the 0.9.0-dev naming #387 failure class).

Enabling the flag activates no ingest surface — P2–P6 of the profile remain unimplemented; this slice only makes the graph durably, universally know whether streaming is enabled, with the off-switch semantics built correctly from day one.

Evidence

• Engine: 348 in-source cells (genesis/CAS/revision/reopen; serde exact incl. explicit-null DLQ slot and absent-key refusal; undrained-refusal with injected OPEN lifecycle), lifecycle.rs public-surface cell, policy-chassis deny/allow pair proving the (Admin, Graph) scope, forbidden_apis 19/19 (new writer + durable-call registered; publisher chokepoints unchanged at 1).
• Cluster: 122 in-source cells incl. full flip e2e (create+flip → idempotent → explicit disable → unmanage keeps graph state), refresh-convergence e2e, import-over-pre-existing-graph flip, drain-pending classification unit, unmanaged-digest-stability (no approval churn).
• CLI: both cluster suites green; lifecycle e2e pins import-records-born-disabled → declared-drift-applies → status-echoes-stored-observations-by-design → refresh-reports-live-truth. S3 lifecycle fixture now exercises the flip on RustFS CI.
• Gates: cargo test --workspace --locked 75/75 suites green; failpoints 141/141; memwal_stream green.

Pre-existing failures on main (not this PR)

• Main's post-merge Test Workspace is currently red with thread 'manifest_reads_capture_warm_probe' has overflowed its stack — on commits predating this branch. The same environment-sensitive overflow reproduces locally in memwal_stream_cost::b2a_retained_history_small_depth_keeps_terms_and_old_roots_separate.
• Under an enlarged stack, that cell reveals a warm-ack flatness failure (table_reads 17 vs 11 across retained depth 8 vs 1). Verified byte-identical at the base commit on the same machine (reads 11/17, bytes 38,399/63,993 on both trees) — this slice is cost-neutral on the warm-ack path to the byte. Both issues deserve their own investigation.

Pre-merge checklist

• Re-pin FINAL_INTERNAL_V9_COMMIT in ci.yml if main advances with any v9-writing commit before this merges (it must remain the last v9 commit; it becomes immutable after merge). A concurrent format-bumping PR would be a real conflict — only one v10 bump can exist.

Greptile Summary

Adds durable graph-wide streaming enablement without activating an ingest surface.

• Introduces internal manifest schema v10 with a required stream_profile singleton and reserved dead-letter attribution slot.
• Adds Cedar-authorized, revisioned streaming enable/disable operations with undrained-stream protection.
• Models streaming enablement as an independently converged cluster resource with refresh and retry behavior.
• Adds genuine v9↔v10 compatibility fencing, tests, CI coverage, and operator documentation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
crates/omnigraph-cluster/src/diff.rs	Adds first-class streaming resource classification and now correctly demotes it when its graph prerequisite fails.
crates/omnigraph-cluster/src/lib.rs	Executes streaming convergence through the engine and records only successfully applied changes.
crates/omnigraph/src/db/omnigraph/stream_profile.rs	Implements the authorized, revisioned streaming profile transition and undrained-disable refusal.
crates/omnigraph/src/db/manifest/stream_profile.rs	Defines durable stream-profile state and exact manifest CAS behavior.
crates/omnigraph/src/db/manifest/migrations.rs	Extends strict internal-format fencing and rebuild guidance to schema v10.
.github/workflows/ci.yml	Builds a pinned final-v9 binary so CI exercises genuine bidirectional v9/v10 refusal and rebuild behavior.

Sequence Diagram

sequenceDiagram
    participant Config as cluster.yaml
    participant Apply as cluster apply
    participant Engine as Omnigraph
    participant Manifest as __manifest v10
    participant Ledger as cluster state

    Config->>Apply: "graphs.<id>.streaming"
    Apply->>Engine: set_streaming_enabled_as(...)
    Engine->>Manifest: exact-CAS stream_profile revision
    alt Flip succeeds
        Manifest-->>Engine: committed profile
        Engine-->>Apply: streaming status
        Apply->>Ledger: "record streaming.<id> digest"
    else Disable blocked by undrained streams
        Engine-->>Apply: StreamingDisablePending
        Apply->>Ledger: mark streaming_drain_pending
    else Graph prerequisite failed
        Apply->>Apply: demote streaming dependency
        Apply->>Ledger: preserve prior streaming digest
    end

Loading

_{Reviews (2): Last reviewed commit: "Fix streaming authority convergence" | Re-trigger Greptile}

Context used:

• Knowledge Base — Cluster Orchestration (omnigraph-cluster)
• Knowledge Base — OmniGraph core storage: manifest, table store, and Lance access