diff --git a/AGENTS.md b/AGENTS.md index e778fdcb..07fc1b28 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,7 +33,7 @@ OmniGraph is a typed property-graph engine built as a coordination layer over ma - **Multi-modal querying**: vector ANN (`nearest`), full-text (`search`/`fuzzy`/`match_text`/`bm25`), Reciprocal Rank Fusion (`rrf`), and graph traversal (`Expand`, anti-join `not { … }`) in one runtime. - **Branches and commits across the whole graph**: Git-style — every successful publish appends to a commit DAG; merges are three-way at the row level. - **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table. Strict insert and upsert both route through the sealed exact-`id`, filter-bearing adapter; bare Lance Append is test-only. Their RFC-022 adapter resolves or rejects relevant recovery intents before base capture, captures `(native branch id, exact graph_head, schema identity)`, then rechecks recovery and authority under schema → branch → table gates. Mutation/Load keeps one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before recovery arm; mutation update scans stream into the remaining table budget after pending-key shadowing, with blob sizes checked before payload reads. It then arms an identity-bearing recovery-v9 sidecar containing exact Lance transaction identities + pre-minted lineage, commits each table with zero transparent conflict retries, confirms the achieved effects, and publishes under the same token. Unrelated retryable pre-effect authority movement may fully reprepare without changing logical mode. A proven effect-free strict conflict returns `KeyConflict` only after a fresh exact-ID probe; no exact match triggers bounded full strict-mode reprepare, while an effect-free upsert conflict also fully reprepares. Any earlier effect or ambiguity returns `RecoveryRequired` with the sidecar retained. Strict read-set conflicts return `ReadSetChanged`. Deletes stage through the same path. D₂ at parse time remains the constructive (insert/update) XOR destructive (delete) boundary. -- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB of logical dense-slice Arrow data), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and the bounded fold mechanics. Cheap raw bounds and exact post-tombstone validation run before recovery; put ownership follows charge → shared admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans seal, drain proof, both table effects, and publication. Admission, replay, and fold all charge `ArrayData::get_slice_memory_size`; fold densifies retained arrays. Backing-buffer capacity and physical allocation are not the limit, while isolated fold RSS remains a 384-MiB remeasurement tripwire. The 8,192-row high-entropy near-cap closure cell remains green. Internal schema v9 adds the private B2 compare-and-chain core: canonical payload and token digests, grammar-impossible trusted hidden row metadata (`__omnigraph_stream_v1$`), same-generation token overlays, and a manifest-selected graph-global `_stream_tokens.lance` authority. Exact base/token lookups and recovery validation materialize at most the requested rows plus one and cap both Arrow batch bytes and cumulative retained bytes. Admission recaptures lifecycle/binding/HEAD authority only after acquiring shared admission and the same-key queue, so a stale provisional capture cannot authorize a WAL put. Recovery-v12 owns exact pre-minted base and token transactions; only their exact joint outcome may drive the sole `__manifest` CAS that advances both pointers, lifecycle state-v2, graph lineage, and a durable fold-attribution summary. Recovery-v11 is historical v8 state and is refused under v9 because it lacks the token participant and complete state-v2 authority. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all profile is active**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and untouched through recovery/reopen; provider exhaustion is loud. The genuine v8↔v9 refusal/rebuild gate preserves logical rows—including an ordinary v8 `__omnigraph_stream_v1` user property—vectors, and PK metadata while omitting hidden trusted stream metadata from export. The core remains crate-private behind one feature-gated, doc-hidden test seam. Explicit production enrollment, lifecycle management/correction/status, authorization, and SDK/CLI/HTTP/OpenAPI surfaces remain inactive. +- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB of logical dense-slice Arrow data), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and the bounded fold mechanics. Cheap raw bounds and exact post-tombstone validation run before recovery; put ownership follows charge → shared admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans seal, drain proof, both table effects, and publication. Admission, replay, and fold all charge `ArrayData::get_slice_memory_size`; fold densifies retained arrays. Backing-buffer capacity and physical allocation are not the limit, while isolated fold RSS remains a 384-MiB remeasurement tripwire. The 8,192-row high-entropy near-cap closure cell remains green. Internal schema v9 adds the private B2 compare-and-chain core: canonical payload and token digests, grammar-impossible trusted hidden row metadata (`__omnigraph_stream_v1$`), same-generation token overlays, and a manifest-selected graph-global `_stream_tokens.lance` authority. Exact base/token lookups and recovery validation materialize at most the requested rows plus one and cap both Arrow batch bytes and cumulative retained bytes. Admission recaptures lifecycle/binding/HEAD authority only after acquiring shared admission and the same-key queue, so a stale provisional capture cannot authorize a WAL put. Recovery-v12 owns exact pre-minted base and token transactions; only their exact joint outcome may drive the sole `__manifest` CAS that advances both pointers, lifecycle state-v2, graph lineage, and a durable fold-attribution summary. Recovery-v11 is historical v8 state and is refused under v9 because it lacks the token participant and complete state-v2 authority. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all profile is active**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and untouched through recovery/reopen; provider exhaustion is loud. The genuine v8↔v9 refusal/rebuild gate preserves logical rows—including an ordinary v8 `__omnigraph_stream_v1` user property—vectors, and PK metadata while omitting hidden trusted stream metadata from export. The core remains crate-private behind one feature-gated, doc-hidden test seam. The graph-scoped `stream_ingest` / `stream_manage` Cedar vocabulary is registered, the v10 enablement flip is gated by `stream_manage`, and the embedded SDK exposes durable, manifest-only read-only `stream_status`. Public row ingest, explicit production enrollment, lifecycle mutation/correction, exclusive-cut physical status, and CLI/HTTP/OpenAPI streaming surfaces remain inactive. - **RFC-026 B2 preprocessing bound**: before blob materialization or canonical encoding, the private adapter reserves a 128-MiB worst-case envelope (original 32-MiB Arrow row + possible 32-MiB replacement + 64-MiB canonical payload) and an inflight slot. The private profile admits exactly two envelopes (256 MiB root-wide), preserving the two-caller stale-authority race without unbounded preprocessing. The slot transfers into queued/worker ownership; scratch releases after digest derivation. Pressure fails effect-free as typed `stream_b2_preprocessing_bytes`; this is process-memory admission, not retained-storage GC or quota. - **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster `, RFC-005) and serves N graphs (N ≥ 1) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart. - **CLI** with two-surface config (RFC-007/008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, clusters, credentials, actor, profiles, aliases, defaults). Graphs are addressed via `--store`/`--server`/`--cluster`/`--profile`/operator defaults (RFC-011). Multi-format output (json/jsonl/csv/kv/table). @@ -282,7 +282,7 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act | Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` derives requested `--keep` / `--older-than` cutoffs from each table's available versions; Lance refs plus OmniGraph's live-lazy-branch and recovery floors may retain additional versions. It fails closed on unopenable pins, recovery intent, or uncovered main-table HEAD drift | | BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: they publish only logical data/schema effects, leaving physical intent pending. `ensure_indices` first runs the roll-forward-only recovery barrier before base capture or planning, then materializes every declared-but-missing artifact for one table through one staged mixed CreateIndex transaction under its identity-bearing recovery-v9 authority/lineage/delta envelope; it continues to report untrainable Vector columns as pending. `Armed` is rollback-only, `EffectsConfirmed` rolls forward only while captured authority holds, and first-touch refs carry exact identity. `optimize` separately restores fragment coverage through its bounded identity-bearing v9 maintenance envelope. | | Strict insert / upsert ingestion | ✅ transaction conflict filters + uncommitted fragment staging | Internal schema v6 introduced the explicit logical mode and v9 preserves it. General strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint the same optional certificate only from its completed effect statistics. Mutation/Load remains one transaction per table, capped before arm at 8,192 rows / 32 MiB. BranchMerge's proven all-new route accepts only a complete certificate chain plus final source/target native-incarnation checks; it stages bounded fragments with `InsertBuilder`, commits them as exact-`id` filtered `Update` transactions, and performs zero target preflights, target merge joins, or committed Appends. Missing or malformed proof falls back to the general ordered diff. Raw Lance writers are outside the supported graph-writer topology, and the certificate is an internal, non-cryptographic capability. The final five-pair production gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes). | -| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v9 introduced — and v10 preserves — exact identity-keyed physical binding, current-HEAD witness, lifecycle state-v2, per-shard epoch floor, stream-config v3, and the manifest-selected `_stream_tokens.lance` pointer; v10 adds the graph-global `stream_profile` enablement singleton (§4.7 P1), obeyed by every opener, with all streaming surfaces still inactive. Phase A's recovery-v10 adapter enrolls one empty unsharded shard. The private B1 worker remains one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB of logical dense-slice Arrow data with watcher-plus-post-fence acknowledgement, fold-only replay, and seal/drain proof; backing-buffer capacity and physical RSS are not admission authority. Private B2 admission adds canonical payload/token digests, trusted hidden metadata, same-key compare-and-chain/idempotency classification, same-generation overlays, and authority recapture after shared admission. Recovery-v12 folds one generation through exact base and token transactions; its sole manifest CAS advances both pointers, lifecycle, lineage, and durable attribution together. The private B2a profile remains unbounded retain-all: no OmniGraph byte/object/file/history quota, no canonical durable `_mem_wal` deletion, typed provider failure, and inert retained orphan residue. The 1/8/32/128 local/RustFS instrument remains advisory. It performs no generation GC or fresh reads. B2b retains the future Lance-owned reclamation design. The genuine v8↔v9 refusal/rebuild gate is active. Explicit production enrollment, revisioned lifecycle management, correction/status, Cedar, SDK, CLI, HTTP, and OpenAPI contracts remain inactive. | +| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v9 introduced — and v10 preserves — exact identity-keyed physical binding, current-HEAD witness, lifecycle state-v2, per-shard epoch floor, stream-config v3, and the manifest-selected `_stream_tokens.lance` pointer; v10 adds the graph-global `stream_profile` enablement singleton (§4.7 P1), obeyed by every opener. Phase A's recovery-v10 adapter enrolls one empty unsharded shard. The private B1 worker remains one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB of logical dense-slice Arrow data with watcher-plus-post-fence acknowledgement, fold-only replay, and seal/drain proof; backing-buffer capacity and physical RSS are not admission authority. Private B2 admission adds canonical payload/token digests, trusted hidden metadata, same-key compare-and-chain/idempotency classification, same-generation overlays, and authority recapture after shared admission. Recovery-v12 folds one generation through exact base and token transactions; its sole manifest CAS advances both pointers, lifecycle, lineage, and durable attribution together. The private B2a profile remains unbounded retain-all: no OmniGraph byte/object/file/history quota, no canonical durable `_mem_wal` deletion, typed provider failure, and inert retained orphan residue. The 1/8/32/128 local/RustFS instrument remains advisory. It performs no generation GC or fresh reads. B2b retains the future Lance-owned reclamation design. The genuine v8↔v9 refusal/rebuild gate is active. Registered graph-scoped Cedar actions are `stream_ingest` and `stream_manage`; `stream_manage` gates the v10 enablement writer, and embedded `Omnigraph::stream_status` exposes the durable manifest projection read-only. Explicit production enrollment, revisioned lifecycle mutation/correction, exclusive-cut physical status, public row ingestion, and CLI/HTTP/OpenAPI streaming contracts remain inactive. | | Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema | | Full-text search | ✅ | `search/fuzzy/match_text/bm25` query ops | | Hybrid ranking | — | `rrf(...)` Reciprocal Rank Fusion in one runtime | diff --git a/crates/omnigraph-cli/src/planes.rs b/crates/omnigraph-cli/src/planes.rs index a0f0a1c8..add2b884 100644 --- a/crates/omnigraph-cli/src/planes.rs +++ b/crates/omnigraph-cli/src/planes.rs @@ -404,6 +404,7 @@ mod tests { use clap::Parser; use super::*; + use crate::cli::PolicyCommand; #[test] fn scope_flag_matrix_matches_capabilities() { @@ -474,6 +475,45 @@ mod tests { ); } + #[test] + fn policy_explain_accepts_stream_action_wire_names_and_kebab_aliases() { + let parse_action = |spelling: &str| { + let command = Cli::try_parse_from([ + "omnigraph", + "policy", + "explain", + "--actor", + "act-operator", + "--action", + spelling, + ]) + .unwrap() + .command; + match command { + Command::Policy { + command: PolicyCommand::Explain { action, .. }, + } => action, + other => panic!("expected policy explain command, got {other:?}"), + } + }; + + for (canonical, alias, expected) in [ + ( + "stream_ingest", + "stream-ingest", + omnigraph_policy::PolicyAction::StreamIngest, + ), + ( + "stream_manage", + "stream-manage", + omnigraph_policy::PolicyAction::StreamManage, + ), + ] { + assert_eq!(parse_action(canonical), expected); + assert_eq!(parse_action(alias), expected); + } + } + #[test] fn every_capability_describes_distinctly() { let phrases = [ diff --git a/crates/omnigraph-policy/src/lib.rs b/crates/omnigraph-policy/src/lib.rs index 46b380a1..67486f56 100644 --- a/crates/omnigraph-policy/src/lib.rs +++ b/crates/omnigraph-policy/src/lib.rs @@ -71,6 +71,29 @@ pub enum PolicyAction { /// is double-gated: `invoke_query` to reach the tool, plus `change` for /// the write itself. InvokeQuery, + /// RFC-026: gates admitting rows into a table's streaming lane. Per-graph + /// and **graph-scoped**: the experimental streaming profile is main-only, + /// so a branch dimension would describe a topology the lane refuses. A + /// rule that sets `branch_scope` or `target_branch_scope` on it is + /// rejected by `validate()`. + /// + /// Deliberately separate from `Change`: a stream append acknowledges + /// durability without graph visibility, so an operator can grant + /// high-rate ingestion without granting direct-lane writes (or the + /// reverse). See RFC-026 §4.6. + #[value(name = "stream_ingest", alias = "stream-ingest")] + StreamIngest, + /// RFC-026: gates streaming *lifecycle management* — enable/disable, + /// fold, quiesce, resume, and abort-drain. Per-graph and graph-scoped for + /// the same main-only reason as `StreamIngest`. + /// + /// Split from `StreamIngest` because the blast radii differ in kind: + /// ingestion adds rows, while management can seal a lane, drain + /// acknowledged data, or reopen it at a new epoch. Read-only stream + /// status is authorized like other graph operational metadata, not by + /// this action. + #[value(name = "stream_manage", alias = "stream-manage")] + StreamManage, } impl PolicyAction { @@ -86,6 +109,8 @@ impl PolicyAction { Self::Admin => "admin", Self::GraphList => "graph_list", Self::InvokeQuery => "invoke_query", + Self::StreamIngest => "stream_ingest", + Self::StreamManage => "stream_manage", } } @@ -116,7 +141,9 @@ impl PolicyAction { | Self::BranchDelete | Self::BranchMerge | Self::Admin - | Self::InvokeQuery => PolicyResourceKind::Graph, + | Self::InvokeQuery + | Self::StreamIngest + | Self::StreamManage => PolicyResourceKind::Graph, } } } @@ -173,6 +200,8 @@ impl FromStr for PolicyAction { "admin" => Ok(Self::Admin), "graph_list" => Ok(Self::GraphList), "invoke_query" => Ok(Self::InvokeQuery), + "stream_ingest" => Ok(Self::StreamIngest), + "stream_manage" => Ok(Self::StreamManage), other => bail!("unknown policy action '{other}'"), } } @@ -845,6 +874,8 @@ namespace Omnigraph { action "branch_merge" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "admin" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "invoke_query" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; + action "stream_ingest" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; + action "stream_manage" appliesTo { principal: Actor, resource: Graph, context: RequestContext }; action "graph_list" appliesTo { principal: Actor, resource: Server, context: RequestContext }; } @@ -1060,7 +1091,7 @@ rules: } use super::{ PolicyAction, PolicyCompiler, PolicyConfig, PolicyEngine, PolicyExpectation, PolicyRequest, - PolicyTestCase, PolicyTestConfig, + PolicyResourceKind, PolicyTestCase, PolicyTestConfig, }; #[test] @@ -1413,6 +1444,53 @@ rules: ); } + #[test] + fn stream_actions_are_graph_scoped_and_reject_branch_qualifiers() { + // RFC-026's experimental streaming profile is main-only, so a branch + // dimension on either stream action would describe a topology the + // lane refuses. Both qualifiers are rejected at validate(). + for (action, qualifier) in [ + ("stream_ingest", "branch_scope: any"), + ("stream_ingest", "target_branch_scope: any"), + ("stream_manage", "branch_scope: any"), + ("stream_manage", "target_branch_scope: any"), + ] { + let policy: PolicyConfig = serde_yaml::from_str(&format!( + r#" +version: 1 +groups: + team: [act-alice] +rules: + - id: team-stream + allow: + actors: {{ group: team }} + actions: [{action}] + {qualifier} +"# + )) + .unwrap(); + let err = policy.validate().unwrap_err().to_string(); + let scope = qualifier.split(':').next().unwrap(); + assert!( + err.contains(scope) && err.contains(action), + "{scope} on {action} must be rejected: {err}" + ); + } + + // Both are per-graph resources, not server-scoped. + for action in [PolicyAction::StreamIngest, PolicyAction::StreamManage] { + assert_eq!(action.resource_kind(), PolicyResourceKind::Graph); + // The stable policy/YAML wire name round-trips through FromStr. + // The CLI owns a separate parser test because it derives its + // accepted values through clap's ValueEnum implementation. + assert_eq!( + action.as_str().parse::().unwrap(), + action, + "wire name must round-trip through FromStr" + ); + } + } + #[test] fn server_scoped_rule_cannot_use_branch_scope() { let policy: PolicyConfig = serde_yaml::from_str( diff --git a/crates/omnigraph/src/db/manifest.rs b/crates/omnigraph/src/db/manifest.rs index b5fb3816..ac0c1bd6 100644 --- a/crates/omnigraph/src/db/manifest.rs +++ b/crates/omnigraph/src/db/manifest.rs @@ -643,6 +643,21 @@ impl Snapshot { self.stream_lifecycles.iter() } + /// Test-only seam for proving that public status resolves the live table + /// registration by immutable identity instead of trusting this explicitly + /// diagnostic alias. + #[cfg(all(test, feature = "failpoints"))] + pub(crate) fn set_stream_diagnostic_table_key_for_test( + &mut self, + identity: TableIdentity, + table_key: &str, + ) { + self.stream_lifecycles + .get_mut(&identity) + .expect("test snapshot must contain the requested stream lifecycle") + .diagnostic_table_key = table_key.to_string(); + } + /// Exact durable pointer to the graph-global RFC-026 token participant. pub(crate) fn stream_token_authority(&self) -> &StreamTokenAuthorityEntry { &self.stream_token_authority diff --git a/crates/omnigraph/src/db/manifest/stream.rs b/crates/omnigraph/src/db/manifest/stream.rs index ca4af50e..1c9d7f9f 100644 --- a/crates/omnigraph/src/db/manifest/stream.rs +++ b/crates/omnigraph/src/db/manifest/stream.rs @@ -418,6 +418,17 @@ pub(crate) enum LastFoldOutcome { StrictBlocked, } +impl LastFoldOutcome { + /// The exact durable wire name, so a status projection reports the stored + /// value instead of minting a second spelling of it. + pub(crate) const fn as_str(self) -> &'static str { + match self { + Self::Published => "PUBLISHED", + Self::StrictBlocked => "STRICT_BLOCKED", + } + } +} + #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(crate) struct LastFoldSummary { diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index 416f66d3..e14dc2c2 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -14,8 +14,8 @@ pub(crate) use omnigraph::{DeferredTableFork, WriteAuthorityToken, WriteTxn}; pub use omnigraph::{ CleanupPolicyOptions, InitOptions, MergeOutcome, Omnigraph, OpenMode, PendingIndex, RepairAction, RepairClassification, RepairOptions, RepairStats, SchemaApplyOptions, - SchemaApplyResult, SkipReason, StreamingProfileResult, TableCleanupStats, TableOptimizeStats, - TableRepairStats, + SchemaApplyResult, SkipReason, StreamStatus, StreamTableStatus, StreamingProfileResult, + TableCleanupStats, TableOptimizeStats, TableRepairStats, }; use crate::error::{OmniError, Result}; diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 9659f418..9135e70d 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -41,10 +41,12 @@ mod schema_apply; mod stream_enrollment; mod stream_ingest; mod stream_profile; +mod stream_status; mod table_ops; pub use optimize::{CleanupPolicyOptions, SkipReason, TableCleanupStats, TableOptimizeStats}; pub use stream_profile::StreamingProfileResult; +pub use stream_status::{StreamStatus, StreamTableStatus}; pub use repair::{ RepairAction, RepairClassification, RepairOptions, RepairStats, TableRepairStats, }; diff --git a/crates/omnigraph/src/db/omnigraph/stream_profile.rs b/crates/omnigraph/src/db/omnigraph/stream_profile.rs index 9cb3c4ea..598d7a82 100644 --- a/crates/omnigraph/src/db/omnigraph/stream_profile.rs +++ b/crates/omnigraph/src/db/omnigraph/stream_profile.rs @@ -54,8 +54,14 @@ impl Omnigraph { enabled: bool, actor: Option<&str>, ) -> Result { + // Streaming lifecycle management, not general graph administration: + // the flag opens or closes a lane, so it belongs with fold, quiesce, + // and resume under `stream_manage` (RFC-026 §4.6). P1 shipped on the + // reserved `Admin` action because `stream_manage` did not exist yet; + // migrating is in-window precisely because §4.7 declares the + // experimental profile's authorization surface unstable. self.enforce( - omnigraph_policy::PolicyAction::Admin, + omnigraph_policy::PolicyAction::StreamManage, &omnigraph_policy::ResourceScope::Graph, actor, )?; diff --git a/crates/omnigraph/src/db/omnigraph/stream_status.rs b/crates/omnigraph/src/db/omnigraph/stream_status.rs new file mode 100644 index 00000000..66c5d8fc --- /dev/null +++ b/crates/omnigraph/src/db/omnigraph/stream_status.rs @@ -0,0 +1,206 @@ +//! RFC-026 read-only stream status. +//! +//! Status answers two operator questions: *is this graph streaming, and what +//! state is each lane in?* It is deliberately the least privileged surface in +//! the streaming family — a pure projection of one `__manifest` snapshot, +//! authorized like other graph operational metadata rather than by +//! `stream_manage` (§4.6), and structurally incapable of moving a lifecycle: +//! it takes no admission lease, resolves no recovery, and publishes nothing. +//! +//! It is also the **compare-token source**. Every mutating management call +//! (fold, quiesce, resume, abort-drain) is compare-and-set against an expected +//! `lifecycle_revision`, and §4.6 requires status to expose the revision +//! callers pass back. Shipping it before those verbs is what lets them be +//! written as CAS from the start rather than retrofitted. +//! +//! **Scope of this slice.** This is the §4.7 *minimal* status: the +//! authoritative manifest row only. The full contract (§4.3) additionally +//! takes stream admission exclusively, settles every writer/watcher/flush +//! owner to a deadline, reads the physical shard witness, and rereads the +//! authorities before release — returning typed `StatusChanged` / `StatusBusy` +//! on movement or failure to settle. Those terms describe *physical* +//! observation; the fields below are exactly the durable ones, so nothing here +//! becomes wrong when that arrives — it gains an observed-physical section. +//! Fields whose only honest source is that physical read (observed epoch, +//! pending generation rows/bytes) are deliberately absent rather than +//! guessed-at from durable state. + +use std::collections::HashMap; + +use crate::db::manifest::StreamLifecycle; +use crate::db::{Omnigraph, ReadTarget, Snapshot}; +use crate::error::{OmniError, Result}; + +/// One lane's durable status, projected from the manifest lifecycle row. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct StreamTableStatus { + /// Current public alias of the enrolled table. The row itself is keyed by + /// immutable identity; status resolves that identity through this exact + /// snapshot's live table registration rather than trusting the lifecycle's + /// diagnostic alias, which may lag a metadata-only rename. + pub table_key: String, + pub stable_table_id: u64, + pub table_incarnation_id: u64, + /// `OPEN` | `DRAINING` | `SEALED`. + pub lifecycle: &'static str, + /// The compare token: every mutating management call passes the revision + /// it expects, and a mismatch refuses without retargeting (§4.6). + pub lifecycle_revision: u64, + /// Exact logical stream incarnation. Unlike the physical enrollment id, + /// this survives a same-table physical rebind and fences delayed requests + /// from a prior logical stream incarnation. + pub stream_incarnation_id: String, + /// Current physical enrollment binding. + pub enrollment_id: String, + /// Authoritative per-shard epoch floor. This is durable authority, not the + /// physically observed writer epoch, which needs the exclusive-cut read. + pub epoch_floor_by_shard: Vec<(String, u64)>, + /// Present only while `DRAINING`. + pub drain_id: Option, + /// Present only while a fold is strict-blocked; the token addresses the + /// block in a future correction call. + pub strict_block_token: Option, + /// Outcome of the last fold this lane recorded, if any. + pub last_fold_outcome: Option, + /// Graph commit the last fold published, when it published one. + pub last_fold_graph_commit_id: Option, +} + +/// Graph-wide streaming status: the enablement flag plus one row per enrolled +/// table. +#[derive(Debug, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct StreamStatus { + /// The §4.7 P1 graph-global enablement flag. + pub streaming_enabled: bool, + /// Revision of the enablement row itself (distinct from per-lane + /// `lifecycle_revision`). + pub profile_revision: u64, + /// Enrolled lanes, ordered by `table_key` so output is deterministic + /// within a snapshot rather than hash-ordered. + pub tables: Vec, +} + +impl StreamStatus { + /// True when any lane is non-terminal — the condition that makes a + /// disable refuse as pending-until-drained, and that a future quiesce + /// clears. + pub fn undrained(&self) -> bool { + self.tables + .iter() + .any(|table| table.lifecycle != StreamLifecycle::Sealed.as_str()) + } +} + +/// Project every durable field from one already-resolved immutable snapshot. +/// +/// Keeping the projection behind one snapshot parameter makes the atomicity +/// boundary structural: callers cannot accidentally source the profile, +/// lifecycle compare tokens, or current aliases from separate manifest reads. +fn project_stream_status(snapshot: &Snapshot) -> Result { + let profile = snapshot.stream_profile(); + let table_keys_by_identity = snapshot + .entries() + .map(|entry| (entry.identity, entry.table_key.as_str())) + .collect::>(); + + let mut tables: Vec = snapshot + .stream_lifecycles() + .map(|(identity, lifecycle)| { + let table_key = table_keys_by_identity.get(identity).ok_or_else(|| { + OmniError::manifest_internal(format!( + "stream status found lifecycle authority for non-live table identity {identity}" + )) + })?; + Ok(StreamTableStatus { + table_key: (*table_key).to_string(), + stable_table_id: identity.stable_table_id, + table_incarnation_id: identity.table_incarnation_id, + lifecycle: lifecycle.lifecycle.as_str(), + lifecycle_revision: lifecycle.lifecycle_revision, + stream_incarnation_id: lifecycle.enrollment_receipt.stream_incarnation_id.clone(), + enrollment_id: lifecycle.binding.enrollment_id.clone(), + epoch_floor_by_shard: lifecycle + .epoch_floor_by_shard + .iter() + .map(|(shard, floor)| (shard.clone(), *floor)) + .collect(), + drain_id: lifecycle.drain.as_ref().map(|drain| drain.drain_id.clone()), + strict_block_token: lifecycle + .strict_block + .as_ref() + .map(|block| block.block_token.clone()), + last_fold_outcome: lifecycle + .last_fold_summary + .as_ref() + .map(|summary| summary.outcome.as_str().to_string()), + last_fold_graph_commit_id: lifecycle + .last_fold_summary + .as_ref() + .and_then(|summary| summary.graph_commit_id.clone()), + }) + }) + .collect::>()?; + // Deterministic output within a snapshot: the source is a HashMap, and + // hash-map iteration order is never allowed to reach a result surface. + tables.sort_by(|a, b| { + a.table_key + .cmp(&b.table_key) + .then(a.stable_table_id.cmp(&b.stable_table_id)) + .then(a.table_incarnation_id.cmp(&b.table_incarnation_id)) + }); + + Ok(StreamStatus { + streaming_enabled: profile.streaming_enabled, + profile_revision: profile.profile_revision, + tables, + }) +} + +impl Omnigraph { + /// Read graph-wide streaming status from one canonical-main snapshot. + /// + /// Read-only and lifecycle-inert by construction: it never mutates, never + /// resolves recovery, and takes no admission lease. Authorized like other + /// graph operational metadata, so it needs no `stream_manage` grant — an + /// operator can always see whether a lane is stuck, including when they + /// lack the rights to act on it. + pub async fn stream_status(&self) -> Result { + // Main-only, matching the profile's topology: lifecycle authority is + // graph-global control state and named branches only ever hold the + // copy they forked with. + let snapshot = self.snapshot_of(ReadTarget::branch("main")).await?; + // The projector accepts only this immutable snapshot. `"main"` + // normalizes to the canonical branch, so the named-branch-only profile + // projection is not involved. + project_stream_status(&snapshot) + } +} + +#[cfg(all(test, feature = "failpoints"))] +mod tests { + use super::*; + + #[tokio::test] + async fn status_uses_live_alias_when_lifecycle_diagnostic_alias_lags() { + let dir = tempfile::tempdir().unwrap(); + let db = Omnigraph::init(dir.path().to_str().unwrap(), "node Person { score: I32 }\n") + .await + .unwrap(); + db.failpoint_enroll_stream_table_for_test("node:Person") + .await + .unwrap(); + + let mut snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); + let identity = snapshot.entry("node:Person").unwrap().identity; + snapshot.set_stream_diagnostic_table_key_for_test(identity, "node:StaleDiagnosticAlias"); + + let status = project_stream_status(&snapshot).unwrap(); + assert_eq!(status.tables.len(), 1); + assert_eq!( + status.tables[0].table_key, "node:Person", + "status must resolve the live registration by immutable identity" + ); + } +} diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 4153c814..69451d97 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -277,6 +277,10 @@ const READ_ONLY_SURFACES: &[(&str, &str)] = &[ ("db/omnigraph.rs", "branch_list"), ("db/omnigraph.rs", "get_commit"), ("db/omnigraph.rs", "list_commits"), + // RFC-026 status is a pure projection of one manifest snapshot: no + // admission lease, no recovery resolution, no publication. Registering it + // read-only is the structural claim that it cannot move a lifecycle. + ("db/omnigraph/stream_status.rs", "stream_status"), ("exec/query.rs", "query"), ("exec/query.rs", "run_query_at"), ]; diff --git a/crates/omnigraph/tests/lifecycle.rs b/crates/omnigraph/tests/lifecycle.rs index d509c0c9..72a5f052 100644 --- a/crates/omnigraph/tests/lifecycle.rs +++ b/crates/omnigraph/tests/lifecycle.rs @@ -1007,3 +1007,35 @@ async fn streaming_flag_flips_are_durable_idempotent_and_versioned() { assert!(!status.enabled); assert!(!status.undrained); } + +/// RFC-026 §4.6/§4.7: read-only stream status projects the durable authority +/// and exposes the compare tokens that future management verbs (fold, +/// quiesce, resume) pass back as their expected revisions. +#[tokio::test] +async fn stream_status_projects_durable_authority_and_compare_tokens() { + let dir = tempfile::tempdir().unwrap(); + let db = init_and_load(&dir).await; + + // A graph with streaming off and no enrollment reports exactly that — + // not an error, and not an empty struct that hides the flag. + let status = db.stream_status().await.unwrap(); + assert!(!status.streaming_enabled); + assert_eq!(status.profile_revision, 1, "genesis profile revision"); + assert!(status.tables.is_empty(), "no lane exists before enrollment"); + assert!(!status.undrained()); + + // The enablement flag is visible immediately, with its own revision + // (distinct from any per-lane lifecycle revision). + db.set_streaming_enabled_as(true, None).await.unwrap(); + let status = db.stream_status().await.unwrap(); + assert!(status.streaming_enabled); + assert_eq!(status.profile_revision, 2); + assert!(status.tables.is_empty()); + + // Status is read-only: observing it neither moves the profile revision + // nor mints a graph commit. + let commits_before = db.list_commits(None).await.unwrap().len(); + let repeat = db.stream_status().await.unwrap(); + assert_eq!(repeat, status, "status is a pure projection"); + assert_eq!(db.list_commits(None).await.unwrap().len(), commits_before); +} diff --git a/crates/omnigraph/tests/memwal_stream.rs b/crates/omnigraph/tests/memwal_stream.rs index b9b8e875..4e27a7c5 100644 --- a/crates/omnigraph/tests/memwal_stream.rs +++ b/crates/omnigraph/tests/memwal_stream.rs @@ -2848,3 +2848,52 @@ async fn s3_provider_shard_manifest_failure_retains_unreferenced_generation() { .await .expect("configured S3/RustFS fixture cleanup must succeed"); } + +/// RFC-026 §4.6: with a lane actually enrolled, status projects the durable +/// logical stream incarnation plus the `lifecycle_revision` that every +/// mutating management verb passes back as its expected-revision compare +/// token. +#[tokio::test] +#[serial] +async fn stream_status_reports_the_enrolled_lane_and_its_compare_token() { + let _scenario = FailScenario::setup(); + let (_dir, db) = init_enrolled().await; + + let status = db.stream_status().await.unwrap(); + assert_eq!(status.tables.len(), 1, "one enrolled lane: {status:?}"); + let lane = &status.tables[0]; + assert_eq!(lane.table_key, TABLE); + assert_eq!(lane.lifecycle, "OPEN"); + assert_eq!( + lane.lifecycle_revision, 1, + "enrollment publishes the initial lifecycle revision" + ); + let expected_stream_incarnation = db + .failpoint_stream_incarnation_for_test(TABLE) + .await + .unwrap(); + assert_eq!( + lane.stream_incarnation_id, expected_stream_incarnation, + "status returns the exact logical incarnation used to fence every stream request" + ); + assert!(!lane.enrollment_id.is_empty()); + assert_ne!( + lane.stream_incarnation_id, lane.enrollment_id, + "logical stream incarnation and physical enrollment are distinct identities" + ); + assert_eq!( + lane.epoch_floor_by_shard.len(), + 1, + "the unsharded profile enrolls exactly one shard" + ); + // Nothing has folded or blocked yet, and the lane is not draining. + assert_eq!(lane.drain_id, None); + assert_eq!(lane.strict_block_token, None); + assert_eq!(lane.last_fold_outcome, None); + // An OPEN lane is undrained — the exact condition that makes a disable + // refuse as pending-until-drained, and that quiesce will clear. + assert!(status.undrained()); + + // Status is lifecycle-inert: re-reading it moves nothing. + assert_eq!(db.stream_status().await.unwrap(), status); +} diff --git a/crates/omnigraph/tests/policy_engine_chassis.rs b/crates/omnigraph/tests/policy_engine_chassis.rs index eef6ab29..39f66045 100644 --- a/crates/omnigraph/tests/policy_engine_chassis.rs +++ b/crates/omnigraph/tests/policy_engine_chassis.rs @@ -428,9 +428,9 @@ async fn branch_merge_as_allows_when_policy_permits_actor() { .expect("act-allowed should be able to BranchMerge"); } -/// RFC-026 §4.7 P1: the streaming-enablement flip is Cedar-gated with the -/// per-graph `admin` action — an actor with no permit rule is denied, and a -/// missing actor with a policy installed fails hard rather than silently +/// RFC-026 §4.7 P1/2a: the streaming-enablement flip is Cedar-gated with the +/// per-graph `stream_manage` action — an actor with no permit rule is denied, +/// and a missing actor with a policy installed fails hard rather than silently /// skipping enforcement. #[tokio::test] async fn set_streaming_enabled_as_denies_when_policy_rejects_actor() { @@ -447,12 +447,26 @@ async fn set_streaming_enabled_as_denies_when_policy_rejects_actor() { } } -/// The permitted-actor arm: an explicit `admin` rule lets the operator flip -/// the flag, proving the enforce scope is `(Admin, ResourceScope::Graph)` — -/// not accidentally branch-scoped. +/// The permitted-actor arm: an explicit `stream_manage` rule lets the operator +/// flip the flag, proving the enforce scope is `(StreamManage, +/// ResourceScope::Graph)` — not accidentally branch-scoped. +/// +/// The negative half is load-bearing for the P1→2a migration: a policy that +/// grants only the old `admin` action must NOT reach the flip any more, or the +/// action split would be cosmetic. #[tokio::test] async fn set_streaming_enabled_as_allows_when_policy_permits_actor() { - const ADMIN_POLICY_YAML: &str = r#" + const STREAM_MANAGE_POLICY_YAML: &str = r#" +version: 1 +groups: + operators: [act-allowed] +rules: + - id: operators-stream-manage + allow: + actors: { group: operators } + actions: [stream_manage] +"#; + const ADMIN_ONLY_POLICY_YAML: &str = r#" version: 1 groups: operators: [act-allowed] @@ -465,7 +479,7 @@ rules: let dir = tempfile::tempdir().unwrap(); let db = init_and_load(&dir).await; let policy_path = dir.path().join("policy.yaml"); - fs::write(&policy_path, ADMIN_POLICY_YAML).unwrap(); + fs::write(&policy_path, STREAM_MANAGE_POLICY_YAML).unwrap(); let engine = PolicyEngine::load_graph(&policy_path, dir.path().to_str().unwrap()).unwrap(); let db = db.with_policy(Arc::new(engine) as Arc); @@ -475,4 +489,17 @@ rules: .expect("permitted operator flips the flag"); assert!(result.changed); assert!(result.streaming_enabled); + + // Granting only the old `admin` action no longer authorizes the flip. + let admin_dir = tempfile::tempdir().unwrap(); + let admin_db = init_and_load(&admin_dir).await; + let admin_policy_path = admin_dir.path().join("policy.yaml"); + fs::write(&admin_policy_path, ADMIN_ONLY_POLICY_YAML).unwrap(); + let admin_engine = + PolicyEngine::load_graph(&admin_policy_path, admin_dir.path().to_str().unwrap()).unwrap(); + let admin_db = admin_db.with_policy(Arc::new(admin_engine) as Arc); + assert_denied( + admin_db.set_streaming_enabled_as(true, Some("act-allowed")).await, + "admin-only policy must not authorize a stream_manage flip", + ); } diff --git a/docs/dev/canon.md b/docs/dev/canon.md index 3afb59c2..ac2f5b4c 100644 --- a/docs/dev/canon.md +++ b/docs/dev/canon.md @@ -516,8 +516,9 @@ exact one-generation fold as a guarded HEAD/witness transition. Private B2a added no format or transition: it established the unbounded retain-all posture, with no canonical durable MemWAL deletion. -Current schema v9/config-v3/state-v2/recovery-v12 implements the private -compare-and-chain row/fold subset. Canonical payload and token digests plus +Schema v9/config-v3/state-v2/recovery-v12 implements the private +compare-and-chain row/fold subset, and current schema v10 adds graph-global +enablement. Canonical payload and token digests plus trusted hidden row metadata bind each admission. A pre-wait authority capture is provisional; the adapter recaptures lifecycle/binding/HEAD/token authority after shared admission and same-key queue ownership. The manifest-selected @@ -526,12 +527,14 @@ transactions owned by one v12 intent, and their sole manifest CAS also stores the fold-attribution commitment. V11 is historical only. Managed reclamation remains a deferred Lance-owned optimization, not an activation gate. -The public/control portion remains inactive: explicit production enrollment, -durable `OPEN -> DRAINING -> SEALED -> OPEN` operations, bounded -`REPLACE`/`WITHDRAW` correction, authoritative status, authorization, and every -public row surface. Phase D owns future automatic operation-scoped drain and -atomic witness advancement/rebind. Resume must recheck the bounded narrow -main-only topology; an incompatible branch operation stays `SEALED`. +The graph-scoped Cedar vocabulary, `stream_manage`-gated enablement, and +embedded manifest-only read-only status are active. Explicit production +enrollment, durable `OPEN -> DRAINING -> SEALED -> OPEN` operations, bounded +`REPLACE`/`WITHDRAW` correction, exclusive-cut physical status, every public +row surface, and CLI/HTTP/OpenAPI parity remain inactive. Phase D owns future +automatic operation-scoped drain and atomic witness advancement/rebind. Resume +must recheck the bounded narrow main-only topology; an incompatible branch +operation stays `SEALED`. --- @@ -962,7 +965,9 @@ on the proposer.) initializer for recoverable empty enrollment; private Phase B1 now consumes its bounded data/ack/replay mechanics rather than building one, and private B2 adds durable compare-and-chain/token-fold authority over those primitives. - Public B2 remains inactive. The selected profile forbids OmniGraph from deleting raw + Public row admission and lifecycle mutation remain inactive; the Cedar + vocabulary, enablement flip, and embedded manifest-only status are active. + The selected profile forbids OmniGraph from deleting raw `_mem_wal` paths and accepts monotonic storage plus loud provider exhaustion; a future managed-reclamation profile would require the missing operation to live in Lance — **(draft RFC-026)**. @@ -1054,7 +1059,7 @@ receipt and complete physical-output envelope remain documented limits on any future bounded-storage claim, not blockers for this profile. B2a itself activated no schema or product surface. -**Implemented private B2 token/fold core; public controls remain inactive:** +**Implemented private B2 token/fold core; row and lifecycle controls remain inactive:** Internal schema v9/config-v3/state-v2 adds canonical payload and token digests, trusted hidden row metadata, exact compare-and-chain/idempotency classification, same-generation token overlays, and the manifest-selected graph-global @@ -1066,10 +1071,14 @@ fold attribution. Recovery-v11 is historical only. The genuine v8↔v9 gate proves two-way refusal and strict rebuild fidelity while hidden trusted metadata remains non-exported. -Explicit production enrollment, persistent lifecycle/correction with monotonic -revisions and bounded terminal receipts, authoritative status, authorization, -and product parity remain inactive. The retain-all profile adds no storage -admission watermark. Managed reclamation remains deferred Lance-owned work. +The graph-scoped `stream_ingest` / `stream_manage` Cedar vocabulary is +registered, the v10 enablement writer is gated by `stream_manage`, and embedded +`Omnigraph::stream_status` projects durable manifest authority read-only. +Explicit production enrollment, persistent lifecycle mutation/correction with +monotonic revisions and bounded terminal receipts, exclusive-cut physical +status, public row admission, and CLI/HTTP/OpenAPI parity remain inactive. The +retain-all profile adds no storage admission watermark. Managed reclamation +remains deferred Lance-owned work. Two checked-in RC.1 guards prove generic cleanup does not reclaim `_mem_wal` and deleting the successor's empty WAL fence sentinel can let a @@ -1088,8 +1097,10 @@ shape)**; public MemWAL row admission and later lifecycle/read phases **(RFC-026, draft; private v9 compare-and-chain/token-fold core and unbounded retain-all profile implemented; managed reclamation is optional later work; public strict -activation still requires explicit enrollment, correction/status, lifecycle, -authorization, and product-parity evidence)**; +activation still requires explicit enrollment, lifecycle mutation/correction, +exclusive-cut physical status, row admission, and transport-parity evidence; +the Cedar vocabulary, `stream_manage`-gated enablement, and embedded +manifest-only status are active)**; lineage-based merge deltas **(RFC-027, research-blocked)**; background reconciler; planner statistics/cost model; policy pushdown; ingest-time embeddings; per-query @@ -1107,7 +1118,7 @@ resource budgets. | **R4: Manifest authority access grows with commit count.** Current-state resolution folds history; a selective index does not by itself bound the complete physical read. | Medium | `optimize` compacts internal tables (keeps periodically-optimized shipped paths flat where separately cost-gated). RFC-024 Gate A rejected durable heads because representative RustFS latest-manifest reads/bytes grow despite flat exact-BTREE row/range work. RFC-025 Gate 0 independently rejected checkpoint-registry activation: at local 10→1,000 on RC.1, uncompacted reconciled work and the eight-fragment tail stay flat, but compacted list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; exact-show bytes and operation counts also grow. Both RFCs are research-blocked; v8 retains the journal fold and internal-table *cleanup* remains deferred behind the resurrection watermark. | | **R5: Schema identity corruption or alias/identity drift.** Internal schema v5 introduced stable IDs/incarnation as durable authority; v6, v7, and v8 preserve them. | Medium | Open/init validate the SchemaIR domain and exact bidirectional IR↔manifest identity/path/alias contract; every active recovery envelope carries the identity pair; zero, duplicate, missing, or mismatched identity fails closed. | | **R6: Merge cost at divergence** — full-width classification and history-growing manifest folds. | Medium | Coherent coordinator scans plus retained probe handles reduced the pre-slice measured depth-5/depth-80 baseline from 59/651 manifest reads to 40/410 and cap the common fast-forward route at three internal opens and three scans, but the uncompacted-history slope remains. `merge_cost.rs` keeps both facts visible; O(delta) merge is blocked on a real deletion-delta source **(RFC-027)**; fragment adoption is **(draft RFC-0001)**. | -| **R7: No public streaming row path** — production writes are still capped by the `graph_head` CAS rate; high-frequency small writes remain wasteful outside the private evidence seam. | Medium | MemWAL is the strategic substrate. Phase A makes its opaque initializer + separate shard effect recoverable for one main-only/unsharded/single-live-writer-process empty enrollment. The bounded worker provides watcher-plus-post-fence acknowledgement and conservative replay; its legal high-entropy near-cap shape closes under logical dense-slice accounting, with physical RSS guarded only as a remeasurement tripwire. Private B2a selects unbounded retain-all with no canonical MemWAL deletion or storage quota. Current v9 adds canonical compare-and-chain tokens, trusted attribution, post-admission authority recapture, graph-global token authority, and recovery-v12 exact base+token publication with durable fold attribution; genuine v8↔v9 refusal/rebuild is green. Public activation remains closed on explicit enrollment, lifecycle/correction/status, authorization, and product parity. Managed reclamation and a whole-root history budget are optional later work; a public exact enrollment receipt plus reversible admission seal gates broader overlapping-process topology. | +| **R7: No public streaming row path** — production writes are still capped by the `graph_head` CAS rate; high-frequency small writes remain wasteful outside the private evidence seam. | Medium | MemWAL is the strategic substrate. Phase A makes its opaque initializer + separate shard effect recoverable for one main-only/unsharded/single-live-writer-process empty enrollment. The bounded worker provides watcher-plus-post-fence acknowledgement and conservative replay; its legal high-entropy near-cap shape closes under logical dense-slice accounting, with physical RSS guarded only as a remeasurement tripwire. Private B2a selects unbounded retain-all with no canonical MemWAL deletion or storage quota. Schema v9 adds canonical compare-and-chain tokens, trusted attribution, post-admission authority recapture, graph-global token authority, and recovery-v12 exact base+token publication with durable fold attribution; genuine v8↔v9 refusal/rebuild is green. V10 adds graph enablement, now gated by the registered `stream_manage` action, and embedded manifest-only status is read-only. Public row activation remains closed on explicit enrollment, lifecycle mutation/correction, exclusive-cut physical status, and CLI/HTTP/OpenAPI parity. Managed reclamation and a whole-root history budget are optional later work; a public exact enrollment receipt plus reversible admission seal gates broader overlapping-process topology. | | **R8: Some operations lack enforced memory/time budgets.** | Medium | Known gap, narrowed and accepted for RFC-023. Its direct-substrate instrument rejected the first whole-delta fenced adopt (~447 MB peak at 100K × 256 versus ~74 MB Append), and the first corrected production 10K series failed at 30.0× / 108,625,920 bytes overhead; both negative results remain evidence. Mutation/Load now refuses a keyed table above 8,192 rows / 32 MiB before arm, while BranchMerge uses a recovery-enrolled chain with the same per-chunk bounds and a 1,024-transaction ceiling. The inductive certificate route removes the general diff, temporary delta, target preflight, and target join without weakening that chain. Final five-pair production medians passed at 31/8 ms (3.875×) for 10K and 136/35 ms (~3.886×) for 100K; maximum signed paired RSS overheads were 24,297,472 and 32,604,160 bytes. Inclusive row/transaction ceilings, byte refusal (including materialized blobs), operation-wide validation retention, exact source/target incarnation revalidation, second-generation certificate composition, and both between-chunk recovery directions are pinned; other operations still need explicit bounds. | | **R9: Local-FS conditional-write emulation** (`write_text_if_match` check-then-act gap). | Low | All current callers sit behind the cluster lock protocol; S3 uses true conditional puts; close before admitting any lock-free caller. | | **R10: Doc/spec drift as the system grows** — this document included. | Low | Maintenance contract (same-PR doc updates, `check-agents-md.sh` link CI, "don't lie" stale markers); this canon defers to area docs by construction. | @@ -1141,9 +1152,11 @@ Live design questions, each owned by an RFC or a known gap — not a wishlist: Phase B1 assigned the data-bearing config-v2/recovery-v11 core to v8. The current private B2 row/fold slice assigns config-v3/state-v2, graph-global token authority, and recovery-v12 to v9; the genuine v8↔v9 gate is green. - Lifecycle management, correction/status, and public surfaces can build on - that format only when their own gates pass; another durable shape change - would require a later strand. + The registered Cedar vocabulary and embedded manifest-only status build on + that format without changing it; lifecycle mutation/correction, + exclusive-cut physical status, row admission, and transport surfaces still + require their own gates. Another durable shape change would require a later + strand. RFC-024's heads, RFC-025's retention, and later RFC-026 phases remain independently reviewable. Any later format activation requires its own export/init/load rebuild unless capabilities deliberately co-release after @@ -1176,7 +1189,7 @@ The plan of record is the RFC-022…028 family (all under | [0023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | Substrate-native keyed-write fencing via Lance's unenforced-PK filter; fleet/format activation barrier | **Implemented** (2026-07-15) | | [0024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | Materialized head-row research; the first exact-BTREE candidate bounded scan work but failed the full latest-manifest/object-byte cost gate | **Research blocked** | | [0025 — Checkpoint-pinned retention](../rfcs/0025-checkpoint-retention.md) | Named checkpoints as authoritative retention roots, materialized as Lance tags; current in-manifest registry lookup rejected by Gate 0 | **Research-blocked** | -| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: bounded watcher-plus-post-fence acknowledgement, canonical compare-and-chain tokens and trusted hidden attribution, then recovery-v12 exact base+token graph-atomic fold; legal near-cap closure and genuine v8↔v9 rebuild are green; unbounded retain-all forbids OmniGraph MemWAL deletion; lifecycle/correction/status and product contracts remain inactive | **Draft; private B2 token/fold core and retain-all profile implemented; public inactive** | +| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: bounded watcher-plus-post-fence acknowledgement, canonical compare-and-chain tokens and trusted hidden attribution, then recovery-v12 exact base+token graph-atomic fold; legal near-cap closure and genuine v8↔v9 rebuild are green; unbounded retain-all forbids OmniGraph MemWAL deletion; graph enablement, registered Cedar vocabulary, and embedded manifest-only status are active, while row admission, lifecycle mutation/correction, exclusive-cut status, and transport surfaces remain inactive | **Draft; private B2 token/fold core and retain-all profile implemented; activation P1 plus authorization/status slice implemented; no public row path** | | [0027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | O(delta) merge classification from row-version lineage | Research-blocked | Deliberately split, not one mega-format: identity, key fencing, head rows, diff --git a/docs/dev/invariants.md b/docs/dev/invariants.md index 9383c93b..b1d2051e 100644 --- a/docs/dev/invariants.md +++ b/docs/dev/invariants.md @@ -196,7 +196,7 @@ converge the physical state. | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) | | Keyed writes | Every current v10 node/edge table declares exact non-null physical `id` as Lance's unenforced PK from creation, using the v6-introduced and later-preserved exact-`id` fence. General production strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint it only when completed effect statistics prove one attempt inserted every source row with zero updates, deletes, or skipped duplicates. Certificate admission is optional: BranchMerge uses the shortcut only when every transaction in the complete contiguous source interval carries v1 and its persisted operation is the full pure-insert `Update` shape (exact parent, no removed or updated fragments, nonempty new fragments with `physical_rows`, no field or generation rewrites, `RewriteRows`, exact-`id` filter, full nested schema preorder, and matching physical-row total). It then rechecks both source and target native ref incarnations and passes owned batches through an opaque capability whose one production mint site is structurally guarded. The proven publisher stages immutable fragments with `InsertBuilder`, replaces the uncommitted Append operation with that filter-bearing `Update`, and performs zero target preflights, target merge joins, or committed Appends. Missing, cleaned, unknown, or malformed proof uses the general ordered diff. Mutation/Load remains one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before arm; BranchMerge keeps its bounded v4 chain and exact-recovery limits. Raw Lance graph writers are unsupported, and the certificate is an internal non-cryptographic capability rather than an authenticity mechanism. The final production cost gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes) | [RFC-023](../rfcs/0023-key-conflict-fencing.md), [writes.md](writes.md), [execution.md](execution.md) | | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) | -| Streaming ingest | RFC-026 adopts Lance MemWAL as the strategic substrate and remains Draft; public row streaming is inactive. Historical schema v7 implemented recoverable main-only empty enrollment, and historical v8/config-v2/recovery-v11 added the private B1 worker and one-generation fold mechanics. Current schema v9/config-v3/state-v2 activates the private B2 token/fold slice behind the feature-gated, doc-hidden seam: canonical payload and compare-and-chain token digests bind table identity, logical key, accepted schema, stream incarnation, predecessor, write ID, trusted contributor, and payload; hidden trusted metadata follows the row; and same-generation overlays preserve idempotency and chaining before fold. Admission treats any pre-wait capture as provisional, then recaptures schema, lifecycle, binding, HEAD, and token authority after shared admission and same-key queue ownership, before `put_no_wait`. The graph-global `_stream_tokens.lance` dataset is durable but its raw HEAD is never authority; `__manifest` selects its exact witness. Recovery-v12 owns exact pre-minted base and token transactions, and only their exact joint outcome may publish both pointers, lifecycle state-v2, lineage, and the fold-attribution commitment in one manifest CAS. Recovery-v11 is historical only and is refused under v9 because it lacks the token participant and complete state-v2 outcome. Admission, replay, and fold share the existing 8,192-row/32-MiB logical dense-slice Arrow gate; backing-buffer capacity and physical RSS are evidence, not admission authority. The selected profile remains unbounded retain-all: no retained-byte, object-count, file-count, or history quota; no deletion of a canonical durable `_mem_wal` object; typed provider failures; and inert complete/partial residue. The genuine v8↔v9 gate proves two-way refusal and export/init/load rebuild while preserving logical rows, vectors, and exact-`id` PK metadata; hidden trusted stream metadata is not exported. Explicit production enrollment, revisioned lifecycle management, correction, authoritative status, authorization, SDK/HTTP/CLI/OpenAPI, generation GC, and fresh reads remain inactive. | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | +| Streaming ingest | RFC-026 adopts Lance MemWAL as the strategic substrate and remains Draft; public row streaming is inactive. Historical schema v7 implemented recoverable main-only empty enrollment, and historical v8/config-v2/recovery-v11 added the private B1 worker and one-generation fold mechanics. Schema v9/config-v3/state-v2 activates the private B2 token/fold slice behind the feature-gated, doc-hidden seam, and current schema v10 adds the required graph-global `stream_profile` enablement singleton: canonical payload and compare-and-chain token digests bind table identity, logical key, accepted schema, stream incarnation, predecessor, write ID, trusted contributor, and payload; hidden trusted metadata follows the row; and same-generation overlays preserve idempotency and chaining before fold. Admission treats any pre-wait capture as provisional, then recaptures schema, lifecycle, binding, HEAD, and token authority after shared admission and same-key queue ownership, before `put_no_wait`. The graph-global `_stream_tokens.lance` dataset is durable but its raw HEAD is never authority; `__manifest` selects its exact witness. Recovery-v12 owns exact pre-minted base and token transactions, and only their exact joint outcome may publish both pointers, lifecycle state-v2, lineage, and the fold-attribution commitment in one manifest CAS. Recovery-v11 is historical only and is refused under v9/v10 because it lacks the token participant and complete state-v2 outcome. Admission, replay, and fold share the existing 8,192-row/32-MiB logical dense-slice Arrow gate; backing-buffer capacity and physical RSS are evidence, not admission authority. The selected profile remains unbounded retain-all: no retained-byte, object-count, file-count, or history quota; no deletion of a canonical durable `_mem_wal` object; typed provider failures; and inert complete/partial residue. The genuine v8↔v9 gate proves two-way refusal and export/init/load rebuild while preserving logical rows, vectors, and exact-`id` PK metadata; hidden trusted stream metadata is not exported. The graph-scoped `stream_ingest` / `stream_manage` Cedar vocabulary is registered, the v10 enablement flip is gated by `stream_manage`, and embedded `Omnigraph::stream_status` exposes a read-only durable projection from one manifest snapshot. Explicit production enrollment, revisioned lifecycle mutation/correction, exclusive-cut physical status, public row ingest, SDK row/control methods, HTTP/CLI/OpenAPI streaming surfaces, generation GC, and fresh reads remain inactive. | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | | Branch create/delete | `__manifest` `BranchContents` is the single logical authority. Lance create is physically two-phase, so OmniGraph prevalidates names, enforces path-prefix-disjoint live graph names, reclaims an absent-ref clone-only tree, and uses a bounded completion classifier; delete removes authority before tree cleanup, so an absent ref is success and derived tree reclaim may converge later. Neither control emits graph lineage. Under schema/source-target/all-table gates, each control uses one operation-local post-gate manifest/namespace capture rather than refreshing the handle-local coordinator around table-gate acquisition; successful ref movement explicitly invalidates derived read caches. Per-table forks are derived state, reclaimed best-effort with `cleanup` as backstop. A target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Cleanup retention | Explicit cleanup derives exact `keep` cutoffs from Lance's available version list, caps each main-table GC cutoff at the oldest exact main version inherited by any live lazy graph branch, and refuses uncovered main HEAD drift. Lance protects native per-table branch refs itself. The graph-wide live-reference preflight fails closed before the first table GC, after which individual table failures remain fault-isolated | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Optimize visibility | One operation-local accepted catalog and fresh main snapshot are planned under schema → main → all-table gates. Every productive table shares one identity-bearing v9 recovery envelope; bounded-parallel physical effects become visible through at most one monotonic manifest/lineage CAS. Complete crash residuals roll forward together and partial residuals compensate before visibility. Optimize's payload remains a bounded maintenance adapter rather than an exact caller-minted Lance transaction proof, so it is supported within the single-writer-process recovery boundary. Exact provenance is deferred until Lance exposes a stable public caller-controlled maintenance transaction API and OmniGraph has distributed recovery fencing | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md), [RFC-022](../rfcs/0022-unified-write-path.md) | @@ -334,11 +334,15 @@ them explicit. complete-output envelope are consequently not blockers for unbounded retain-all. They remain relevant to the future B2b Lance-owned inspect/plan/execute, checkpoint, inventory/accounting, and reclamation - design. Explicit production enrollment, lifecycle management receipts, - strict correction/status, authorization, and product parity remain required - before a public profile; + design. Explicit production enrollment, lifecycle-management receipts, + strict correction, exclusive-cut physical status, row admission, and + transport parity remain required before a public profile. The + `stream_ingest` / `stream_manage` Cedar vocabulary is already registered, + `stream_manage` gates the enablement writer, and the embedded SDK already + exposes durable manifest-only status; a `GraphHistoryBudget` or retained-storage watermark belongs only to a future - bounded/managed profile, not this one. Every product surface remains inactive. + bounded/managed profile, not this one. CLI/HTTP/OpenAPI streaming surfaces + remain inactive. The exact upstream receipt and cross-process seal remain required to broaden topology, not to run the existing private seam inside its current process boundary. See diff --git a/docs/dev/rfc-022-027-architecture-review.md b/docs/dev/rfc-022-027-architecture-review.md index 86618d37..de753358 100644 --- a/docs/dev/rfc-022-027-architecture-review.md +++ b/docs/dev/rfc-022-027-architecture-review.md @@ -4,10 +4,11 @@ RFC-027 research-blocked; RFC-026 Phase A implemented, private B1's legal near-cap closure repaired and green, private B2a unbounded retain-all gate implemented, the private common-B2 compare-and-chain/token-fold core -implemented, B2b optional, and public streaming plus lifecycle -management/correction/status inactive +implemented, B2b optional, activation P1 plus the Cedar vocabulary and embedded +manifest-only status implemented, and public row streaming, lifecycle +mutation/correction, exclusive-cut status, and transport surfaces inactive **Date:** 2026-07-11 -**Last updated:** 2026-07-22 +**Last updated:** 2026-07-28 **Audience:** RFC authors, engine/storage maintainers, and release reviewers **Reviewed against:** OmniGraph 0.8.1; Lance 9.0.0-beta.15 at `f24e42c11a742581365e1cbe17c906ea2dac1bc6`; full Lance transaction, diff --git a/docs/dev/testing.md b/docs/dev/testing.md index d42f5d98..30dd1a78 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -24,7 +24,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `merge_fast_forward.rs` | Branch-adopt cost + correctness under RFC-023. The one-batch and 8,193-row fixtures prove that a complete v1 insertion-absence history chain publishes one/two bounded exact-`id` filtered `Update` transactions with zero target strict-insert preflights, target MergeInsert joins, committed Appends, ordered-cursor scans, or whole-delta staged combines. `pure_insert_fast_forward_retains_value_constraint_validation` proves the certificate skips only redundant key work, not logical row constraints; its all-new Upsert source is certified from completed effect statistics. `proven_fast_forward_certificate_composes_across_merge_generation` proves the publisher re-mints v1 and a second merge consumes that output as the next proof-chain link. A missing intermediate transaction proves cleaned history is an optimization miss: the merge enters the general ordered diff, preserves exact rows, and leaves no recovery residue. `lazy_target_ref_only_fast_forward_uses_pin_after_main_advances` distinguishes a valid old lazy graph pin from drift when the inherited main ref advances. A nested `main → feature → experiment` cell prevents a deeper valid `BranchIdentifier` from becoming a false read-set conflict. Every general-route base/source/target `OrderedTableCursor` scan applies both Lance `batch_size(8,192)` and `batch_size_bytes(32 MiB)`. Validation streams projected `id`/`src`/`dst`/scalar batches, charges exact Arrow memory before retention, and shares one 32 MiB operation-wide budget across candidate tables; `branch_merge_validation_delta_is_aggregate_bounded_pre_arm` crosses it with two individually valid ~18 MiB deltas while proving zero HEAD/manifest/lineage/sidecar movement. Deletes use exact escaped-filter chunks with the same row/byte and retained-plan bounds. Production-helper unit cells pin chain/delete/recovery limits. The subprocess scenario owns the final production latency/RSS evidence; these integration tests own route semantics, not timings | | `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); RFC-023 pins the inclusive 8,192-row keyed input ceiling, the same exact/+1 boundary on streamed mutation-update matches, no-effect state for both refusals, and oversized stored-Blob rejection before payload read. Crate-internal pending-scan cells pin inclusive/+1 32 MiB accounting plus pending-key shadow-before-charge. The lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) | | `src/table_store/staged_tests.rs` | Crate-internal staged primitives. RFC-023 pins one exact target preflight for general StrictInsert, durable v1 mint/commit/reopen/history persistence, exact-`id` filter emission, typed `KeyConflict`, and missing/wrong PK refusal. `all_new_upsert_certifies_insert_absence_and_persists_it_in_history` proves an all-new completed Upsert receives the optional certificate, a mixed/update Upsert does not, unrelated transaction properties survive, and UUID rebinding does not erase it. Proven-insert cells show the opaque path performs zero strict preflights; stages with `InsertBuilder` but commits the full pure-insert `Update` shape (exact parent and `id` filter, `RewriteRows`, no updates/removals, full nested schema preorder, physical rows); persists/re-admits its own output for proof composition; leaves new fragments outside old index coverage; and fails same-key races loudly in proven/proven and proven/general orders. The in-source `exec/merge.rs` certificate unit table rejects missing/unknown properties, wrong parent/filter/full-preorder/mode/offsets, rewrite/removal shapes, missing `physical_rows`, and Append. Source-interval cells pin exact selection, lazy retained-parent splitting, coalescing, and pinned Lance's approximate raw-emission boundary while every normalized/writer chunk remains hard-capped. Generic `stage_append`/`stage_merge_insert` remain primitive tests only. The file also owns index staging and `commit_staged{,_exact}` | -| `forbidden_apis.rs` | Defense-in-depth syntax-tree/source guard over the whole engine. The primary boundary is Rust visibility: raw storage/coordinator/handle-cache modules are crate-private; public `Snapshot::open` returns `SnapshotTable`; and `SnapshotScanner` executes reads without exposing Lance's raw scanner or physical plan. The guard pins those visibility/return-type boundaries, classifies public async inherent `Omnigraph` methods plus loader conveniences, classifies every crate-visible async method on `GraphCoordinator` / `ManifestCoordinator`, and exact-counts registered method/UFCS durable-call shapes including recovery. RFC-023 rejects production graph call sites of generic `stage_append{,_stream}` and `proven_insert_capability_has_one_production_mint_site` pins `ProvenInsertChunk::from_verified_history` to the complete-history classifier in `exec/merge.rs`, preventing the no-preflight capability from becoming a reusable bypass. At the RFC-026 Phase-A checkpoint the guard registered only the exact v10 enrollment gateway and feature-gated test seam, counted its sidecar/index/shard durability primitives, and kept every row-put/ack/fold surface absent; the Phase-B1 owner below now exact-counts only the approved crate-private put/fold durable-call sites while retaining the absence of public schema, SDK, HTTP, CLI, and OpenAPI side doors. B2a inventories the only production `_mem_wal` literal owners, forbids MemWAL reclamation/adoption symbols and destructive primitives in the adapter, keeps generic maintenance unaware of MemWAL, and keeps raw inventory/classifier helpers private. This remains defense in depth rather than macro expansion, alias, or data-flow analysis; the visibility boundary and behavior tests are still primary. It also counts selected raw `SnapshotHandle` / Dataset shapes, rejects renamed-owner/macro/include/path-lookalike forms, skips structurally test-only code, and pins retired escape hatches absent. `// forbidden-api-allow: ` exempts reviewed inline-Lance lines only | +| `forbidden_apis.rs` | Defense-in-depth syntax-tree/source guard over the whole engine. The primary boundary is Rust visibility: raw storage/coordinator/handle-cache modules are crate-private; public `Snapshot::open` returns `SnapshotTable`; and `SnapshotScanner` executes reads without exposing Lance's raw scanner or physical plan. The guard pins those visibility/return-type boundaries, classifies public async inherent `Omnigraph` methods plus loader conveniences, classifies every crate-visible async method on `GraphCoordinator` / `ManifestCoordinator`, and exact-counts registered method/UFCS durable-call shapes including recovery. RFC-023 rejects production graph call sites of generic `stage_append{,_stream}` and `proven_insert_capability_has_one_production_mint_site` pins `ProvenInsertChunk::from_verified_history` to the complete-history classifier in `exec/merge.rs`, preventing the no-preflight capability from becoming a reusable bypass. At the RFC-026 Phase-A checkpoint the guard registered only the exact v10 enrollment gateway and feature-gated test seam, counted its sidecar/index/shard durability primitives, and kept every row-put/ack/fold surface absent; the Phase-B1 owner below now exact-counts only the approved crate-private put/fold durable-call sites while retaining the absence of public row/control schema, SDK, HTTP, CLI, and OpenAPI side doors. The embedded SDK's read-only `stream_status` is classified separately and must remain free of durable calls. B2a inventories the only production `_mem_wal` literal owners, forbids MemWAL reclamation/adoption symbols and destructive primitives in the adapter, keeps generic maintenance unaware of MemWAL, and keeps raw inventory/classifier helpers private. This remains defense in depth rather than macro expansion, alias, or data-flow analysis; the visibility boundary and behavior tests are still primary. It also counts selected raw `SnapshotHandle` / Dataset shapes, rejects renamed-owner/macro/include/path-lookalike forms, skips structurally test-only code, and pins retired escape hatches absent. `// forbidden-api-allow: ` exempts reviewed inline-Lance lines only | | `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `cached_and_zero_cache_sessions_share_store_registry_not_metadata_cache` proves a cached data Session and zero-cache control Session reuse one live `ObjectStoreRegistry` client while their metadata caches remain isolated. `_compile_uncommitted_full_table_vector_index_shape` pins the public `IndexMetadata` shape suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics. RFC-023's `unenforced_pk_filter_shape_is_route_dependent` explicitly forces v2 versus indexed routes and pins the `Some(populated)` / `Some(empty)` / `None` key-filter shapes; `unenforced_pk_conflict_matrix_is_directional` pins the directional filtered/unfiltered and filtered/Append matrix. RFC-024's compile guard pins the public `BranchIdentifier` + current table version + current `Transaction.uuid` + `ManifestLocation.e_tag` current-HEAD witness; the local/shared-`Session` guard proves unchanged-reopen stability, ordinary-commit movement, and same-version ABA, while RustFS covers object-store ABA. RFC-025 adds exact main/named-branch tag-target, sparse cleanup pin/unpin, and branch-tree-deletion guards. RFC-026 pins doc-hidden `has_successor_version`, initializer/readback/shard-writer/durability/fencing, flush/drain, replay watermark, scanner, and merged-generation shapes; runtime Gate E0 classification belongs to `memwal_enrollment_gate.rs`, while v7 Phase-A and v8 B1 publication/recovery belong to the manifest/failpoint suites. B2b adds `cleanup_old_versions_does_not_reclaim_mem_wal_objects` and `mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance`: the first proves generic cleanup leaves the present MemWAL fixture unchanged and the second proves deleting the successor's empty fence sentinel is unsafe. The pinned source audit, not those two tests alone, establishes that stock RC.1 exposes no owned MemWAL reclamation API. The RC.1 compiler guard pins the five surveyed public Lance virtual system-column constants to early `.pg` rejection. These guards prove substrate shapes/tokens and negative ownership boundaries; they do not by themselves prove heads/checkpoint activation, the current publisher, or a safe reclamation implementation | | `memwal_enrollment_gate.rs` | RFC-026's green production-neutral Gate E0 harness, isolated from the production manifest and graph writer. Fourteen substantive local cells plus one explicit unconfigured-S3 skip cover exact no-effect / `N + 1` index / pre-minted empty-shard classification, buried-effect refusal, marker survival, strict inventory/error handling, and the broad fail-closed matrix. The rejected first instrument used `checkout_latest` plus `IOTracker`, which missed local `read_dir`. The accepted exact-version classifier pins doc-hidden `has_successor_version`; its `AttemptTracker` records failed/`NotFound` attempts before forwarding and proves the identical complete six-attempt shape at baseline versions 8/80: four successful manifest HEADs, one `NotFound` manifest HEAD, one successful manifest GET, zero lists. A Unix execute-only `_versions` tripwire proves exact probing works when latest enumeration fails and an unreadable exact HEAD errors. The configured RustFS exact cell passes non-vacuously with the same zero-list shape and owns the positive lost-result/index/empty-shard/reopen sequence plus foreign shard, malformed/loose root, durable WAL, persisted cursor, and corrupt-manifest negatives. S3 ABA remains in `lance_surface_guards.rs`; CI rejects skipped E0/ABA cells. This file never mutates production manifest/schema state or deletes ambiguous artifacts; Phase A consumes its classifier through the private adapter | | `memwal_stream.rs` | Feature-gated RFC-026 private B1 mechanics, implemented private B2-common token/fold behavior, and B2a provider-failure evidence. B1 owns bounded put/ack/replay/fold, authority, cancellation, and manifest-only visibility. B2-common owns compare-and-chain/idempotency conflicts, same-generation overlays, stale-authority recapture after shared admission, exact base+token fold, recovery-v12 crash completion, and durable attribution. B2a injects a recording/failing store at the real Lance table-store boundary and covers effect-free cold-claim failure, post-invocation WAL ambiguity, complete post-cut unreferenced generation output, and partial generation output after the data object. Exact local and configured-RustFS cells prove typed `Lance` / `AckUnknown` / `RecoveryRequired` outcomes, blocked fresh progress, one fresh authoritative retry root, retained non-authoritative residue, and zero access below the orphan root through retry and cold reopen. Parent shard discovery may observe the common prefix; it may not descend into or adopt the subtree. | @@ -403,8 +403,9 @@ base/token refusal; `db/manifest/recovery.rs` owns the v12 two-participant effect matrix and confirmation. Manifest tests own the graph-global selected token pointer and fold-attribution publication. `lifecycle.rs` proves hidden trusted metadata and the MemWAL system index stay out of public reflection, -while `forbidden_apis.rs` proves no SDK/CLI/HTTP/OpenAPI streaming side door was -introduced. +while `forbidden_apis.rs` proves no SDK row/control or CLI/HTTP/OpenAPI +streaming side door was introduced; it classifies the embedded read-only +`stream_status` surface separately. Run the focused private B2-common owners with: @@ -417,8 +418,25 @@ cargo test -p omnigraph-engine --test lifecycle public_snapshot_wildcard_omits_p cargo test -p omnigraph-engine --test forbidden_apis ``` -B1/B2a and the private B2-common slice do not add parser/server/CLI tests because -they have no public surface. The +RFC-026 §4.7 P1 and 2a own the first public streaming surfaces, so their +evidence extends the existing owners rather than opening a new silo: +`policy_engine_chassis.rs` proves the `stream_manage` gate in both directions +(including that an `admin`-only policy no longer authorizes the enablement +flip after the 2a migration); `omnigraph-policy`'s in-source suite proves both +stream actions are graph-scoped and reject `branch_scope` / +`target_branch_scope`; `omnigraph-cli/src/planes.rs` parses the underscore +policy wire names and their kebab-case compatibility aliases through the real +Clap surface; `lifecycle.rs` owns read-only `stream_status` against a graph +with no lanes; and `memwal_stream.rs` owns it against an enrolled lane, +pinning both the `lifecycle_revision` compare token and the logical +`stream_incarnation_id` that fences every row request from a prior incarnation. +The feature-gated `stream_status` unit test deliberately makes the lifecycle's +diagnostic alias stale and proves status resolves the current registration by +immutable identity. `forbidden_apis.rs` classifies `stream_status` read-only — +the structural claim that status cannot move a lifecycle. + +The remaining B1/B2a and private B2-common work adds no parser/server/CLI +tests because it has no public surface. The common product contract inventory specifies the missing contracts without activating them. Any public implementation must extend the existing compiler, server/OpenAPI, CLI parity, Cedar, shutdown, @@ -491,8 +509,10 @@ local/RustFS matrix validates that bound across schemas, fragmentation, crashes, and retries; measurement alone does not establish it. Until all of that is green, no B2b bounded/managed-reclamation route is active. This future matrix does not gate the selected unbounded retain-all profile; that profile remains -private because the common token, attribution, lifecycle, correction, -authorization, and product-parity contracts above are still inactive. A +private because public row admission, lifecycle mutation/correction, +exclusive-cut physical status, and transport-parity contracts above are still +inactive. The Cedar vocabulary, `stream_manage`-gated enablement, and embedded +manifest-only status are already active. A separate bounded-profile matrix initializes and validates the manifest-authoritative graph-global `GraphHistoryBudget`, then charges every manifest-writer class and its pending recovery sidecars through reserve, effect, @@ -851,7 +871,7 @@ Correctness bugs fail loudly in tests; cost-scaling bugs pass every test and deg - **Keep decision instruments honest when the answer is no.** RFC-024's `durable_head_lookup_cost.rs` attaches tracking before the cold dataset load through `open_tracked_lance_dataset`, then reports object-store wrapper I/O separately from Lance execution-summary I/O. Its reconciled BTREE row/range curve is flat, but its required RustFS cold-open and compacted-byte curves grow; those red design facts are asserted as the current result rather than erased because some counters pass. Run the default local 20/80 matrix with `cargo test -p omnigraph-engine --test durable_head_lookup_cost local_durable_head_lookup_matrix_is_correct_and_observable -- --exact --nocapture`; run the ignored 10/100/1,000 local matrix with `cargo test -p omnigraph-engine --test durable_head_lookup_cost local_durable_head_lookup_matrix_at_one_thousand_commits -- --ignored --exact --nocapture`. The bucket-gated S3 command is in the RustFS section above and remains on demand. - **Apply the same rule to RFC-025.** `checkpoint_retention_cost.rs` keeps live checkpoint count and catalog width fixed while unrelated journal history grows, and counts complete list/show/cleanup-root authority reads. The uncompacted reconciled counters and bounded tail are flat; compacted scan bytes and the 1,000-commit operation boundary are not, so the assertions preserve a no-go. Run the default local matrix with `cargo test -p omnigraph-engine --test checkpoint_retention_cost local_checkpoint_retention_matrix_is_exact_and_records_the_current_no_go -- --exact --nocapture`; run the ignored decision scale with `cargo test -p omnigraph-engine --test checkpoint_retention_cost local_checkpoint_retention_matrix_at_one_thousand_commits -- --ignored --exact --nocapture`. A green test means the known result was reproduced, not that RFC-025 passed Gate 0. - **Keep RFC-026 Gate E0 reproducible.** The first `checkout_latest`/`IOTracker` instrument was false-green because local `read_dir` escaped tracking; it is not acceptance evidence. The green harness uses the public but guide-hidden `Dataset::has_successor_version` from freshly ABA-verified exact `N`, probes only `N + 1`, then uses exact `N + 1` to reject buried `N + 2`. `AttemptTracker` records before forwarding, including failed/`NotFound` HEADs, and versions 8/80 must retain the identical four-success-HEAD + one-NotFound-HEAD + one-success-GET shape with zero lists. The Unix execute-only `_versions` tripwire must keep exact probing green while latest enumeration fails, and an unreadable exact HEAD must error. Run the 14-substantive-cell local file with `cargo test -p omnigraph-engine --test memwal_enrollment_gate -- --nocapture`; its fifteenth bucket-gated cell logs an explicit skip when unconfigured. Run the exact configured RustFS command above for its positive plus listing-dependent negative matrix. Green E0 authorized only Phase A; Phase A has now activated v7 foundation state, but E0 never authorizes row admission, acknowledgement, or fold. -- **Preserve RFC-026 Gate R0's closure and retain-all evidence.** Its current-object census is useful only inside its stated observation boundary; never label LIST totals as provider-retained or billed bytes, and never turn them into a quota claim. The source audit remains pinned to the exact Lance lockfile revision. The high-entropy cell must continue to reject 3,743 payload bytes per row effect-free at the exact B2-attributed 33,558,528-byte charge, admit 3,742 at 33,550,336 bytes, keep acknowledgement graph-invisible, close through logical-slice charge plus dense per-scanner-batch take, publish exactly once, and preserve every previously listed immutable path/class/size. Use `cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost gate_r0_ -- --nocapture`. Also run `widest_legal_generation_records_no_roll_estimates_and_peak_rss` exactly: its reference-environment paired fold peak-RSS lift was 286,441,472 bytes (about 273 MiB), and 384 MiB is a one-sided remeasurement tripwire for the single-exclusive-fold implementation shape, not a runtime hard allocator limit. A zero or negative paired lift is valid when common initialization dominates the lifetime high-water mark; the child separately asserts that the exact workload completed. A green run proves the private evidence only; public product contracts remain inactive. +- **Preserve RFC-026 Gate R0's closure and retain-all evidence.** Its current-object census is useful only inside its stated observation boundary; never label LIST totals as provider-retained or billed bytes, and never turn them into a quota claim. The source audit remains pinned to the exact Lance lockfile revision. The high-entropy cell must continue to reject 3,743 payload bytes per row effect-free at the exact B2-attributed 33,558,528-byte charge, admit 3,742 at 33,550,336 bytes, keep acknowledgement graph-invisible, close through logical-slice charge plus dense per-scanner-batch take, publish exactly once, and preserve every previously listed immutable path/class/size. Use `cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost gate_r0_ -- --nocapture`. Also run `widest_legal_generation_records_no_roll_estimates_and_peak_rss` exactly: its reference-environment paired fold peak-RSS lift was 286,441,472 bytes (about 273 MiB), and 384 MiB is a one-sided remeasurement tripwire for the single-exclusive-fold implementation shape, not a runtime hard allocator limit. A zero or negative paired lift is valid when common initialization dominates the lifetime high-water mark; the child separately asserts that the exact workload completed. A green run proves the private evidence only; it does not activate row admission, lifecycle mutation, full physical status, or transport surfaces. - **Preserve RFC-026 B2a's no-delete and provider-failure evidence.** Reuse the shared strict MemWAL classifier; do not weaken it with a second path parser. Complete and partial unreferenced generation roots must stay non-authoritative and receive zero subtree reads, writes, deletes, or adoption through retry/reopen. Parent shard discovery may observe their prefix. Canonical durable MemWAL deletion remains zero; only an exact losing shard-manifest-CAS `.binpb.tmp.` staging path is allowed. Run `cargo test -p omnigraph-engine --features failpoints --test memwal_stream provider_` and `cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost b2a_`. Run the ignored 1/8/32/128 local/RustFS sweeps when the retained-history shape or Lance pin changes. Keep every term separate and describe LIST bytes, wall time, and RSS as advisory diagnostics, never as a quota, SLO, provider bill, or isolated WAL slope. - **Count on the handle that does the reads, not just the one a measured op opens.** Lance's IO-counted tests attach the `IOTracker` to the (warm, cached) dataset and read `incremental_stats()` per request — the tracker MUST be on the handle performing the reads, or warm-handle reads escape. A per-op tracker installed at measure time cannot see reads on a long-lived handle opened earlier (the warm coordinator's `__manifest` handle, reused across writes), so such reads were silently undercounted. Wrap a depth-swept body in `cost_harness` so the manifest tracker is installed before the graph opens and `manifest_reads` is **ground truth** (handle-age-irrelevant). The `version_probes` counter is the freshness-probe *call* count; ground truth additionally reveals that a write's probe does ~3 object-store RPCs (a read's probe is a 0-IO cache hit). `manifest_reads_capture_warm_probe` is the guard that this stays true. - This is the testing companion to invariant 15 in [docs/dev/invariants.md](invariants.md) (hot-path cost is bounded by work, not history). diff --git a/docs/dev/writing-path-state-of-affairs.md b/docs/dev/writing-path-state-of-affairs.md index 97a17922..66197bb7 100644 --- a/docs/dev/writing-path-state-of-affairs.md +++ b/docs/dev/writing-path-state-of-affairs.md @@ -236,7 +236,7 @@ These operations do not create a side door around the protocol: | Recovery | Same-process handles share ordered queues for the same canonical local root or identical normalized opaque remote/custom URI. Stream-admission domains are acquired outside schema → branch → stream-token → table gates where applicable. Mutation/load, SchemaApply, BranchMerge, EnsureIndices, StreamFold, and `refresh` heal roll-forward-only; Optimize, Repair, and Cleanup refuse pending recovery, while branch controls use specialized barriers. Read-write open performs the quiesced full sweep, including exact v10 enrollment and v12 base+token fold completion; read-only open never repairs and refuses unresolved stream recovery. Historical v11 folds are refused under v9. A resolved intent is audited internally with the original actor when present; no-effect cleanup need not create graph lineage. | | RFC-026 Phase A (v7 foundation) | Internal schema v7 introduced identity-keyed lifecycle rows and exact empty main/unsharded enrollment. At that Phase-A-only boundary, any lifecycle row—including `SEALED`—rejected base-table, schema, maintenance, repair-adoption, and recovery effects under a process-local admission lease because no drain/fold witness-update adapter existed. Native branch create/delete alone could proceed at `SEALED` because it did not move table HEAD; `OPEN`/`DRAINING` refused it. Enrollment and open-time validation rejected named-branch overlap and any uncovered lifecycle/MemWAL mismatch. Schema v9 preserves these foundation guarantees. | | RFC-026 Phase B1 (historical private core) | Internal schema v8/config-v2 added one root-scoped, cross-handle serialized worker and a hard-bounded 8,192-row/32-MiB logical dense-slice Arrow no-roll generation. Watcher success proves durability; a clean `DurableBatchAck` additionally requires the same `ShardWriter::check_fenced()` to succeed immediately afterward. Fence loss, epoch-read failure, owner-task failure, or deadline ambiguity is post-invocation `AckUnknown` plus worker retirement. Reopen/replay is conservative and exact drain proof precedes quiesced abort. Gate R0's deterministic legal high-entropy near-cap cell exposed sparse scanner arrays; logical-slice charging plus dense copies repaired it. Physical RSS is evidence only. V9 preserves those worker/closure mechanics, but recovery-v11 itself is historical only. | -| RFC-026 private B2 token/fold core | Internal schema v9/config-v3/state-v2 adds canonical payload/token digests, hidden trusted row metadata, exact idempotency and compare-and-chain classification, same-generation token overlays, and post-admission authority recapture. `_stream_tokens.lance` is graph-global durable sequencing state selected only by its manifest witness. Recovery-v12 owns exact base+token transactions and atomically publishes both pointers, lifecycle, lineage, and durable fold attribution. The genuine v8↔v9 refusal/rebuild gate is green. The slice remains private; lifecycle management, correction/status, public enrollment, authorization, and product surfaces are inactive. | +| RFC-026 private B2 token/fold core | Internal schema v9/config-v3/state-v2 adds canonical payload/token digests, hidden trusted row metadata, exact idempotency and compare-and-chain classification, same-generation token overlays, and post-admission authority recapture. `_stream_tokens.lance` is graph-global durable sequencing state selected only by its manifest witness. Recovery-v12 owns exact base+token transactions and atomically publishes both pointers, lifecycle, lineage, and durable fold attribution. The genuine v8↔v9 refusal/rebuild gate is green. The row/fold slice remains private; the graph-scoped Cedar vocabulary, `stream_manage`-gated v10 enablement, and embedded manifest-only status are active, while lifecycle mutation/correction, exclusive-cut physical status, public enrollment/row admission, and CLI/HTTP/OpenAPI surfaces remain inactive. | | Lance access | One process-wide `ObjectStoreRegistry` reuses clients. Each `Omnigraph` handle owns its cached data-table `Session`; one process-wide zero-cache control `Session` opens mutable tips. Only the object-store registry is shared between the data and control sessions. This is “cache the past, never the present,” not one global cached session. | | Maintenance | EnsureIndices stages exact missing-index transactions. Optimize coordinates graph-wide compaction/index work under one bounded recovery envelope. Periodic optimize compacts `__manifest`, but unmaintained history-dependent paths are not globally flat. | @@ -261,7 +261,7 @@ retry rules, recovery classification, and the full owned limits. | [023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | **Implemented** | Internal schema v6 introduced exact-`id` PK metadata, closed keyed routing, typed conflicts, bounded replay, rebuild/refusal, and accepted performance evidence; v9 preserves that contract. | | [024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | **Research-blocked** | The first in-manifest BTREE candidate has a specified logical contract and flat indexed row/range work, but fails the complete physical-I/O gate. No head rows or heads format are active. | | [025 — Checkpoint retention](../rfcs/0025-checkpoint-retention.md) | **Research-blocked** | Lance tag/pin semantics pass, but the proposed in-manifest registry access shape is not history-flat after compaction. No checkpoint rows, `ogcp_` production tags, API, or cleanup integration are active. | -| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; private B2 token/fold core and retain-all profile implemented; public inactive** | Schema v9 preserves v7/recovery-v10 enrollment and the bounded B1 worker while activating config-v3/state-v2, canonical payload/token digests, trusted hidden attribution, manifest-selected `_stream_tokens.lance`, stale-authority revalidation after shared admission, and recovery-v12 exact base+token fold. One manifest CAS publishes both pointers, lifecycle, lineage, and fold attribution. Recovery-v11 is historical only. Gate R0's no-go still prohibits a finite storage promise, so the selected profile never deletes a canonical MemWAL object and sets no retained-byte/object/file/history quota. Genuine v8↔v9 refusal/rebuild is green. Explicit production enrollment, lifecycle management/correction/status, authorization, and all product surfaces remain inactive; B2b managed reclamation is optional future work. | +| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; private B2 token/fold core and retain-all profile implemented; activation P1 plus authorization/status slice implemented; no public row path** | Schema v9 preserves v7/recovery-v10 enrollment and the bounded B1 worker while activating config-v3/state-v2, canonical payload/token digests, trusted hidden attribution, manifest-selected `_stream_tokens.lance`, stale-authority revalidation after shared admission, and recovery-v12 exact base+token fold. One manifest CAS publishes both pointers, lifecycle, lineage, and fold attribution. Recovery-v11 is historical only. V10 adds graph enablement; the registered `stream_manage` action gates its writer, and embedded manifest-only status is read-only. Gate R0's no-go still prohibits a finite storage promise, so the selected profile never deletes a canonical MemWAL object and sets no retained-byte/object/file/history quota. Genuine v8↔v9 refusal/rebuild is green. Explicit production enrollment, lifecycle mutation/correction, exclusive-cut physical status, public row admission, and CLI/HTTP/OpenAPI surfaces remain inactive; B2b managed reclamation is optional future work. | | [027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | **Research-blocked** | The desired O(delta) classifier and fallback contract are specified. Selective live-row and deletion-delta discovery are not yet bounded, so `OrderedTableCursor` remains the correctness path. | | [028 — Stable schema identity](../rfcs/0028-stable-schema-identity.md) | **Implemented** | Rename-stable IDs, table incarnation, identity-derived paths, schema/recovery integration, and strict rebuild activation were introduced in v5 and remain active in v9. | diff --git a/docs/rfcs/0026-memwal-streaming-ingest.md b/docs/rfcs/0026-memwal-streaming-ingest.md index 3206a166..ae04a931 100644 --- a/docs/rfcs/0026-memwal-streaming-ingest.md +++ b/docs/rfcs/0026-memwal-streaming-ingest.md @@ -11,9 +11,11 @@ owner: OmniGraph maintainers # RFC-026 — MemWAL streaming ingest **Status:** Draft / private B2 compare-and-chain/token-fold core and B2a -unbounded retain-all profile implemented; public streaming and -lifecycle management/correction/status inactive; experimental activation -profile selected 2026-07-27 (§4.7), unimplemented +unbounded retain-all profile implemented; experimental activation P1, +graph-scoped Cedar vocabulary, and embedded manifest-only read-only status +implemented; public row streaming, explicit enrollment, lifecycle +mutation/correction, exclusive-cut physical status, and CLI/HTTP/OpenAPI +streaming surfaces inactive **Date:** 2026-07-10 **Gate E0 evaluated:** 2026-07-18 **Phase A foundation completed:** 2026-07-18 @@ -47,8 +49,9 @@ v9, stream-config v3, lifecycle state-v2, manifest-selected graph-global `_stream_tokens.lance`, canonical payload/token digests, post-admission authority recapture, recovery-v12 exact base+token fold, durable graph-commit attribution, and the genuine v8↔v9 refusal/rebuild gate passed (§11/§12.6); -explicit production enrollment, lifecycle management, correction/status, and -all product surfaces remain inactive +at that checkpoint explicit production enrollment, lifecycle management, +correction/status, and all product surfaces remained inactive (the +authorization/status slice below changed that boundary on 2026-07-28) **Experimental activation profile selected:** 2026-07-27 — cluster-only, manifest-propagated enablement, lazy graph-wide enrollment, caller-supplied vectors, per-key object-form dead letter with fail-closed structural refusal, @@ -60,6 +63,20 @@ dead-letter slot), the Cedar-gated single-CAS `set_streaming_enabled_as` flip, cluster-apply-only propagation with refresh convergence and typed pending-until-drained refusal, and the genuine v9↔v10 refusal/rebuild fence in CI; P2–P6 remain unimplemented and no ingest surface is active +**Streaming authorization split and read-only status implemented:** +2026-07-28 — the `stream_ingest` / `stream_manage` Cedar actions of §4.6 are +registered (both graph-scoped; the main-only profile makes a branch dimension +meaningless, so both scope qualifiers are rejected at validation), the P1 +enablement flip migrated from the reserved `Admin` action onto +`stream_manage`, and `Omnigraph::stream_status` projects the durable +enablement and per-lane authority read-only. Status deliberately ships before +the management verbs because §4.6 makes it the compare-token source: it +exposes the `lifecycle_revision` those verbs pass back as their expected +revision, so they can be written as compare-and-set from the start. This is +the §4.7 *minimal* status — the authoritative manifest row only; §4.3's +exclusive-cut physical observation (observed epoch, pending generation +rows/bytes, `StatusChanged`/`StatusBusy`) arrives with the verbs that need +it, as an additive observed-physical section. **Author track:** Maintainer design series **Depends on:** [RFC-022](0022-unified-write-path.md)'s unified write and generic recovery-sidecar protocol, plus @@ -546,9 +563,12 @@ exact incarnation and returns `StreamNotEnrolled` before body admission on This section owns both the implemented private B2 row/fold contract and the remaining future-public/control contract. Internal schema v9 now implements the §4.1 token/attribution substrate and recovery-v12 fold portion of §4.4. It does -not expose a production caller. Explicit enrollment, lifecycle management, -correction, authoritative status, authorization, and product parity remain -future gates. **B2a unbounded retain-all** is the selected first profile: it +not expose a production row caller. The graph-scoped `stream_ingest` / +`stream_manage` Cedar vocabulary and embedded manifest-only read-only status +are active under §4.7, and `stream_manage` gates the v10 enablement writer. +Explicit enrollment, lifecycle mutation/correction, exclusive-cut physical +status, public row admission, and transport parity remain future gates. +**B2a unbounded retain-all** is the selected first profile: it deletes no MemWAL object and performs no physical-storage admission or accounting. **B2b** is the deferred managed-reclamation profile through the Lance-owned protocol in §4.5.2. B2a retains individually bounded protocol @@ -1144,8 +1164,14 @@ after residue is graph-visible and the block is cleared, an otherwise eligible unguarded drain may abort. A quiesce that keeps `goal = SEALED` may instead continue to the sealed proof. This rule is identical in §8 and the B2 gates. -Minimum status is one authoritative manifest row plus a bounded cut-consistent -physical observation. In the supported B2 profile it takes stream admission +The full B2 status contract is one authoritative manifest row plus a bounded +cut-consistent physical observation. The experimental §4.7 slice currently +exposes only the manifest projection through embedded +`Omnigraph::stream_status`; it takes no admission lease, reads no physical +shard witness, and has no CLI/HTTP/OpenAPI surface. The full status below +arrives additively with the lifecycle verbs that need its cut. + +In the supported full B2 profile, status takes stream admission exclusively, read-only lists/classifies relevant recovery intents, settles every writer/watcher/flush/retirement owner plus every selected-profile claim owner to the status deadline, reads lifecycle/token authority plus the authoritative @@ -1981,12 +2007,13 @@ enroll, block inspection, correct, and rebuild-preflight use the separate `stream_manage` action. Status is authorized like other graph operational metadata. The same engine gates apply to embedded and remote CLI use. -Phase B2 initially exposes only strict compare-and-chain upsert. Public activation requires the -minimum `status`, explicit `fold`, persistent `quiesce`, `resume`/abort-drain, -rebuild preflight, a bounded strict-correction workflow, durable +Phase B2 initially exposes only strict compare-and-chain upsert. The full +product surface described in this section requires the full exclusive-cut +`status`, explicit `fold`, persistent `quiesce`, `resume`/abort-drain, rebuild +preflight, a bounded strict-correction workflow, durable authenticated-contributor attribution, a same-key `AckUnknown` sequencing/idempotency contract, and one proved physical-retention profile. -Minimum status includes lifecycle and +Full status includes lifecycle and binding/revision, current epoch, pending generations/bytes, last fold error, receipt-history pagination, and whether strict fold is blocked. It never claims to resolve one caller's `AckUnknown`. @@ -2002,19 +2029,28 @@ narrows §4.6 and amends §7 as stated below; where this profile and an earlier section disagree, this profile governs the experimental activation and the earlier text continues to describe the full product surface. -**Implementation status (2026-07-28): P1 (enablement authority) is -implemented as internal schema v10** — the pre-implementation format audit +**Implementation status (2026-07-28): P1 (enablement authority), the +authorization split, and embedded manifest-only status are implemented.** The +pre-implementation format audit resolved to its pre-registered default (v9 decoders skip unknown row kinds silently, and the correct shape is a required genesis singleton), so v10 adds the graph-global `stream_profile` row and reserves the fold-attribution dead-letter slot in the same bump. The flip ships as the Cedar-gated -(`Admin`) `set_streaming_enabled_as` writer — single exact-entry CAS with a +(`stream_manage`) `set_streaming_enabled_as` writer — single exact-entry CAS with a strict revision advance, one ordinary lineage commit, no recovery sidecar — and `cluster apply` is its sole production caller, with refresh converging the ledger to engine truth and disable refusing typed (`StreamingDisablePending`) while any lifecycle is non-terminal. The genuine -v9↔v10 refusal/rebuild fence is pinned in CI (`OMNIGRAPH_V9_BIN`). P2–P6 -remain unimplemented; enabling the flag activates no ingest surface. +v9↔v10 refusal/rebuild fence is pinned in CI (`OMNIGRAPH_V9_BIN`). +`stream_ingest` and `stream_manage` are registered graph-scoped actions; both +reject branch and target-branch qualifiers. Embedded +`Omnigraph::stream_status` reads the enablement and per-lane durable authority +from one canonical-main manifest snapshot, including the lifecycle revision +future management verbs pass back as their compare token. It is read-only, +takes no admission lease, resolves no recovery, and deliberately omits +physical observations. Full exclusive-cut status and all CLI/HTTP/OpenAPI +streaming surfaces remain inactive. P2–P6 remain unimplemented; enabling the +flag activates no ingest surface. **Profile boundaries.** Main-only; unsharded; one resident writer; one live writer process; unbounded retain-all (§4.5.1); cluster deployments only; @@ -2701,8 +2737,10 @@ and publishes the §4.2 winner summary plus current-token evidence. The graph commit stores a durable commitment to the visible contributor/write winner set; ordinary commits carry no fold summary. Provenance is never inferred from MemTable positions or WAL cursor statistics. This row/fold design is active in -v9; public exposure, lifecycle/correction/status, and restart-stable dead-letter -row identity remain later gates. +v9; public row exposure, lifecycle mutation/correction, exclusive-cut physical +status, CLI/HTTP/OpenAPI parity, and restart-stable dead-letter row identity +remain later gates. The registered Cedar vocabulary and embedded manifest-only +status do not widen the private row/fold seam. ## 7. Fold-time rejection is atomic @@ -2923,7 +2961,9 @@ those limits wherever the tier is exposed. ## 10. Observability and resource contracts -Phase B1 keeps observability internal and does not create a public status +Phase B1 keeps row/fold observability internal. The later §4.7 slice exposes +only durable manifest authority through embedded `Omnigraph::stream_status`; +it does not expose physical worker state or create a CLI/HTTP/OpenAPI status contract. The private implementation exposes test seams at its durability, replay, fencing, resource, cut, fold, visibility, and recovery boundaries. The 2026-07-21 dense-scan repair and near-cap/RSS cell in §12.3 re-prove closure for @@ -2931,8 +2971,8 @@ the widest admitted shape. That evidence is not a public latency SLO, group-commit multiplier, current object-store result, physical-storage bound, or claim that retained metadata work is history-flat. -B2's minimum status surface is the authority-plus-observation contract in -§4.3. It includes the exact lifecycle/binding, active epoch, drain operation, +B2's full status surface is the authority-plus-observation contract in §4.3. +It includes the exact lifecycle/binding, active epoch, drain operation, pending generation/row/byte totals, merged progress, last fold outcome, strict block, receipt-history pagination, lifecycle revision, relevant recovery, and rebuild readiness. Optional retained-object counts/bytes are advisory current- @@ -3398,8 +3438,10 @@ RFC remains draft. remain B2b managed-reclamation and broader-topology gates. They did not block the completed private no-delete, single-live-writer-process B2a gate, which retains the wrapper's post-watcher fence check. The later private v9 - token/fold slice passed independently; public activation still waits on the - remaining lifecycle/correction/status, authorization, and product gates. + token/fold slice passed independently; public row activation still waits on + the remaining lifecycle/correction, exclusive-cut physical-status, and + transport gates. The authorization vocabulary and embedded durable-only + status are already active. **Phase B1 disposition — accepted 2026-07-19, closure gap found 2026-07-20 and repaired 2026-07-21** @@ -3424,7 +3466,9 @@ dense-slice Arrow closure check; physical RSS remains a separate evidence tripwire. The near-cap cell now closes. This amendment changed no product surface. The subsequent v9 slice implements §4.1's private token/attribution and §4.4's base+token fold core. Explicit enrollment, lifecycle management, -correction/status, and product parity remain inactive; §4.5.1's B2a profile is +correction, exclusive-cut physical status, public row admission, and transport +parity remain inactive; the Cedar vocabulary and embedded manifest-only status +are active. §4.5.1's B2a profile is implemented, while §4.5.2's B2b managed-reclamation profile remains optional and inactive. @@ -3523,8 +3567,10 @@ Advisory listed bytes, wall times, and whole-process RSS are printed as diagnostics only and enforce no product threshold. This B2a result itself added no schema or product surface. The subsequent -private B2-common row/fold slice activated schema v9; production callers and -lifecycle/correction/status remain §12.6 work. +private B2-common row/fold slice activated schema v9; production row callers, +lifecycle mutation/correction, exclusive-cut physical status, and transport +parity remain §12.6 work. The authorization/status slice shipped later under +§4.7. ### 12.6 Private B2-common implementation and remaining public/B2b gates @@ -3533,18 +3579,23 @@ implemented core covers schema v9/config-v3/state-v2 format activation, canonical payload/token derivation, trusted row attribution, graph-global token authority, stale-authority admission revalidation, same-generation chaining, recovery-v12 exact base+token folding, durable graph-commit attribution, and -genuine v8↔v9 refusal/rebuild. Public activation still waits for explicit -production enrollment, lifecycle/correction/status, authorization, -cancellation/shutdown, API compatibility, and product parity. This section also +genuine v8↔v9 refusal/rebuild. Public row activation still waits for explicit +production enrollment, lifecycle mutation/correction, exclusive-cut physical +status, cancellation/shutdown, API compatibility, and transport parity. The +graph-scoped Cedar vocabulary, `stream_manage`-gated enablement, and embedded +manifest-only status are already active under §4.7. This section also owns B2b's optional managed-reclamation gates; B2a does not need any B2b-only bullet. The design does not waive the persistent escape requirement: a user must never be left with a table that ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. -- **Inactive product surface:** `@stream(mode="upsert", on_reject="strict")`, - production first use, SDK, HTTP, CLI, Cedar, and OpenAPI must all route - through the same private core. Existing `/ingest` behavior must remain +- **Inactive row/control product surface:** `@stream(mode="upsert", + on_reject="strict")`, production first use, SDK row/control methods, HTTP, + CLI, and OpenAPI must all route through the same private core. The Cedar + actions are registered, but `stream_ingest` has no production caller and + `stream_manage` currently reaches only the enablement writer. Existing + `/ingest` behavior must remain compatible, and embedded/remote command parity must be tested. Accepted-schema and runtime guards must refuse `@stream` on a type requiring `@embed` or any external/provider-derived field; caller-supplied physical @@ -3598,9 +3649,11 @@ ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. protocol digests; uppercase, base64, whitespace, bad prefix, and wrong length must refuse pre-effect. Public caller ordering, mixed request shaping, cancellation/shutdown, and transport parity retain their gates above. -- **Inactive operator controls:** minimum controls must expose authoritative status, explicit fold, plus - persistent quiesce, resume/abort-drain, correction, and rebuild preflight. - Status must take an exclusive cut, settle owners without mutating recovery, and +- **Inactive operator controls:** minimum controls must expose full + exclusive-cut status, explicit fold, plus persistent quiesce, + resume/abort-drain, correction, and rebuild preflight. Embedded durable-only + status is already active; full status must take an exclusive cut, settle + owners without mutating recovery, and include exact lifecycle/binding/revision, epoch, advisory pending generations/bytes, paginated correction and management/claim receipt summaries, last fold summary, and strict-blocked state. Every mutating call after @@ -3761,8 +3814,8 @@ ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. | R0 | production-neutral retained-growth/source audit; current-object census; referenced-cut retry; legal high-entropy near-cap materialize/fold cell; no schema, public caller, or deletion | **Historical bounded-retention no-go 2026-07-20; disposition amended 2026-07-21 (§0.2/§12.4):** RC.1 still exposes neither a complete reserve-first physical envelope/receipt nor a durable cross-open randomized-attempt cap. Those facts prohibit a finite storage promise but do not block selected unbounded retain-all. The formerly red widest cell is now green locally and on the configured-RustFS CI path; current-object observations remain advisory retention evidence, not provider billing/accounting | | B2a | selected unbounded retain-all/no-GC profile on stock Lance | **Private gate implemented 2026-07-21 (§12.5):** no OmniGraph byte/object/file/history quota; zero canonical `_mem_wal` deletion; complete/partial provider residue remains retained, unreferenced, and untouched below its root through retry/reopen; provider failures are loud; local/configured-RustFS history sweeps are advisory. This gate itself activated no schema or product surface; the later private B2-common slice activates v9 | | B2b | candidate managed-reclamation retention profile | Inactive. Requires the Lance-owned durable inspect/plan/execute + receipt, post-success fencing, bounded checkpoint/inventory/accounting, local/RustFS enforced-bound validation, and the profile-specific crash matrix (§4.5.2/§12.6). Passing it alone activates no product surface | -| B2-common | schema v9/config-v3/state-v2, compare-and-chain token/attribution, graph-global token authority, recovery-v12 base+token fold; then explicit enrollment, revision-fenced lifecycle/correction/status, SDK, HTTP, CLI, Cedar, and OpenAPI | **Private row/fold subset implemented 2026-07-22 (§11/§12.6):** canonical digests, hidden attribution, stale-authority revalidation after shared admission, same-generation chains, exact two-participant recovery/publication, durable fold attribution, retain-all, and genuine v8↔v9 refusal/rebuild are green. Explicit production enrollment, lifecycle/correction/status, authorization, cancellation/shutdown, API compatibility, and product parity remain inactive. `GraphHistoryBudget` belongs only to a future bounded/managed profile | -| EXP | experimental cluster-only activation of the §4.7 profile: manifest-propagated enablement via `cluster apply`, lazy graph-wide enrollment, caller-supplied vectors, per-key object-form dead letter with fail-closed structural refusal, no read-your-writes bridge, dependency-ordered resident fold driver, upsert-only | **Selected 2026-07-27 (§4.7); P1 enablement authority implemented 2026-07-28 as internal schema v10** (required genesis `stream_profile` singleton + reserved dead-letter slot; Cedar-gated single-CAS flip; cluster-apply-only propagation with refresh convergence and typed pending-until-drained refusal; genuine v9↔v10 fence in CI). P2–P6 remain unimplemented and ship under the §4.7 Hyrum boundary | +| B2-common | schema v9/config-v3/state-v2, compare-and-chain token/attribution, graph-global token authority, recovery-v12 base+token fold; then explicit enrollment, revision-fenced lifecycle/correction/full status, SDK row/control methods, HTTP, CLI, and OpenAPI | **Private row/fold subset implemented 2026-07-22 (§11/§12.6):** canonical digests, hidden attribution, stale-authority revalidation after shared admission, same-generation chains, exact two-participant recovery/publication, durable fold attribution, retain-all, and genuine v8↔v9 refusal/rebuild are green. Explicit production enrollment, lifecycle mutation/correction, exclusive-cut physical status, public row admission, cancellation/shutdown, API compatibility, and transport parity remain inactive. The Cedar vocabulary and embedded manifest-only status shipped in the later EXP slice. `GraphHistoryBudget` belongs only to a future bounded/managed profile | +| EXP | experimental cluster-only activation of the §4.7 profile: manifest-propagated enablement via `cluster apply`, lazy graph-wide enrollment, caller-supplied vectors, per-key object-form dead letter with fail-closed structural refusal, no read-your-writes bridge, dependency-ordered resident fold driver, upsert-only | **Selected 2026-07-27 (§4.7); P1 enablement authority plus the authorization/status slice implemented 2026-07-28** (internal schema v10 required genesis `stream_profile` singleton + reserved dead-letter slot; `stream_manage`-gated single-CAS flip; registered graph-scoped `stream_ingest` / `stream_manage`; embedded manifest-only read-only status; cluster-apply-only propagation with refresh convergence and typed pending-until-drained refusal; genuine v9↔v10 fence in CI). P2–P6, full exclusive-cut status, and CLI/HTTP/OpenAPI streaming surfaces remain unimplemented and ship under the §4.7 Hyrum boundary | | C | restart-stable reject-row identity, atomic dead letter, richer status, and evidence-backed configurable bounds | reject crash matrix; reject-retention proof; backpressure and RSS/latency evidence. The §4.7 profile pulls a bounded object-form dead-letter subset forward using the §4.1 token as reject identity | | D | automatic operation drain, schema/branch/upgrade integration, and rematerialization rebind | two-coordinator race, old/new physical-binding crash matrix, and format-transition suite | | E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion | diff --git a/docs/user/concepts/storage.md b/docs/user/concepts/storage.md index 3c6abb8a..0bd43897 100644 --- a/docs/user/concepts/storage.md +++ b/docs/user/concepts/storage.md @@ -47,7 +47,7 @@ The on-disk shape of `__manifest` is reconciled with the binary via a single ver - The stamp is read with no object-store writes, so the check is safe under a read-only open. Operators can see a graph's stamp with `omnigraph snapshot` and the binary's served version with `omnigraph version` (the `internal-schema` line). The stamp values below are historical; this binary serves only the current one -(`v9`). An earlier-stamped graph is rebuilt via export/import, not migrated in +(`v10`). An earlier-stamped graph is rebuilt via export/import, not migrated in place. | Stamp | Shape | @@ -60,7 +60,8 @@ place. | v6 | Preserves v5 identity and activates RFC-023: every graph node/edge dataset has exact non-null physical `id` as Lance's unenforced PK, and every production strict insert/upsert uses the exact-`id` filter-bearing adapter. | | v7 | Preserves v5/v6 identity and keyed-write contracts, and adds RFC-026 identity-keyed `stream_state` authority: physical enrollment binding, mutable current-HEAD witness, lifecycle, and per-shard epoch floor. Phase A can recover one exact empty private enrollment; no production row-streaming API is active. | | v8 | Preserves the v5/v6/v7 contracts and activates RFC-026 stream-config v2 plus recovery-v11 for the private B1 row/fold core: one no-roll generation, watcher success plus a same-writer post-durability epoch check before clean acknowledgement, and one exact fold whose manifest publish advances the table pointer, lifecycle witness, and graph lineage together. The legal near-cap closure cell is green after logical-slice accounting and dense scanner-batch copies. | -| v9 | Preserves the bounded B1 worker mechanics and activates RFC-026 stream-config v3, lifecycle state v2, the grammar-impossible trusted-attribution field, manifest-selected `_stream_tokens.lance` authority, compare-and-chain tokens, and recovery-v12's exact base-plus-token fold. The selected B2a profile retains all canonical MemWAL objects and imposes no retained-byte/object/file/history quota. Public enrollment, lifecycle management, correction/status, SDK/HTTP/CLI/OpenAPI, generation GC, and fresh reads remain inactive. **The only version this binary serves.** | +| v9 | Preserves the bounded B1 worker mechanics and activates RFC-026 stream-config v3, lifecycle state v2, the grammar-impossible trusted-attribution field, manifest-selected `_stream_tokens.lance` authority, compare-and-chain tokens, and recovery-v12's exact base-plus-token fold. The selected B2a profile retains all canonical MemWAL objects and imposes no retained-byte/object/file/history quota. | +| v10 | Preserves v9 and adds RFC-026 §4.7's required graph-global `stream_profile` enablement singleton plus the reserved fold-attribution dead-letter slot. The graph-scoped `stream_ingest` / `stream_manage` Cedar vocabulary is registered, `stream_manage` gates enablement, and embedded manifest-only status is read-only. Public row admission, explicit enrollment, lifecycle mutation/correction, exclusive-cut physical status, CLI/HTTP/OpenAPI streaming surfaces, generation GC, and fresh reads remain inactive. **The only version this binary serves.** | ## On-disk layout diff --git a/docs/user/operations/policy.md b/docs/user/operations/policy.md index 54fbea5a..384f0a00 100644 --- a/docs/user/operations/policy.md +++ b/docs/user/operations/policy.md @@ -14,19 +14,26 @@ Per-graph actions (bind to `Omnigraph::Graph::""`): 6. `branch_delete` 7. `branch_merge` 8. `admin` — reserved for policy-management surfaces (hot reload, audit log, approvals). No call site today. -9. `invoke_query` — gates invoking a server-side stored query (the `queries:` registry). Graph-scoped (like `admin`) — per-branch access is enforced by the inner `read` / `change` gate, so a rule that sets `branch_scope` on `invoke_query` is rejected. Coarse in this release: an `invoke_query` allow rule permits any stored query on the graph; a future, additive refinement adds an optional per-query-name scope without changing rules written against the coarse action. Enforced at `POST /queries/{name}` (see [server](server.md)). A stored *mutation* is double-gated: `invoke_query` to reach the tool, plus `change` for the write itself (the engine `_as` writers still enforce per the query body). +9. `stream_ingest` — **experimental (RFC-026)**; gates admitting rows into a table's streaming lane. Graph-scoped: the experimental streaming profile is main-only, so a rule that sets `branch_scope` or `target_branch_scope` on it is rejected at validation. Deliberately separate from `change`: a stream append acknowledges *durability* without graph visibility, so an operator can grant high-rate ingestion without granting direct-lane writes, or the reverse. No production ingest surface exists yet. +10. `stream_manage` — **experimental (RFC-026)**; gates streaming *lifecycle management*: enabling or disabling the graph's streaming flag today, plus fold / quiesce / resume / abort-drain as they ship. Graph-scoped for the same main-only reason. Split from `stream_ingest` because the blast radii differ in kind — ingestion adds rows, while management can seal a lane, drain acknowledged data, or reopen it at a new epoch. Read-only stream *status* is authorized like other graph operational metadata rather than by this action, so an operator can always see whether a lane is stuck even without the rights to act on it. +11. `invoke_query` — gates invoking a server-side stored query (the `queries:` registry). Graph-scoped (like `admin`) — per-branch access is enforced by the inner `read` / `change` gate, so a rule that sets `branch_scope` on `invoke_query` is rejected. Coarse in this release: an `invoke_query` allow rule permits any stored query on the graph; a future, additive refinement adds an optional per-query-name scope without changing rules written against the coarse action. Enforced at `POST /queries/{name}` (see [server](server.md)). A stored *mutation* is double-gated: `invoke_query` to reach the tool, plus `change` for the write itself (the engine `_as` writers still enforce per the query body). Server-scoped action (v0.6.0+; binds to `Omnigraph::Server::"root"`): -10. `graph_list` — `GET /graphs` registry enumeration (multi-graph mode) +12. `graph_list` — `GET /graphs` registry enumeration (multi-graph mode) Server-scoped actions cannot use `branch_scope` or `target_branch_scope` — they operate on the registry, not on a graph's branches. A rule cannot mix server-scoped and per-graph actions; split into separate rules. (Runtime `graph_create` / `graph_delete` over HTTP are reserved but not shipped; operators add/remove graphs by editing the cluster's `cluster.yaml`, running `omnigraph cluster apply`, and restarting the server.) +Policy YAML and `omnigraph policy explain --action` use `stream_ingest` and +`stream_manage` as the canonical spellings. The CLI also accepts +`stream-ingest` and `stream-manage` as aliases. + ## Scope kinds - `branch_scope` — applied to source branch (`read`, `export`, `change`) - `target_branch_scope` — applied to destination (`schema_apply`, branch ops, run ops) - `protected_branches` — named list with special rules; rule scopes are `any | protected | unprotected` +- Graph-scoped per-graph actions (`admin`, `invoke_query`, `stream_ingest`, `stream_manage`) take **neither** scope; a rule that sets one is rejected at validation. ## Per-graph vs. server-level policy diff --git a/skills/omnigraph/references/server-policy.md b/skills/omnigraph/references/server-policy.md index 225c7082..3192325f 100644 --- a/skills/omnigraph/references/server-policy.md +++ b/skills/omnigraph/references/server-policy.md @@ -113,14 +113,25 @@ Per-graph actions (evaluated against the graph being addressed): | `branch_create` | branch creation | | `branch_delete` | branch deletion | | `branch_merge` | merges (especially into protected branches) | +| `stream_ingest` | RFC-026 streaming row admission. The action is registered and graph-scoped, but no production ingest caller exists yet. | +| `stream_manage` | RFC-026 streaming lifecycle management. It currently gates the graph enable/disable writer; fold, quiesce, resume, abort-drain, enrollment, and correction callers remain inactive. | -`admin` exists but is reserved (no call site yet — don't write rules for it). A server-scoped `graph_list` action gates `GET /graphs`; declare it in a `[cluster]`-scoped bundle. +Both stream actions are graph-scoped but not branch-scoped: a rule that sets +`branch_scope` or `target_branch_scope` on either action is rejected. Policy +YAML and `omnigraph policy explain --action` use the underscore spellings as +canonical; the CLI also accepts `stream-ingest` / `stream-manage` as aliases. +Embedded read-only `stream_status` is operational metadata and does not require +`stream_manage`; there is no CLI/HTTP/OpenAPI stream-status surface yet. + +`admin` exists but is reserved (no call site yet — don't write rules for it). +A server-scoped `graph_list` action gates `GET /graphs`; declare it in a +`[cluster]`-scoped bundle. For any shared repo, gate at least `schema_apply` and `branch_merge`. ### Where policy is declared -Cedar bundles are declared in `cluster.yaml` and attach via `applies_to`: `[cluster]` is the server-level engine (gates `graph_list` / `GET /graphs`); `[]` is that graph's engine (gates `invoke_query`, `read`, `change`, `branch_*`, `schema_apply`). `cluster apply` publishes them and the `--cluster` server enforces the applied revision. The `policy.yaml` rule format (below) is the bundle content. +Cedar bundles are declared in `cluster.yaml` and attach via `applies_to`: `[cluster]` is the server-level engine (gates `graph_list` / `GET /graphs`); `[]` is that graph's engine (gates `invoke_query`, `read`, `change`, `branch_*`, `schema_apply`, and the two stream actions). `cluster apply` publishes them and the `--cluster` server enforces the applied revision. The `policy.yaml` rule format (below) is the bundle content. ### `policy.yaml` shape