feat: port entity schema to gen-schema#563
Draft
sini wants to merge 29 commits into
Draft
Conversation
sini
force-pushed
the
feat/entity-gen-schema-port
branch
from
6574fac to
a370d30
Compare
sini
force-pushed
the
feat/entity-gen-schema-port
branch
7 times, most recently
from
b2bcfd4 to
1b56211
Compare
sini
force-pushed
the
feat/entity-gen-schema-port
branch
3 times, most recently
from
20fedec to
91ded60
Compare
Replaces hand-rolled schemaEntryType with gen-schema mkSchemaOption. Sidecars: includes, excludes. Computed: isEntity (structural content only). Extracts resolvedCtxModule (id_hash, resolved, collisionPolicy) to _types.nix for entity type reuse. collisionPolicy flows through deferred module merge to entity instances (not a sidecar) preserving existing ctx.host.collisionPolicy resolution path.
den.hosts now accepts both forms:
- Legacy: den.hosts.x86_64-linux.igloo = { ... }
- Flat: den.hosts.igloo = { system = "x86_64-linux"; ... }
The outer option type uses a permissive submodule with deepMergeAttrs
freeformType (lib.recursiveUpdate-based merge that avoids the infinite
recursion lib.types.anything causes with cross-option references).
The apply function preprocesses flat entries into two-level form and
re-evaluates through the original attrsOf systemType, so all 6
consumers see the canonical { system.name = hostConfig } shape.
Same pattern as den.hosts: deepMergeAttrs + preprocessHosts + apply. Cross-entity host lookup and osConfig injection preserved.
Covers: id_hash, freeform, topology, meta introspection, isEntity computed, schema includes sidecar.
Update flake inputs and references to match the renamed repo at github:sini/gen-schema.
gen-schema flattened _meta into _-prefixed options and renamed sidecars → collections. nix-effects changed bindAttrs so true is a literal param, not an optionality marker — translate __args values to fx.bind.optionalArg before bind.fn.
Build host/user/home submodules with gen-schema's mkInstanceType, which injects name, strict/freeform, _module.args.<kind>, and schema-owned id_hash (gen-algebra mkIdentityModule) — replacing the hand-rolled id_hash reflection in resolvedCtxModule. Identity now shares the exact algorithm gen-schema's ref/setOf dedup compares against, removing the drift hazard between den's copy and the library's. Add an entity-gen-schema test pinning _roots: host is a root scope and user/home are not (the buildRoots root-detection contract).
den.reservedKeys fed structuralKeysSet, which the pipeline classifier and the synthetic-_ childKeys filter honored — but the aspect submodule's freeform value type (aspectKeyType) wrapped every undeclared key into the __contentValues/__provider provenance shape regardless. A reserved key was excluded from dispatch but its value was still mangled, so consumers could not read it back as the metadata den.reservedKeys promises. Route structural/reserved keys through a passthrough merge (last def wins) in aspectKeyType — the per-key dispatch the type comment already anticipated. den.aspects.<name>.<reservedKey> now returns the raw value.
sini
force-pushed
the
feat/entity-gen-schema-port
branch
from
91ded60 to
a22c657
Compare
The `self` dispatch guard (fire-once at the registration scope; always bound in ctx so flake-scope resolution policies can fire) was added on this branch but never made it into origin/main's squash of the cherry-picked fixes — denful#603 covered spawn subtree routes and multi-def identity, not `self`. Rebasing onto origin/main therefore silently dropped it. Without `self`, `{ self, ... }:` policies never dispatch (flake isn't bound in its own ctx), collapsing any flake-scope resolution cascade — e.g. a consumer's flake -> fleet -> environment -> host walk produces zero host outputs. Re-applies the dispatch.nix `self` injection and the policy/schema.nix late-fan-out exclusion, plus the self-guard regression test.
schema-util and entities/_types each hand-rolled `filter (k != "conf" && !hasPrefix "_")` over `attrNames den.schema`. gen-schema already exposes `_kindNames` (sorted, _-prefixed introspection keys excluded), so consume that as the canonical kind list and drop the duplicated _-prefix filtering. Entity detection still uses the schema's `isEntity` collection.
… entity-gen-schema-port base)
Re-applies df21485 (lost during the entity-gen-schema-port rebase, never cherry-picked to main): host.aspects exposes the flat list of all resolved aspect nodes, each augmented with .identity (base FQN), .identityKey (ctx-qualified), and .isNamed — symmetric with host.hasAspect, sharing the same per-class fxFullResolve. Resolves merge against the projected-hasAspect work landed since: pathSetByScope (gate !isExcluded, entity root included) and resolvedNodes (gate !isExcluded && !isEntityRoot) now coexist in collectPathsHandler, and pathSetByScope keys on the renamed local nodeBaseKey rather than the top-level baseKey helper. Unblocks nix-config's colmena battery, which reads host.aspects for deploy tags. 916/916 CI green, including Group K aspects tests.
…scope A kind declaring isolated = true marks its entity scopes as delivered: their content belongs to their own system and must not be absorbed across the entity boundary. push-scope records a scopeIsolated map (scopeId -> bool) on pipeline state, mirroring scopeEntityKind.
…ecoupling
Both subtree-collection sites (extractSubtreeModules, collectFromSubtree)
now skip isolated descendant scopes and everything below them; the
collection root stays exempt so an isolated entity's own delivery route
still collects itself. Compose entities (user/home) carry no marker and
are unaffected.
A route's single sourceScopeId doubled as collection root and append
target, which cannot express delivery from an isolated child: rooting at
the host over-collects, appending at the (extraction-skipped) child drops
the content. appendToParent = true decouples them — collection roots at
the registering scope, the append lands at scopeParent.${sourceScopeId}.
Routes omitting the field are byte-identical to before.
mkInstantiateArgs' internal subtree selection stays isolation-blind on
purpose: the guest's class imports must remain visible to the delivery
route in the per-host re-walk; only final extraction filters.
… remap The guest-os class remap kept the guest's walked content out of the parent's nixos partition, but only for content AUTHORED as guest-os — reused fleet aspects authored as nixos.* leaked into the parent (option 'microvm.guest' does not exist on the parent toplevel). Entity isolation closes that leak at the source, so the guest keeps its honest nixos identity: isolated = true on the kind, class = nixos, and a guest-scope delivery route (fromClass nixos, appendToParent) carrying the guest subtree to microvm.vms.<name>.config exactly once. With class = nixos the standard home-manager battery fires for the guest, replacing the parallel guest-os home-env; the per-user bridge stays (simple routes collect pre-route per-scope state, so the battery's user-scope forward is invisible to the delivery route).
Review follow-ups on the entity-isolation work (0465898..e11081b): spawn-node's final extraction walk is isolation-blind by design — isolated entities resolve via resolve.to in the host pipeline, never through spawnNode. Name the invariant at the walk so a future change doesn't silently reintroduce the cross-boundary leak. BREAKING CHANGE (e11081b): the guest-os class no longer exists. Delivered-guest content must be authored as nixos.* (was guest-os.*). Parents whose host-includes now also fire at the parent scope (e.g. agenix age.*, microvm options) need the corresponding option stubs or modules on the parent.
The delivered-child delivery route lands collected nixos at
microvm.vms.<n>.config, whose option type RE-INSTANTIATES it as a full
NixOS system (eval-config + base module-list). nestPlain pre-evaluated the
payload in an isolated freeform evalModules and delivered the *resolved*
config (no base-module defaults), after unwrapping mod.imports — discarding
the per-module key/_file that wrap-classes assigns (<class>@<identity>). At
a re-instantiating target this poisoned every namespace aggregate
(boot/system/networking become valueless) and double-declared the now
keyless modules collected across {host,user} scopes (e.g. lix.enable).
Add an opt-in `reinstantiate` route flag. When set, deliver each collected
keyed wrapper verbatim ({ imports = [mod]; }): the target's own eval-config
applies base defaults and dedups identical re-declarations by key — exactly
as spawn-node's instantiation walk does. The delivered-child primitive sets
it on its delivery route.
Existing routes default reinstantiate=false and are byte-identical
(verified: axon-01 toplevel drvPath unchanged across the bump; den CI
924/924).
The existing acceptance tests use a freeform `microvmSlot` stub that stores the delivered config but never re-evaluates it, so they structurally cannot catch a delivery that strips base-module context (the resolved-config bug the route `reinstantiate` flag fixes). Add a faithful `reinstantiatingSlot` whose option type re-runs `evalModules` over the delivered defs together with a base module-list (mirroring microvm's eval-config), and `test-reinstantiation-applies-base-context`: a guest module that READS a base-module default must see it through re-instantiation. This FAILS on the pre-fix route (nestPlain pre-evaluates each module in an isolated freeform evalModules with no base — the read throws) and PASSES with verbatim keyed delivery.
The in-context (projected) hasAspect looks up an aspect's path key in the owning entity's `__pathSetByScope`, bucketed by scopes WITHIN that entity's own resolution — keyed by the owner kind and its descendants (host → user/home), e.g. "host=igloo". But `decomposeSchemaEffect` computed the lookup scopeId from EVERY entity-kind binding present in the production ctx, including ANCESTOR kinds the host inherits under a multi-tier topology (a fleet environment: host.parent = environment). So the lookup key "environment=prod,host=axon-01" never matched the bucket key "host=axon-01", and every in-context hasAspect on such a host read false. In nix-config this surfaced as agenix computing `host.hasAspect den.aspects.core.impermanence == false` on the axon hosts, pointing identityPaths at /etc/ssh instead of /persist/etc/ssh. den's default flake→system→host walk hid it because `system` is a plain string, not an entity kind, so it was already filtered out. Restrict the projected scopeId to the owner subtree: keep a kind only if it is the owner or a descendant of the owner (via den.schema.<kind>.parent). The hasAspect override still applies to every in-ctx entity kind; only the scope key is narrowed. Regression test reproduces a flake→tier→host topology where the host inherits an ancestor `tier` binding.
…d policy The delivered-child-host policy was a consumer composition built entirely from public den primitives (resolve.to.withIncludes, route, schema registration) yet lived in den core, auto-registering a delivered-guest entity kind and a deliveredChildren host option for every consumer — core bloat for a single (nix-config microvm) use case. Keep only the genuine route-engine primitive in den: the route `reinstantiate` flag (wrap.nix/apply.nix), now covered by a standalone, hand-rolled route test (resolve + route + a re-instantiating slot) instead of the policy-coupled acceptance suite. The policy itself moves to its sole consumer (nix-config), where it is named for its domain (host.guests). - remove modules/policies/delivered-child-host.nix - remove the policy acceptance suite (public-api/delivered-child-host.nix) - add route.test-route-reinstantiate-base-context (flag tested directly) - entity-topology test: host.children no longer includes delivered-guest
…ope-string
The projected (in-context) hasAspect resolved an in-flight scope's membership
against the owning entity's `__pathSetByScope`, which is PRODUCED by that
entity's standalone resolve (host as root, no ancestors) but CONSUMED in the
fleet resolve (host nested under environment/fleet). Both keyed by `mkScopeId`,
but on different ctx — so the consumer had to reconstruct an owner-relative
scope-string by walking the parent DAG to strip ancestor entity-kinds and
filtering non-entity keys. That reconciliation was the whole exception cluster
(ownerKind / inOwnerSubtree / scopeIdKinds / a second mkScopeId), and the
class of bug behind the agenix /persist-vs-/etc regression.
Re-key the buckets by entity identity (`id_hash`) at the entity surface
instead. id_hash is context-free (kind+name, not ancestry) and stable across
the standalone-produce and fleet-consume runs, so each in-ctx entity looks up
its OWN delivered set by its OWN id_hash — no scope-string, no ancestor
stripping, no parent-DAG walk. The produce side is untouched (the structural
walk still buckets by scope-string); the re-key is a single pure transform in
entities/_types.nix:pathSetByScopeOption, where the root scope's entity is
`config` itself (its kind passed in, since the root is seeded without a
push-scope record).
- resolve.nix: surface scopeContexts + scopeEntityKind on the resolve result
- _types.nix: re-key pathSetByScope by id_hash (fold-union on id_hash collision
— id_hash is parent-blind, so same-named siblings union rather than last-wins;
over-approximation is the safe direction, dropping a bucket false-negatives)
- schema.nix decomposeSchemaEffect: drop ownerKind/inOwnerSubtree/scopeIdKinds/
scopeId; each entity binding's hasAspect keys on its own id_hash
- has-aspect.nix mkProjectedHasAspect: { pathSetByScope, key } (null-guarded)
den CI 915/915; agenix hasAspect core.impermanence resolves /persist on the
axon/cortex fleet and the delivered cortex-cuda guest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
schemaEntryTypeinoptions.nixwith gen-schema'smkSchemaOption(sidecars:includes,excludes; computed:isEntity)_topologyand_metaintrospection from gen-schemaresolvedCtxModuleto shared_types.nixfor reuse across entity typesDetails
Schema port:
den.schemanow uses gen-schema'smkSchemaEntryTypewhich provides sidecar extraction, computed fields, and__functorwrapping. TheresolvedCtxModule(id_hash, resolved, collisionPolicy) is extracted to_types.nixand injected into entity submodule imports.Flat form: Both
den.hostsandden.homesnow accept flat declarations:A
deepMergeAttrscustom type accepts both forms, andapplypreprocesses flat entries into the canonical two-level shape viapreprocessHosts. All 6 consumers see the unchanged{ system.name = entity }shape.Tests: 18 new tests (843 total, up from 825) covering flat hosts, flat homes, id_hash, freeform attrs, topology, meta introspection, isEntity computed, and schema sidecars.
Test plan
nix develop -c just ci)default,minimal,example) unaffected