Add customize_config API for customizing index datastore configuration - #1366
Conversation
b73a943 to
24b99f1
Compare
Implements the general-purpose customization hook proposed in block#1364: t.index "campaigns" do |i| i.rollover :monthly, "createdAt" i.customize_config do |config| config["aliases"] = { "campaigns_read" => {}, "campaigns_active" => {"filter" => {"term" => {"status" => "ACTIVE"}}} } end end The block mutates a defensive deep copy of the index configuration (the template body for rollover index definitions, the index config otherwise), and the result is baked into `datastore_config.yaml`. Three admin-side changes make customizations fully respected: * Absent means "leave alone": on template updates, desired aliases are merged over the template's current ones instead of replacing them, so an alias created outside of ElasticGraph is never clobbered by a routine `clusters:configure` (fixing that regression independently of the new API). * `ForIndexTemplate#configure!` now diffs the full desired vs. current template config instead of just mappings and settings, so a customization-only change still reaches the datastore. * `ForIndex#configure!` reconciles aliases on existing concrete indices via the `_aliases` API (adding/updating desired aliases, never removing undeclared ones), so a newly declared alias lands on pre-existing rollover indices rather than only ones created later. `IndexConfigNormalizer` now treats an empty `aliases` hash the same as an absent one, since the datastore doesn't consistently echo the key back, and both datastore clients gained an `update_index_aliases` method. Generated with Claude Code
Field aliases declared via `customize_config` (mapping properties with `type: alias`) ride the existing mappings reconciliation. This covers that end-to-end for both `ForIndex` and `ForIndexTemplate`: they are included at creation, reconciled onto already-existing indices via the mapping update path, and idempotent on reconvergence. Generated with Claude Code
24b99f1 to
e1e6910
Compare
myronmarston
left a comment
There was a problem hiding this comment.
Looking great! Almost ready to merge. Left a few suggestions.
One more (which didn't have a changed file I could hang it off of): I'd like to see the new customize_config API used in config/schema for our test/local schema (ideally, for both an index alias and a field alias). While we won't do anything specific with it, one of the goals of the test schema is to exercise a wide variety of EG features, and including it there would cause it to naturally get exercised every time the schema is applied to the test datastore, or when we run rake boot_locally, etc.
…tion/indexing/index.rb Co-authored-by: Myron Marston <myron.marston@gmail.com>
- Replace the defensive-copy spec with one that actually fails without the deep copy: it asserts the index's own `settings` (and the very array the caller passed to `sort:`) survive customization mutations. The prior spec mutated per-type mapping hashes that are never shared across indexed types, so it passed even with the copy removed. - Add an integration spec proving mapping updates are applied before alias updates, using a `nested` alias filter that the datastore rejects while its path is unmapped. Verified it fails (with `BadDatastoreRequest`) when the order is swapped, under `NO_VCR=1` (VCR playback masks the ordering). - Prove and document `_aliases` `add` semantics: it is an upsert, so a re-declared definition fully replaces the existing one (a dropped `filter` is removed, not merged). - Simplify the empty-`aliases` check in `IndexConfigNormalizer`. The suggested `&.empty?` form trips `Lint/SafeNavigationWithEmpty`, so standardrb's corrected form is used instead. - Rename `simulate_presence_of_external_alias` to `create_external_alias`. - Use `customize_config` in the test/local schema (`teams` index): index aliases (plain and filtered) plus a field alias, so the feature is exercised whenever the schema is applied to a datastore. Generated with Claude Code
Reverts the review round-trip on this line: the suggested `&.empty?` form trips `Lint/SafeNavigationWithEmpty`, so per review we are sticking with the original form. Generated with Claude Code
myronmarston
left a comment
There was a problem hiding this comment.
LGTM! I noticed one thing but I don't want to block merge on it (it's really unimportant) so I'm going to set this to merge on green.
…definition/datastore_config/customize_config_spec.rb Co-authored-by: Myron Marston <myron.marston@gmail.com>
Head branch was pushed to by a user without write access
…definition/datastore_config/customize_config_spec.rb Co-authored-by: Myron Marston <myron.marston@gmail.com>
|
Thanks for the contribution, @markyang-toast! |
Implements the
i.customize_configproposal from #1364 (#1364 (comment)).Schema definition API
Semantics match the proposal:
mappingsare shared across indices).datastore_config.yaml, so it's diffable in PRs andelasticgraph-admintreats it identically to built-in config.customize_configblocks compose in registration order.Admin changes so customizations are fully respected
The three changes outlined in the proposal:
ForIndexTemplate#configure!now diffs the full desired vs. current template body (not just mappings/settings), so a customization-only change still reaches the datastore. To keep this comparison stable,IndexConfigNormalizernow treats an emptyaliaseshash the same as an absent one (the datastore doesn't consistently echo the key back).ForIndex#configure!gained an_aliasesstep (after mappings, since alias filters can reference newly-added fields): it adds/updates desired aliases and never removes undeclared ones. A newly declared alias now lands on pre-existing rollover indices rather than only on ones created afterward.Both datastore clients gained an
update_index_aliasesmethod (plus dry-run decorator no-op, RBS interface entry, and parallel-spec-runner support).Testing
ForIndexandForIndexTemplate): create-with-aliases + idempotent reconvergence, adding/updating aliases on existing indices and templates, and leaving undeclared (out-of-band) aliases alone. A template-specific spec verifies aliases are stamped onto concrete rollover indices at creation and reconciled onto pre-existing ones.customize_config, applied at creation, reconciled onto an already-existing index/template via the mapping update path, and idempotent on re-run.datastore_config.yamlemission is unchanged for schemas that don't use the new API (repo artifacts show no diff afterschema_artifacts:dump).script/type_checkclean, 100% doc coverage maintained,site:doctestpasses.🤖 Generated with Claude Code