Skip to content

Add customize_config API for customizing index datastore configuration - #1366

Merged
myronmarston merged 7 commits into
block:mainfrom
markyang-toast:customize-config
Aug 27, 2026
Merged

Add customize_config API for customizing index datastore configuration#1366
myronmarston merged 7 commits into
block:mainfrom
markyang-toast:customize-config

Conversation

@markyang-toast

@markyang-toast markyang-toast commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Implements the i.customize_config proposal from #1364 (#1364 (comment)).

Schema definition API

schema.object_type "Campaign" do |t|
  # ...
  t.index "campaigns" do |i|
    i.rollover :monthly, "createdAt"

    i.customize_config do |config|
      # Index aliases
      config["aliases"] = {
        "campaigns_read" => {},
        "campaigns_active" => {"filter" => {"term" => {"status" => "ACTIVE"}}}
      }

      # Field alias: lets direct datastore queries reference `created` as an alias of `createdAt`
      config["mappings"]["properties"]["created"] = {"type" => "alias", "path" => "createdAt"}
    end
  end
end

Semantics match the proposal:

  • The block mutates the yielded config hash; the return value is ignored. ElasticGraph yields a defensive deep copy so mutations can't corrupt its internal structures (parts of mappings are shared across indices).
  • The yielded hash is "the config every concrete index for this index definition gets" — the template body for rollover index defs, the index config otherwise.
  • The result is baked into datastore_config.yaml, so it's diffable in PRs and elasticgraph-admin treats it identically to built-in config.
  • Multiple customize_config blocks compose in registration order.
  • Nested customizations — like the field alias above — work for both indices and index templates, since the block mutates the full config in place. On the admin side they ride the existing mappings reconciliation: included at index/template creation, added to already-existing indices (including pre-existing rollover indices) via the mapping update path, and idempotent on reconvergence.
  • The YARD docs carry the "ElasticGraph passes this through and makes no claims about safety/correctness — use with care!" caveat.

Admin changes so customizations are fully respected

The three changes outlined in the proposal:

  1. Absent means "leave alone". On template updates, desired aliases are merged over the template's current ones instead of replacing them — the same policy admin already applies to no-longer-desired mapping fields. This fixes the out-of-band alias clobbering regardless of whether anyone uses the new API.
  2. Template update trigger. ForIndexTemplate#configure! now diffs the full desired vs. current template body (not just mappings/settings), so a customization-only change still reaches the datastore. To keep this comparison stable, IndexConfigNormalizer now treats an empty aliases hash the same as an absent one (the datastore doesn't consistently echo the key back).
  3. Reconciliation of existing concrete indices. ForIndex#configure! gained an _aliases step (after mappings, since alias filters can reference newly-added fields): it adds/updates desired aliases and never removes undeclared ones. A newly declared alias now lands on pre-existing rollover indices rather than only on ones created afterward.

Both datastore clients gained an update_index_aliases method (plus dry-run decorator no-op, RBS interface entry, and parallel-spec-runner support).

Testing

  • Unit specs for the schema definition API (application to index config vs. template body, block composition, defensive copy, nested customizations, no-block error).
  • Shared admin integration specs (run for both ForIndex and ForIndexTemplate): create-with-aliases + idempotent reconvergence, adding/updating aliases on existing indices and templates, and leaving undeclared (out-of-band) aliases alone. A template-specific spec verifies aliases are stamped onto concrete rollover indices at creation and reconciled onto pre-existing ones.
  • A shared integration spec covers field aliases end-to-end for both configurators: declared via customize_config, applied at creation, reconciled onto an already-existing index/template via the mapping update path, and idempotent on re-run.
  • datastore_config.yaml emission is unchanged for schemas that don't use the new API (repo artifacts show no diff after schema_artifacts:dump).
  • Full suite green (5315 examples, 0 failures), script/type_check clean, 100% doc coverage maintained, site:doctest passes.

🤖 Generated with Claude Code

Implements the general-purpose customization hook proposed in
block#1364:

  t.index "campaigns" do |i|
    i.rollover :monthly, "createdAt"

    i.customize_config do |config|
      config["aliases"] = {
        "campaigns_read" => {},
        "campaigns_active" => {"filter" => {"term" => {"status" => "ACTIVE"}}}
      }
    end
  end

The block mutates a defensive deep copy of the index configuration
(the template body for rollover index definitions, the index config
otherwise), and the result is baked into `datastore_config.yaml`.

Three admin-side changes make customizations fully respected:

* Absent means "leave alone": on template updates, desired aliases are
  merged over the template's current ones instead of replacing them, so
  an alias created outside of ElasticGraph is never clobbered by a
  routine `clusters:configure` (fixing that regression independently of
  the new API).
* `ForIndexTemplate#configure!` now diffs the full desired vs. current
  template config instead of just mappings and settings, so a
  customization-only change still reaches the datastore.
* `ForIndex#configure!` reconciles aliases on existing concrete indices
  via the `_aliases` API (adding/updating desired aliases, never
  removing undeclared ones), so a newly declared alias lands on
  pre-existing rollover indices rather than only ones created later.

`IndexConfigNormalizer` now treats an empty `aliases` hash the same as
an absent one, since the datastore doesn't consistently echo the key
back, and both datastore clients gained an `update_index_aliases`
method.

Generated with Claude Code
Field aliases declared via `customize_config` (mapping properties with
`type: alias`) ride the existing mappings reconciliation. This covers
that end-to-end for both `ForIndex` and `ForIndexTemplate`: they are
included at creation, reconciled onto already-existing indices via the
mapping update path, and idempotent on reconvergence.

Generated with Claude Code

@myronmarston myronmarston left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great! Almost ready to merge. Left a few suggestions.

One more (which didn't have a changed file I could hang it off of): I'd like to see the new customize_config API used in config/schema for our test/local schema (ideally, for both an index alias and a field alias). While we won't do anything specific with it, one of the goals of the test schema is to exercise a wide variety of EG features, and including it there would cause it to naturally get exercised every time the schema is applied to the test datastore, or when we run rake boot_locally, etc.

markyang-toast and others added 3 commits August 26, 2026 15:26
…tion/indexing/index.rb

Co-authored-by: Myron Marston <myron.marston@gmail.com>
- Replace the defensive-copy spec with one that actually fails without
  the deep copy: it asserts the index's own `settings` (and the very
  array the caller passed to `sort:`) survive customization mutations.
  The prior spec mutated per-type mapping hashes that are never shared
  across indexed types, so it passed even with the copy removed.
- Add an integration spec proving mapping updates are applied before
  alias updates, using a `nested` alias filter that the datastore
  rejects while its path is unmapped. Verified it fails (with
  `BadDatastoreRequest`) when the order is swapped, under `NO_VCR=1`
  (VCR playback masks the ordering).
- Prove and document `_aliases` `add` semantics: it is an upsert, so a
  re-declared definition fully replaces the existing one (a dropped
  `filter` is removed, not merged).
- Simplify the empty-`aliases` check in `IndexConfigNormalizer`. The
  suggested `&.empty?` form trips `Lint/SafeNavigationWithEmpty`, so
  standardrb's corrected form is used instead.
- Rename `simulate_presence_of_external_alias` to `create_external_alias`.
- Use `customize_config` in the test/local schema (`teams` index):
  index aliases (plain and filtered) plus a field alias, so the feature
  is exercised whenever the schema is applied to a datastore.

Generated with Claude Code
Reverts the review round-trip on this line: the suggested `&.empty?`
form trips `Lint/SafeNavigationWithEmpty`, so per review we are
sticking with the original form.

Generated with Claude Code

@myronmarston myronmarston left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I noticed one thing but I don't want to block merge on it (it's really unimportant) so I'm going to set this to merge on green.

@myronmarston
myronmarston enabled auto-merge (squash) August 26, 2026 18:26
…definition/datastore_config/customize_config_spec.rb

Co-authored-by: Myron Marston <myron.marston@gmail.com>
auto-merge was automatically disabled August 26, 2026 22:24

Head branch was pushed to by a user without write access

…definition/datastore_config/customize_config_spec.rb

Co-authored-by: Myron Marston <myron.marston@gmail.com>
@myronmarston
myronmarston merged commit 2b9463c into block:main Aug 27, 2026
16 checks passed
@myronmarston

Copy link
Copy Markdown
Collaborator

Thanks for the contribution, @markyang-toast!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants