feat(agent): add Hudi Architect design-time skill

[nsivabalan]

Describe the issue this Pull Request addresses

Part of #19263 (specialized sub-agents), under the #19256 umbrella. Milestone 1, raised
for community feedback — it targets pipeline modelling, with multi-writer, table-services
operationalization, and Flink as tracked extensions. See Scope.

The problem: onboarding onto Hudi is harder than it should be.

Hudi is a good deal more than a table format — it ships ingestion utilities, self-managing
table services, indexing, catalog sync, and a CLI, across three engines (Spark, Flink, Java).
Each of those engines offers more than one way to ingest, with different deployment models
behind them: HoodieStreamer in continuous or run-once mode, Spark DataSource, Spark SQL,
Structured Streaming, Flink SQL and the Flink datastream API. Table services can run inline,
async in the writing job, or as separate jobs. That breadth is a genuine strength — it is what
lets one system serve a 100GB dimension table and a petabyte fact table, streaming and batch —
but the feedback we consistently get from the community is that getting started feels
complex, and that the number of configs is overwhelming when you are still working out which
of those paths you are even on.

So the distance between "I have a Kafka topic of trip updates" and "here is a correct table
configuration" gets bridged today by one of three things:

1. Assembling the answer yourself from scattered sources. Combing through Slack history
   and GitHub issues for someone with a similar-sounding workload, reading through dozens of
   blog posts of varying vintage, and cross-referencing a config reference that is organized
   by config rather than by decision. Nothing in that reference tells you that
   hoodie.compaction.target.io is denominated in megabytes, or that partitioning cannot be
   changed after table creation. Worse, a Slack thread or blog post that was correct for
   0.14 may quietly not be for 1.2, and nothing signals which.
2. Copying someone else's config bundle. Fast, and produces a table sized for someone
   else's workload. This is the most common path and the most quietly damaging one.
3. Asking an experienced Hudi practitioner. Reliable, and doesn't scale. A large share
   of our community support traffic is exactly this request.

This PR aims at the onboarding experience, by automating option 3 so nobody has to fall
back on 1 or 2. Instead of combing through Slack, GitHub issues, and a hundred blog posts
to work out how to design a pipeline with Hudi, every user gets their own Hudi practitioner
to talk to — and walks out with a config bundle, an ADR, and a runnable submit command in a
matter of minutes.

The platform breadth above is exactly what an interview is good at absorbing. A user does not
need to know that HoodieStreamer continuous mode gets async compaction in-process while Spark
DataSource does not; they need to answer "where does the data come from?" and "how is the
pipeline expressed?" and have the engine, writer, deployment model, and service posture fall
out of those answers. The optionality stays — it just stops being the user's problem to
navigate cold.

The valuable thing a solutions architect brings isn't knowing config names — it's knowing
which questions to ask. They don't open the config reference. They ask: how often does
data land? Do records change after insert? What do consumers filter on? Can that column's
value change for a record? Then they derive the Hudi decisions from those answers, and tell
you which decisions you can't take back.

That interview is a formalizable artifact: the questions are finite, their gating is
deterministic, the derivations are table lookups, and the traps are enumerable. This skill
encodes it, and it stays in the repo next to the configs it reasons about — so unlike a blog
post, it can be kept honest against the code (see the key validator below).

It also directly answers a question raised on #19264. @vaibhavk1992 asked whether the
agent would have Hudi-specific knowledge baked in rather than behaving like a generic
text-to-SQL layer, and the reply was that we plan to land skills and tools that make it
really know Hudi internals. This is one of those, aimed at design time.

Summary and Changelog

Adds a design-time architect agent: an interactive skill that interviews a user about
their workload in plain language and produces a Hudi table design — an Architecture
Decision Record, a grouped hoodie.* config bundle, and a runnable submit command. It
asks workload questions ("how often does data land?", "what do consumers filter on?"),
never Hudi questions ("which index type?"), and derives the Hudi decisions from the
answers.

What a user gets. A ~15-minute conversation instead of days of research across Slack,
GitHub issues, blog posts, and the config reference — ending in three artifacts they can put
straight through normal design review: an ADR with tradeoff tables and measurable revisit
conditions, a config bundle where every property encodes a decision, and a submit command
whose flags are split into load-bearing (derived from the design) versus environment
placeholders (paths, memory, Scala/Spark versions the flow deliberately never asks about).

It is also, deliberately, an interview rather than a questionnaire — it explains the
tradeoff before each durable decision, recommends one with reasoning, and lets the user
override. Someone who has never designed a Hudi table gets a working one; someone who has
gets a second opinion and a written record of why.

Why this is a separate agent rather than part of the gateway loop. The landed loop is
grounded and read-only — it answers from tool results over tables that exist, and its
system prompt tells it to never invent table or column names. Table design is the opposite
shape: generative, interrogative, and about a table that does not exist yet, so there is
nothing to query and nothing to ground against. That is the narrower-toolset argument in
#19263 applied to design time. #19263 names optimization, analysis, and visualization as
its examples; this is a fourth specialist alongside them.

Worth noting one asymmetry, since it came up on #19264: the observation that CoW vs MoR
shouldn't matter because queries have the same semantics is true for reads, which is
all the current loop does. At design time it's one of the most consequential choices
on the table, with a whole tradeoff surface behind it and a table rewrite as the cost of
changing your mind. Design-time and read-time agents need different knowledge.

Three properties that distinguish this from asking a foundation model directly:

• It asks workload questions, never Hudi questions. A user who has never heard of an
  index can still answer "can this record move to a different date after insert?" — and
  that answer is what selects index scope.
• It refuses to guess at config. Every hoodie.* key it emits comes from the checked-in
  reference tables, and validate_config_keys.py verifies all 83 of them against
  ConfigProperty definitions in the source tree. A model configuring Hudi from memory
  produces plausible properties that don't exist — and unlike a wrong query, a wrong table
  property is durable.
• It surfaces irreversibility as a first-class output. Table type, partitioning, record
  key, bucket count, and record-index file-group count are one-way. Every ADR carries a
  durability table naming each applicable one and what unlocking costs.

Changelog — files added, all under hudi-agent-gateway/skills/hudi-architect/:

File	What it holds
SKILL.md	Flow control, four-tier gating, question-delivery contract, guardrails
references/question-flow.md	Round-by-round questions with conditional gating per tier
references/decision-tables.md	Derivation rules per decision domain (engine, writer, table type, index, partitioning, retention, services, meta-fields, key)
references/warnings.md	14 named traps, each with trigger condition, message template, and firing point
references/config-templates.md	hoodie.* templates per decision, plus sample bundles for three workload archetypes
references/adr-template.md	ADR output structure
README.md	What it is, how to invoke, what to stress-test during review
RUNBOOK.md	Operator guide for data engineering / ETL teams: setup, pre-session workload checklist, how to read the output, limitations
validate_config_keys.py + allowlist	Validates every hoodie.* key referenced by the skill against the source tree; exit 1 on unknown keys; allowlist entries each carry a documented reason

Tiering. The interview scales to intent, gated by one opening question: exploring
(concept explanation), prototyping (minimum questions for a genuinely runnable table),
productionizing (full workload interview), production-at-scale (adds index sizing math and
a derived-facts checkpoint). A prototyping user gets disclosed defaults they consent to
rather than a bundle full of placeholders.

Playtesting. Four workload archetypes were driven end-to-end while writing this, and
the fixes are folded in. Three were config defects that would otherwise have shipped:

• hoodie.compaction.target.io is denominated in MB, not bytes (default 512000 =
  500GB). Earlier drafts emitted byte counts, off by ~10^6. It is also an IO ceiling that
  throttles compaction, not a sizing target.
• hoodie.metadata.record.index.* are deprecated aliases for the global properties.
  Partitioned RLI uses hoodie.metadata.record.level.index.* and is per partition.
  Templates were setting global knobs under a partitioned heading.
• An initialized record index is not resizable. Adding an RLI later is free; the
  file-group count is durable at initialization, like bucket count.

That last class of bug is the reason validate_config_keys.py exists: correctness here has
to be mechanically checkable, not reviewed by eye.

Scope

Milestone 1 targets pipeline modelling — taking a workload description and deriving the
durable shape of the table and the ingestion pipeline around it: table type, partitioning,
record key and ordering, index, writer selection, retention, and the table-service posture
that follows from those. That's the slice where a wrong answer is most expensive, because
most of it is fixed at table creation.

This patch is raised to get community feedback on the direction and the interview itself
while the extensions are being built. Deliberately out of M1 scope, each already sketched
in the flow so it can be filled in rather than retrofitted:

Extension	Where M1 stops today
Multi-writer	The skill asks whether anything else writes the table and, if so, emits the OCC skeleton plus a blocking pre-launch item in the ADR. It does not pick a lock provider or model conflict behavior. Provider choice, OCC vs NBCC, and conflict resolution are the extension.
Table-services operationalization	Cleaner, archival, and compaction posture are derived from cadence, but only as design-time settings. Tuning against observed behavior — compaction backlog, clustering, layout optimization — is deliberately pushed to an operations agent and named as such in the guardrails. This is the natural join with the optimization sub-agent in #19263.
Flink	Asked as an engine option and routed, but the Flink derivation path (bucket-index dominance, different reader configs, checkpoint interaction) is not yet built out. Spark is what has been driven end-to-end.
Catalog / metastore sync	hoodie.datasource.hive_sync.* is out of scope; the ADR flags it as a pre-launch item when a catalog-based engine reads the table.
Non-Kafka source configs	Kafka gets full source-class and schema-provider derivation. DFS/JDBC/Pulsar/Kinesis are routed but leave source properties to the user.

Also deferred: benchmarking, record-level TTL, z-order/layout guidance, multi-table
transactions, CONSISTENT_HASHING bucket sizing, and versions other than 1.2.0. All of
these are stated as limitations in RUNBOOK.md §8 rather than being silently missing — the
skill defers honestly instead of guessing, and anything it declines to decide lands in the
ADR's open-questions section as a pre-launch action item.

One structural gap worth naming rather than leaving to be discovered: there is no rule
engine. M1 is knowledge and flow only, so config correctness rests on the checked-in
reference tables plus the key validator, not on model inference. A hallucinated hoodie.*
becomes a durable table property, not a re-runnable wrong answer, so the derivation logic —
workload profile schema, decision functions, property generator, fixture tests — is owed as
a follow-up in this module's idiom before this drives anything unattended. Landing the
knowledge first makes that work reviewable against something concrete, and the feedback from
this patch is what should shape it.

How to try it

The skill is a directory of markdown loaded by an agent host — nothing to build, and no Hudi
cluster, installation, or cloud access needed. It designs; it never deploys, modifies tables,
or applies configuration.

With Claude Code (or any agent host that supports skills):

# from a checkout of this branch
mkdir -p ~/.claude/skills
cp -r hudi-agent-gateway/skills/hudi-architect ~/.claude/skills/

claude          # then type:  /hudi-architect

Or drop it in a project instead of user-level: mkdir -p .claude/skills && cp -r hudi-agent-gateway/skills/hudi-architect .claude/skills/. For a data platform team, the
project-level copy checked into your pipelines repo is the pattern worth using — everyone gets
the same advisor, and upgrades ship through normal review.

Without Claude Code, the skill degrades to a readable spec: paste SKILL.md as a system
prompt into any capable LLM chat (the question flow falls back from interactive widgets to
numbered prose blocks), or just read references/ as design documentation — the decision
tables and warnings stand on their own.

The first question is always the tier gate (exploring / prototyping / productionizing /
production-at-scale); answer it honestly, because it controls which checks run. Then describe
your workload. RUNBOOK.md §4 has a pre-session checklist worth skimming first — sessions go
much faster if you have the source, cadence, mutability, and read-pattern facts to hand.

To verify the config keys independently:

python3 hudi-agent-gateway/skills/hudi-architect/validate_config_keys.py

Exits 0 when every hoodie.* key referenced anywhere in the skill resolves to a real
ConfigProperty in this source tree (83 keys at present); exit 1 lists any that don't. Run it
after editing a reference file. It must run from inside a Hudi checkout — it resolves keys from
the actual source, not a snapshot.

What feedback would be most useful

The interview is the part that benefits most from review by people who have designed real
Hudi tables, since its quality is entirely in which questions fire and in what order:

• Run a table you already operate through the flow and compare the design it produces
  against what you actually built. Divergence in either direction is the most valuable
  signal here — either the skill is wrong, or it found something.
• Questions that shouldn't be asked, or that are asked in Hudi terms where a workload
  phrasing exists. The design rule is that no question should be asked whose answer wouldn't
  change the recommendation.
• Traps that are missing from references/warnings.md, or that fire at the wrong moment.
  A warning whose only firing point is a checkpoint a tier skips is a warning that never
  fires — one round of that class of bug is already fixed in this patch, and there may be more.
• Derivations you disagree with in references/decision-tables.md, especially around
  index selection and the retention safe-maximums per commit cadence.
• Whether the extension ordering above matches what users actually hit first — multi-writer
  and table-services operationalization are my read of the priority; Flink ahead of both is a
  reasonable argument.

Impact

No impact on existing behavior. Purely additive: 10 new files, all documentation and one
standalone validation script. No existing file is modified, no Hudi API, config, or storage
format is touched, and nothing is wired into the gateway's runtime — the skill is loaded by
an external agent host (Claude Code, or any MCP-speaking client), not by
hudi_agent_gateway. setuptools package discovery is where = ["src"], so skills/ is
not swept into the wheel.

User-facing gain: a supported path from workload requirements to a reviewed Hudi table
design, without needing to read the config reference first.

Risk Level

low

Nothing executes in any Hudi code path. The only executable file is
validate_config_keys.py, a developer-run script that reads markdown and Java/Scala
sources and exits non-zero on unknown config keys; it never writes.

The residual risk is advice quality — a user could act on a design that is wrong for
their workload. Mitigations:

• Every hoodie.* key is machine-verified against ConfigProperty definitions (83 keys,
  currently clean). Run: python3 hudi-agent-gateway/skills/hudi-architect/validate_config_keys.py
• The skill is explicitly barred from inventing config combinations not present in
  references/; unresolved decisions must be surfaced as open questions instead.
• It cannot deploy, modify tables, or apply configuration — design-time only, by guardrail.
• Every output separates confirmed facts from inferred facts from assumptions, so a reviewer
  can tell what was checked from what was guessed.
• One-way decisions are flagged inline during the conversation and again in the ADR's
  durability table.
• Four workload archetypes driven end-to-end; the resulting corrections are in this PR.

Known gaps in that verification: the exploration tier and the Flink path have not been
driven end-to-end yet, and the multi-writer question path (production-at-scale only) hasn't
been exercised live.

Documentation Update

None required for this PR — no new configs, no changed defaults, no public API or storage
change. The skill is self-documenting (README.md for reviewers, RUNBOOK.md for
operators).

Website documentation would be worth adding once the direction is agreed and the follow-up
rule engine lands; happy to take that as a separate contribution if reviewers want it
surfaced on hudi.apache.org rather than discovered in-repo.

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[nsivabalan]

Sample run w/ the architect agent:

[bhasudha]

Thanks for contributing this. Seems valuable.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.19%. Comparing base (4a5d5b0) to head (c98712c).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19380      +/-   ##
============================================
+ Coverage     70.49%   72.19%   +1.69%     
+ Complexity    32775    32574     -201     
============================================
  Files          2700     2561     -139     
  Lines        152896   148305    -4591     
  Branches      19009    18651     -358     
============================================
- Hits         107778   107062     -716     
+ Misses        36596    32774    -3822     
+ Partials       8522     8469      -53     

Components	Coverage Δ
hudi-common	81.39% <ø> (-0.02%)	⬇️
hudi-client	81.58% <ø> (+<0.01%)	⬆️
hudi-flink	78.61% <ø> (-0.02%)	⬇️
hudi-spark-datasource	58.66% <ø> (+0.28%)	⬆️
hudi-utilities	70.42% <ø> (+0.04%)	⬆️
hudi-cli	15.32% <ø> (ø)
hudi-hadoop	63.49% <ø> (ø)
hudi-sync	70.62% <ø> (ø)
hudi-io	79.57% <ø> (ø)
hudi-timeline-service	83.44% <ø> (-0.30%)	⬇️
hudi-cloud	64.00% <ø> (ø)
hudi-kafka-connect	53.20% <ø> (ø)

Flag	Coverage Δ
common-and-other-modules	46.68% <ø> (+0.58%)	⬆️
flink-integration-tests	47.71% <ø> (+0.17%)	⬆️
hadoop-mr-java-client	43.32% <ø> (-0.02%)	⬇️
integration-tests	13.59% <ø> (+0.01%)	⬆️
spark-client-hadoop-common	48.71% <ø> (-0.01%)	⬇️
spark-java-tests	49.32% <ø> (+0.08%)	⬆️
spark-scala-tests	44.86% <ø> (+0.20%)	⬆️
utilities	36.48% <ø> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 159 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hudi-bot]

CI report:

• c98712c Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build