feat(cli): add table set-meta-fields-mode command - #19206
Conversation
7112d91 to
eb0cc53
Compare
…pulation on CoW tables
Introduces `hoodie.meta.fields.mode` — a comma-separated list of meta columns to populate
when `hoodie.populate.meta.fields=false`. Allowed tokens are `_hoodie_commit_time` and
`_hoodie_file_name`; any other token is rejected up-front. The mode is immutable at runtime
(settable only at table creation, via hudi-cli, or during upgrade).
Motivation: the existing all-or-nothing `populate.meta.fields=false` disables incremental
queries (needs `_hoodie_commit_time`) and file-level pruning / debugging lookups
(needs `_hoodie_file_name`). This lets a table opt into either or both without paying for
the remaining three meta columns.
Resulting modes:
- populate.meta.fields=true → ALL (default; mode list ignored)
- populate.meta.fields=false, mode="" → NONE
- populate.meta.fields=false, mode=<subset> → selective (COMMIT_TIME_ONLY /
FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME)
- populate.meta.fields=true, mode=<subset> → rejected at writer init
Scope in this patch:
- Spark writer path (Avro + Row): HoodieAvroParquetWriter, HoodieSparkParquetWriter,
HoodieRowCreateHandle take a Set<String> meta-fields mode and populate the columns
selectively; bloom filter / record-key index registration stays skipped when record key
is not populated.
- Incremental read path: CoW IncrementalRelationV1/V2 accept the new mode via
isCommitTimePopulated(); MoR incremental relations keep the strict populate=true guard
until the MoR log-write path is updated in a follow-up.
Fail-fast at writer init:
- Unknown token in the mode list (parser throws).
- populate.meta.fields=true combined with a non-empty mode (ambiguous).
- MERGE_ON_READ + non-empty mode (log-write path not yet wired; follow-up).
- Non-Spark engine + non-empty mode (Flink RowData / Java-client not yet wired; follow-up).
Runtime immutability:
- HoodieWriterUtils.validateTableConfig now explicitly rejects a null-on-disk → non-empty
transition on the mode property (existing loop only flagged the mismatch when the on-disk
value was non-null, which let this silent-drop path slip through).
Tests:
- TestHoodieMetaFieldsMode (hudi-hadoop-common) — accessor coverage for every combination
plus unknown-token rejection and whitespace tolerance.
- TestHoodieWriteConfigMetaFieldsMode (hudi-client-common) — builder + validate() coverage.
- TestMetaFieldsMode (hudi-spark) — end-to-end write / re-read for every mode plus the
three rejection paths (populate=true+mode, unknown token, MoR+mode).
…enum Addresses review feedback on apache#19205: - Fold (populateMetaFields boolean, comma-separated mode) into a single MetaFieldsMode enum (ALL / NONE / COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME). The enum is the sole argument passed through every writer factory and writer constructor. populate.meta.fields stays on the config surface for backward compat and is auto-derived from the enum. - On-disk representation is now the enum name (e.g. "COMMIT_TIME_ONLY") instead of a comma-separated token list. Existing tables without the property continue to fall back to ALL or NONE based on the legacy hoodie.populate.meta.fields boolean. - Drop _hoodie_commit_seqno population from the selective write path — only _hoodie_commit_time is needed for incremental queries. (Review comments on HoodieRowCreateHandle:193, HoodieSparkParquetWriter:100, HoodieAvroParquetWriter:103.) - Extend the IncrementalRelation error message to state that hoodie.meta.fields.mode is physical-storage and cannot be flipped by changing write options — existing tables must be recreated to change it. - StreamSync.initializeEmptyTable() was missing the mode pass-through — wired via setMetaFieldsModeFromString. All other non-test initTable call sites either use fromProperties() (which now routes the mode through) or intentionally don't touch meta fields (CLI table create, MDT, sample writes). Test coverage additions: - TestMetaFieldsMode (hudi-spark): end-to-end assertions read parquet back and verify which meta columns are populated / null per mode. Covers row-writer path (bulk_insert with row.writer.enable=true, default) AND non-row-writer path (insert with row.writer.enable=false). Adds inline-clustering coverage for all 5 modes verifying that clustered files preserve the mode's column population semantics. - TestHoodieStreamerMetaFieldsMode (hudi-utilities): parameterized end-to-end test through the HoodieStreamer entrypoint covering all 5 modes plus MoR+selective rejection.
eb0cc53 to
d265102
Compare
|
Rebased onto the refreshed PR #19205 head (which is itself now rebased on latest master, 2026-07-19). No new code changes on top of what's in #19205 aside from the CLI commit itself. Self-review polish:
Prior CI failures inherited from PR #19205 (co-author trailer, TestHoodieTableConfig, TestHoodieSparkSqlWriter, TestHoodieStreamerMetaFieldsMode) should now be resolved automatically since we're on the rebased #19205 head. One flaky non-related CI failure from the last run: Discussion points for reviewer:
Not blocking either way — happy to fold in whichever direction you prefer. |
Previously the populateMetaFields=false branch of HoodieDatasetBulkInsertHelper projected the five Hudi meta columns as Alias(Literal(UTF8String.EMPTY_UTF8, StringType), name). A non-null Literal derives a non-null Alias, so the resulting StructField had nullable=false. Under ALL mode this was harmless because HoodieRowCreateHandle.writeRow unconditionally fills every meta slot with a non-null value. Once selective / NONE modes started leaving the opted-out columns null on the row (per hoodie.meta.fields.mode), the non-nullable StructType led SparkToParquetSchemaConverter to declare the columns as `required binary` on disk, and any subsequent read failed with ParquetDecodingException: could not read bytes at offset 0. Switch the stub to Alias(Literal.create(null, StringType), name) so the projected StructField is nullable and the physical parquet column is written as OPTIONAL. Handle-level population (writeRow / writeRowNoMetaFields / writeRowSelectiveMetaFields) is unchanged — the stub value is never observed on disk; the fix only affects the parquet schema. Fixes all row-writer + clustering selective-mode failures in TestMetaFieldsMode (commitTimeOnly / fileNameOnly / commitTimeAndFileName / none, plus clustering variants). 18/18 tests pass locally.
…elds modes TableSchemaResolver.getTableSchema() and getTableSchema(String timestamp) decided whether to include the five Hudi meta columns in the projected schema based on populateMetaFields(). Under the selective modes introduced in this PR (COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME), populateMetaFields() returns false but the opted-in meta columns are still physically present in every Parquet file. The old logic silently stripped those columns from the read schema, so downstream consumers (notably the incremental relations, whose range filter projects _hoodie_commit_time) saw no meta columns at all and produced zero-row reads. Switch the decision to "include meta fields whenever the table's mode is not NONE." That preserves the previous behaviour for the two originally supported modes (ALL -> true, NONE -> false) and correctly turns the flag on for the three new selective modes. No schema change on disk; this only changes what the read side projects.
Adds a hudi-cli command to toggle hoodie.meta.fields.mode on an existing table: table set-meta-fields-mode --target-mode <MODE> [--force true|false] Where <MODE> is one of ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME. This is the sanctioned way to change the property outside of table creation — hoodie.meta.fields.mode is immutable at runtime because it is a physical-storage decision baked into files at write time. Safety guard: - On a table that already has commits, the command refuses to change the mode by default. Existing files were written under the current mode and are not rewritten by this command; new commits would be written under the new mode, producing mixed-mode files whose incremental / file-pruning semantics differ between old and new data. - Pass --force to override the guard. A warning is logged describing the specific data correctness impact. Behavior: - ALL and NONE are persisted implicitly (via populate.meta.fields=true/false); the hoodie.meta.fields.mode property is cleared when transitioning to ALL/NONE. - Selective modes (COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME) are persisted explicitly on hoodie.properties alongside populate.meta.fields=false. - No-op when target matches current mode. Test coverage (TestTableCommand): - All 5 modes settable on a fresh table. - Selective → ALL clears the property. - No-op when target matches current mode. - Rejects unknown enum values. - Refuses on a populated table without --force; the mode does not change. - Accepts on a populated table with --force; the mode changes. Follow-up to PR apache#19205 (hoodie.meta.fields.mode). PR-C (MoR support) is the next follow-up.
d265102 to
4f5b396
Compare
voonhous
left a comment
voonhous
left a comment
There was a problem hiding this comment.
Reviewed the CLI commit (4f5b396) on its own; the rest of the diff belongs to #19205 and I have kept off what danny0405 and cshuo already raised there.
The command itself is a reasonable thing to add, but I do not think it is mergeable in this shape. Main themes, details inline:
- It can brick a table. No MoR / non-Spark guard, and on a fresh table
--forceis not required, so a selective mode gets written onto a table the writer will then refuse to write to. --forcewidening is worse than documented. It is an NPE on read, not dropped rows, and it permanently disables the guard that has forbidden re-enabling meta fields since 2021.- The property write is not atomic.
delete()thenupdate()passes through -- and can strand the table in --NONE, whichtable recover-configscannot undo. - Four of the nine tests can pass without testing what their names say, and
hudi-clitests do not run in CI at all, so the green Azure run does not cover any of this. - The stack is 9 commits stale, and the ALL/NONE clearing is the design #19205 dropped in
c78dc962/664ff2ecanswering the earlier P1.
Two things you flagged as open questions both check out fine, no action needed:
- MoR safety guard: works.
getCommitsTimeline()is{commit, deltacommit, replacecommit, clustering}(BaseTimelineV2.java:89-91) and filters on action only, so deltacommit-only and replacecommit-only tables all trip it. - Bare
--force: parses totrue. Spring Shell 2.1.1 gives abooleanparameter arity zero and defaults a valueless flag totrue(CommandParser.java:421-427). Same as every other boolean flag in hudi-cli.
| } | ||
|
|
||
| // Safety check: refuse to change the mode on a table with commits unless --force. | ||
| int commitCount = client.getActiveTimeline().getCommitsTimeline().countInstants(); |
There was a problem hiding this comment.
This lets you set a selective mode on a MoR table, and on a fresh table there are no commits so --force is not even needed. The writer already rejects that combination.
HoodieWriteConfig.java:3936-3947 (added by this stack's own base commit 2b11a4fa) throws for MERGE_ON_READ + selective and for engine != SPARK + selective. HoodieSparkSqlWriter.scala:1102-1106 copies every table property into the write config, so once this command has written the mode, every subsequent write fails validation. The table is bricked and the CLI gave no warning.
Please reject selective modes up front when the table is MoR (or the writer is not Spark), reusing the writer's message. --force should not override this one -- it is an unimplemented code path, not a data-correctness tradeoff.
Worth a create --tableType MERGE_ON_READ case too; all 9 new tests go through prepareTable(), which is CoW only.
| commitCount, currentMode, targetMode)); | ||
| } | ||
| if (commitCount > 0) { | ||
| log.warn("--force passed: changing hoodie.meta.fields.mode from {} to {} on a table with " |
There was a problem hiding this comment.
The warning undersells this. Widening (COMMIT_TIME_ONLY -> ALL, NONE -> anything) does not silently drop rows -- it throws.
Under a selective mode _hoodie_record_key is a physically present NULL column. Once this sets populate.meta.fields=true, RecordContext.java:76-78 switches to metadataKeyExtractor(), which is an unconditional .toString() on that null (RecordContext.java:438-439). Every read of a pre-existing file group NPEs.
It is also the one way to defeat the guard permanently: BaseHoodieWriteClient.java:1552 compares writer vs table config, so rewriting the table config means it can never fire again. That invariant goes back to d5026e9a2485 (HUDI-2161, 2021), and #19205 restates it per-mode via isWiderThan.
Suggest making widening a hard refusal that --force cannot override, and keeping --force for narrowing only.
| Properties toUpdate = new Properties(); | ||
| toUpdate.setProperty(HoodieTableConfig.POPULATE_META_FIELDS.key(), | ||
| String.valueOf(targetMode == MetaFieldsMode.ALL)); | ||
| if (targetMode == MetaFieldsMode.ALL || targetMode == MetaFieldsMode.NONE) { | ||
| // Selective mode is being cleared; delete the property rather than write empty string. | ||
| HoodieTableConfig.delete(client.getStorage(), client.getMetaPath(), | ||
| Collections.singleton(HoodieTableConfig.META_FIELDS_MODE.key())); | ||
| HoodieTableConfig.update(client.getStorage(), client.getMetaPath(), toUpdate); | ||
| } else { | ||
| toUpdate.setProperty(HoodieTableConfig.META_FIELDS_MODE.key(), targetMode.name()); | ||
| HoodieTableConfig.update(client.getStorage(), client.getMetaPath(), toUpdate); | ||
| } |
There was a problem hiding this comment.
These are two independent modify() cycles (HoodieTableConfig.java:552-600), and the state between them is NONE -- meta fields off entirely, worse than either endpoint. If the second call fails the table stays there: modify() drops its backup on success, and recoverIfNeeded only kicks in when hoodie.properties is missing or invalid, so table recover-configs cannot undo it.
That breaks the one-transaction-per-mutation invariant from ae0c67d9fc7c (HUDI-2795). updateAndDeleteProps already exists for exactly this "set some, delete some" shape (HoodieTableConfig.java:616, added by b60d38c40fa1).
| Properties toUpdate = new Properties(); | |
| toUpdate.setProperty(HoodieTableConfig.POPULATE_META_FIELDS.key(), | |
| String.valueOf(targetMode == MetaFieldsMode.ALL)); | |
| if (targetMode == MetaFieldsMode.ALL || targetMode == MetaFieldsMode.NONE) { | |
| // Selective mode is being cleared; delete the property rather than write empty string. | |
| HoodieTableConfig.delete(client.getStorage(), client.getMetaPath(), | |
| Collections.singleton(HoodieTableConfig.META_FIELDS_MODE.key())); | |
| HoodieTableConfig.update(client.getStorage(), client.getMetaPath(), toUpdate); | |
| } else { | |
| toUpdate.setProperty(HoodieTableConfig.META_FIELDS_MODE.key(), targetMode.name()); | |
| HoodieTableConfig.update(client.getStorage(), client.getMetaPath(), toUpdate); | |
| } | |
| Properties toUpdate = new Properties(); | |
| toUpdate.setProperty(HoodieTableConfig.POPULATE_META_FIELDS.key(), | |
| String.valueOf(targetMode == MetaFieldsMode.ALL)); | |
| boolean implicitMode = targetMode == MetaFieldsMode.ALL || targetMode == MetaFieldsMode.NONE; | |
| if (!implicitMode) { | |
| toUpdate.setProperty(HoodieTableConfig.META_FIELDS_MODE.key(), targetMode.name()); | |
| } | |
| // Single modify() cycle: never leave the table in an intermediate mode. | |
| HoodieTableConfig.updateAndDeleteProps(client.getStorage(), client.getMetaPath(), toUpdate, | |
| implicitMode ? Collections.singleton(HoodieTableConfig.META_FIELDS_MODE.key()) | |
| : Collections.emptySet()); |
If you take the rebase fix in my other comment, the delete goes away entirely and this collapses to a plain update().
| currentMode, targetMode, commitCount); | ||
| } | ||
|
|
||
| // Persist. ALL and NONE are implicit (derived from populate.meta.fields); selective modes are |
There was a problem hiding this comment.
This PR is stacked on a snapshot of #19205 that is now 9 commits behind, and it encodes the semantics #19205 has since dropped.
c78dc962 + 664ff2ec made hoodie.meta.fields.mode the source of truth, persisted verbatim including ALL and NONE, with the legacy boolean derived from it -- that was the fix for danny0405's P1. And 4134dac6's NineToTenUpgradeHandler now writes the property explicitly onto exactly the tables this command would strip it from.
Please rebase and drop the clearing branch: always write META_FIELDS_MODE=targetMode.name() plus the derived boolean, in one update(). The assertion at TestTableCommand.java:374 pins the old representation and will need inverting.
| } | ||
|
|
||
| @ShellMethod(key = "table set-meta-fields-mode", | ||
| value = "Set hoodie.meta.fields.mode on an existing table. Refuses to change the mode on a " |
There was a problem hiding this comment.
Accuracy point on "the sanctioned way": table update-configs sits 30 lines above and will set hoodie.meta.fields.mode from a props file with zero validation (:228-244) -- atomically, in a single modify cycle. table delete-configs and repair overwrite-hoodie-props are equally open.
Either move the transition check somewhere all writers go through, or have update-configs / delete-configs refuse this key and point here. Otherwise worth softening the claim in the PR description.
| assertFalse(ShellEvaluationResultUtil.isSuccess(result)); | ||
| assertTrue(result.toString().contains("BOGUS_MODE"), | ||
| "expected error message to name the rejected value, got: " + result); |
There was a problem hiding this comment.
This passes even if the command is not registered at all.
ShellEvaluationResultUtil.isSuccess is just !(x instanceof Throwable), and Spring Shell 2.1.1 returns a CommandNotFound (a RuntimeException) whose message echoes the input line verbatim. So a typo'd @ShellMethod key yields No command found for 'table set-meta-fields-mode --target-mode BOGUS_MODE', which satisfies both assertions.
Assert on text only the production code can emit:
| assertFalse(ShellEvaluationResultUtil.isSuccess(result)); | |
| assertTrue(result.toString().contains("BOGUS_MODE"), | |
| "expected error message to name the rejected value, got: " + result); | |
| assertInstanceOf(HoodieException.class, result); | |
| assertTrue(result.toString().contains("Unsupported --target-mode 'BOGUS_MODE'"), | |
| "expected error message to name the rejected value, got: " + result); |
(needs org.junit.jupiter.api.Assertions.assertInstanceOf and org.apache.hudi.exception.HoodieException imports)
| assertTrue(result.toString().contains("Refusing to change") || result.toString().contains("--force"), | ||
| "expected refusal message, got: " + result); | ||
|
|
||
| // Mode must not have changed. | ||
| assertEquals(MetaFieldsMode.ALL, | ||
| HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode()); |
There was a problem hiding this comment.
Two things here.
The || is redundant slack -- I checked the plausible alternate failures and none satisfy it today, but there is no upside to keeping it.
More importantly, line 415 reads getTableConfig() off the metaclient snapshot taken before the command ran (HoodieTableMetaClient holds tableConfig as a plain field), and the command throws at :298 before its final refresh. So this would read ALL even if the command had written to disk and then thrown -- it cannot catch a write-before-guard regression. HoodieTableMetaClient.reload(...) is the idiom used in TestRepairsCommand.java:218.
| assertTrue(result.toString().contains("Refusing to change") || result.toString().contains("--force"), | |
| "expected refusal message, got: " + result); | |
| // Mode must not have changed. | |
| assertEquals(MetaFieldsMode.ALL, | |
| HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode()); | |
| assertTrue(result.toString().contains( | |
| "Refusing to change hoodie.meta.fields.mode on a table that already has 1 commit(s)"), | |
| "expected refusal message, got: " + result); | |
| // Mode must not have changed. | |
| assertEquals(MetaFieldsMode.ALL, | |
| HoodieTableMetaClient.reload(HoodieCLI.getTableMetaClient()).getTableConfig().getMetaFieldsMode()); |
| assertTrue(ShellEvaluationResultUtil.isSuccess(result)); | ||
| assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, | ||
| HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode()); |
There was a problem hiding this comment.
These two assertions are a strict subset of testSetMetaFieldsModeOnFreshTableToCommitTimeOnly. The only thing making this a distinct scenario is that createDummyCommitFile produced a counted instant, and nothing here pins that -- if the fixture ever stops being recognised, this quietly becomes a duplicate of the fresh-table test and still passes green.
| assertTrue(ShellEvaluationResultUtil.isSuccess(result)); | |
| assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, | |
| HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode()); | |
| assertTrue(ShellEvaluationResultUtil.isSuccess(result)); | |
| // Pin the precondition: --force must be the branch actually under test. | |
| assertEquals(1, HoodieCLI.getTableMetaClient().getActiveTimeline() | |
| .getCommitsTimeline().countInstants()); | |
| assertEquals(MetaFieldsMode.COMMIT_TIME_ONLY, | |
| HoodieCLI.getTableMetaClient().getTableConfig().getMetaFieldsMode()); |
| } | ||
|
|
||
| private void createDummyCommitFile(String instantTime) throws IOException { | ||
| // Timeline v2 layout: files live under .hoodie/timeline/. Writing an empty completed commit |
There was a problem hiding this comment.
This hand-rolls the v2 timeline layout when the helper already exists and is already used in this same file at line 217: HoodieTestDataGenerator.createCommitFile(tablePath, instantTime, storageConf()). It is imported at line 37, and the sibling command's test does the same thing (ITTestTableCommand.java:81-82).
The helper also writes real commit metadata and a valid instant time -- instantTime + "1" here is an 18-char string that is not a valid Hudi timestamp.
Suggest deleting this method and calling the helper at both call sites (405 and 422). That also removes the fully-qualified java.nio.file.* references.
| assertFalse(client.getTableConfig().populateMetaFields()); | ||
| } | ||
|
|
||
| @Test |
There was a problem hiding this comment.
nit, feel free to ignore: these three fresh-table tests differ only by the mode string. @ParameterizedTest + @EnumSource(names = {"COMMIT_TIME_ONLY", "FILE_NAME_ONLY", "COMMIT_TIME_AND_FILE_NAME"}) would collapse them and let all three carry the full assertion set -- right now FILE_NAME_ONLY and COMMIT_TIME_AND_FILE_NAME skip the assertFalse(populateMetaFields()) check the first one has. That check stops being implied once #19205's resolve() lands, since it no longer consults the boolean.
Keep ...ToNone separate (it deletes the key rather than writing it), and do not hoist prepareTable() into @BeforeEach -- testCreateWithSpecifiedValues and testRefresh would start failing with "Table already existing".
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for working on this! This PR adds a table set-meta-fields-mode hudi-cli command and wires hoodie.meta.fields.mode through the write and HoodieStreamer table-init paths, preserving backward-compatible behavior for existing tables. One edge case is worth a closer look — in HoodieTableMetaClient.TableBuilder.build(), a selective metaFieldsMode being set while populateMetaFields is still null. Please take a look at the inline comments, and this should be ready for a Hudi committer or PMC member to take it from here. A couple of small naming/style suggestions below.
| // - Explicit selective mode wins → validate compatibility with populateMetaFields and write. | ||
| // - populateMetaFields=false + no explicit mode → NONE (leave property empty for backward compat). | ||
| // - populateMetaFields=true + no explicit mode → ALL (implicit; leave property empty). | ||
| if (null != metaFieldsMode) { |
There was a problem hiding this comment.
🤖 If a selective metaFieldsMode is set while populateMetaFields is still null (e.g. a properties source that has hoodie.meta.fields.mode but not hoodie.populate.meta.fields, since fromProperties only calls setPopulateMetaFields when the key is present), neither guard fires: the selective mode gets persisted but POPULATE_META_FIELDS is never written (line 1546). On read, populateMetaFields() defaults to true, so getMetaFieldsMode() resolves back to ALL and the selective mode is silently dropped — leaving hoodie.properties claiming e.g. COMMIT_TIME_ONLY while writers actually run in ALL mode. Should build() default populateMetaFields=false (or reject) when a selective mode is set without an explicit populate flag?
| @@ -57,6 +57,9 @@ protected HoodieFileWriter newParquetFileWriter( | |||
| String instantTime, StoragePath path, HoodieConfig config, HoodieSchema schema, | |||
| TaskContextSupplier taskContextSupplier) throws IOException { | |||
There was a problem hiding this comment.
🤖 nit: MetaFieldsMode is referenced with its full package name here — could you add it to the import block at the top of the file instead? The same pattern is used in HoodieAvroFileWriterFactory (line 71).
| * record key is not registered with the write support (bloom filter / RLI hooks are meaningless | ||
| * without the record-key column). | ||
| */ | ||
| private void writeRowSelectiveMetaFields(InternalRow row) { |
There was a problem hiding this comment.
🤖 nit: the magic 5 here is the number of Hoodie meta columns — could you use HoodieRecord.HOODIE_META_COLUMNS.size() (or a named constant) so a future reader doesn't have to count the fields to understand where the number comes from?
4 participants
Describe the issue this Pull Request addresses
Adds a hudi-cli command to change
hoodie.meta.fields.modeon an existing table. This is the sanctioned way to change the property outside of table creation — the property is immutable at runtime because it is a physical-storage decision baked into files at write time.Stacked on top of #19205. This PR is best reviewed after #19205 lands — the base of the diff currently includes PR-A's changes. Once #19205 merges to master, this PR will show only the ~209-line CLI addition.
Summary and Changelog
New shell command:
Where
<MODE>is one ofALL,NONE,COMMIT_TIME_ONLY,FILE_NAME_ONLY,COMMIT_TIME_AND_FILE_NAME.Safety guard
On a table that already has commits, the command refuses to change the mode by default:
Pass
--forceto override. A warning is logged describing the data-correctness impact.Behavior
populate.meta.fields=true/false; thehoodie.meta.fields.modeproperty is cleared when transitioning to ALL/NONE.hoodie.propertiesalongsidepopulate.meta.fields=false.Test coverage (`TestTableCommand`)
All 9 tests pass locally.
Impact
Risk Level
low
Narrow, additive change with a strong default-safe guard. `--force` is required to change mode on a populated table.
Documentation Update
Contributor's checklist
🤖 Generated with Claude Code