feat(writer): add COMMIT_TIME_ONLY meta-fields mode for incremental queries on minimal-meta-fields tables - #19047
Conversation
…ueries on minimal-meta-fields tables
Today the only way to opt out of populating Hudi's five meta columns
(_hoodie_commit_time, _hoodie_commit_seqno, _hoodie_record_key,
_hoodie_partition_path, _hoodie_file_name) is the all-or-nothing
hoodie.populate.meta.fields=false. That saves storage but disables
incremental queries (which require _hoodie_commit_time).
This patch adds a small, additive opt-in -- hoodie.meta.fields.commit.time.enabled --
that, when set together with hoodie.populate.meta.fields=false, additionally populates
_hoodie_commit_time on every row so incremental queries remain functional. The other
four meta columns stay null on disk, preserving the storage saving.
The three resulting modes:
- ALL (default): hoodie.populate.meta.fields=true. All five meta fields populated.
- NONE: hoodie.populate.meta.fields=false. No meta fields populated; incremental
queries rejected (existing behavior).
- COMMIT_TIME_ONLY: hoodie.populate.meta.fields=false +
hoodie.meta.fields.commit.time.enabled=true. Only _hoodie_commit_time populated.
The combination populate.meta.fields=true + commit.time.enabled=true is
ambiguous and is rejected up-front at writer init.
Default behavior is unchanged. The new property defaults to false; tables that
opt out of populate.meta.fields without setting the new flag continue to behave
exactly as before (NONE mode).
Why a separate boolean instead of a single enum:
- Bit-identical backward compatibility. Every existing table on disk resolves to
ALL or NONE without any new property being read. No reader-side migration.
- Pre-1.3.0 readers ignore the new property safely: they see
populate.meta.fields=false and behave as a NONE reader, which is consistent
(they cannot do incremental queries; the _hoodie_commit_time column they see
is just unobserved).
- Encodes "additive" structurally. The new flag only modifies a NONE table -- it
is literally a NONE table plus one populated column. Most existing code paths
that branch on populate.meta.fields keep working unchanged; only paths that
specifically need commit_time need to consult the new accessor.
Changes:
- New table property HoodieTableConfig.META_FIELDS_COMMIT_TIME_ENABLED.
- New accessors HoodieTableConfig.isCommitTimeOnlyMetaFieldsMode(),
isCommitTimePopulated(), isRecordKeyPopulated() -- three named predicates so
callers do not need bool[5]-style per-field plumbing.
- HoodieWriteConfig pass-through accessors + Builder.withMetaFieldsCommitTimeEnabled().
- HoodieWriteConfig.validate() rejects the populate=true + commit.time=true combination.
- HoodieTableMetaClient.TableBuilder.setMetaFieldsCommitTimeEnabled() persists the flag.
- HoodieSparkSqlWriter wires both fresh-table and bootstrap creation paths.
- Writer-engine touchpoints (HoodieAvroParquetWriter, HoodieSparkParquetWriter,
HoodieRowCreateHandle): when commitTimeOnly is true, populate _hoodie_commit_time
and the derived seq id; leave the other four columns null. Bloom-filter / record-
key index registration is intentionally skipped because the record-key column is
not populated.
- Incremental query rejection sites (IncrementalRelationV1/V2,
MergeOnReadIncrementalRelationV1/V2) now check isCommitTimePopulated() rather
than populateMetaFields(), so COMMIT_TIME_ONLY tables are accepted.
Tests:
- TestHoodieMetaFieldsMode (7 tests): all three modes resolve correctly from
HoodieTableConfig; the ambiguous combo is reported as ALL mode for accessor
defensiveness; HoodieConfig view yields identical semantics.
- TestHoodieWriteConfigMetaFieldsMode (6 tests): writer-builder surface; the
invalid combination is rejected at build() with a message naming both
properties.
- TestMetaFieldsCommitTimeOnly (4 tests, end-to-end through Spark datasource):
ALL / NONE / COMMIT_TIME_ONLY all persist the right properties and the
read-back table config reports the right mode; the invalid combination is
rejected at writer init.
- TestHoodieTableConfig: definedTableConfigs() count assertion updated to 45.
Closes apache#18383
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for working on this! This PR adds an additive hoodie.meta.fields.commit.time.enabled flag so that tables written with populate.meta.fields=false can still populate _hoodie_commit_time and keep incremental queries functional, wiring it through the write/table configs, the Spark and Avro parquet writers, the row bulk-insert handle, and the incremental relations. The CoW write/read paths look consistent; the main thing worth double-checking is how this interacts with MoR tables, since the incremental guards are relaxed for MoR too but the log-write path isn't updated for this mode (details in the inline comments). Please take a look at any inline comments, and this should be ready for a Hudi committer or PMC member to take it from here. A few small readability nits below — an unnecessary string concat in the doc, a bypassed accessor in validate(), and inconsistent assertion style in the test.
|
|
||
| if (!this.tableConfig.populateMetaFields()) { | ||
| throw new HoodieException("Incremental queries are not supported when meta fields are disabled") | ||
| if (!this.tableConfig.isCommitTimePopulated()) { |
There was a problem hiding this comment.
🤖 This also enables incremental queries on MoR tables in COMMIT_TIME_ONLY mode, but the MoR log-write path (HoodieAppendHandle.populateMetadataFields) still only writes _hoodie_commit_time when populateMetaFields() is true — so log-file records get a null commit-time and would be silently dropped by the IsNotNull/range filters in incrementalSpanRecordFilters (and the merge also needs _hoodie_record_key, which is null in this mode). Should the mode be restricted to CoW, or should the append handle populate commit-time too? Same applies to V2. @yihua does this match your read of the MoR incremental path?
| .withDocumentation("When enabled, populates all meta fields. When disabled, no meta fields are populated " | ||
| + "and incremental queries will not be functional. This is only meant to be used for append only/immutable data for batch processing"); | ||
|
|
||
| public static final ConfigProperty<Boolean> META_FIELDS_COMMIT_TIME_ENABLED = ConfigProperty |
There was a problem hiding this comment.
🤖 Is enabling this flag on a pre-existing populate.meta.fields=false table meant to be guarded? HoodieWriterUtils.validateTableConfig only flags a conflict when the existing value is non-null, so a null→true transition on an older NONE table isn't blocked. After that, incremental queries over a range that includes pre-enablement commits would silently drop those rows since their _hoodie_commit_time is null on disk.
| public static final ConfigProperty<Boolean> META_FIELDS_COMMIT_TIME_ENABLED = ConfigProperty | ||
| .key("hoodie.meta.fields.commit.time.enabled") | ||
| .defaultValue(false) | ||
| .withDocumentation("Only meaningful when " + "hoodie.populate.meta.fields=false. When set to true, the writer " |
There was a problem hiding this comment.
🤖 nit: the "Only meaningful when " + splits a single thought across two adjacent string literals for no reason — could you merge them into one string so the concatenation isn't surprising to whoever reads this next?
| // COMMIT_TIME_ONLY mode is an additive opt-in on top of populate.meta.fields=false. Setting | ||
| // both flags to true is ambiguous (the COMMIT_TIME_ONLY flag has no effect when all meta | ||
| // fields are already populated) so reject it explicitly rather than silently ignore. | ||
| boolean populateMetaFields = writeConfig.getBooleanOrDefault(HoodieTableConfig.POPULATE_META_FIELDS); |
There was a problem hiding this comment.
🤖 nit: writeConfig.populateMetaFields() already encodes this read — could you use it here instead of reaching through getBooleanOrDefault directly?
| .withMetaFieldsCommitTimeEnabled(false) | ||
| .build(); | ||
| assertEquals(false, cfg.populateMetaFields()); | ||
| assertEquals(false, cfg.isCommitTimeOnlyMetaFieldsMode()); |
There was a problem hiding this comment.
🤖 nit: the three assertEquals(false, ...) calls here are inconsistent with assertFalse(...) used everywhere else in this class — could you swap them for assertFalse to match?
|
closing this in favor of #19205 |
5 participants
Describe the issue this Pull Request addresses
Today the only way to opt out of populating Hudi's five meta columns is the all-or-nothing
hoodie.populate.meta.fields=false. That saves storage but disables incremental queries (which require_hoodie_commit_time).A community user surfaced this trade-off (#18383, also discussed at #17959). The concrete ask was: "give me the storage saving without giving up incremental queries." A separate exploratory PR (#18384) attempted a fully orthogonal exclude-list with per-field branching across the writer/reader paths; that surface ended up being ~2300 lines across 87 files. This PR proposes a simpler, scoped alternative: three named modes instead of the full 2^5 matrix.
Closes #18383.
Summary and Changelog
Adds an additive opt-in flag,
hoodie.meta.fields.commit.time.enabled, that — when set together withhoodie.populate.meta.fields=false— additionally populates_hoodie_commit_timeso incremental queries remain functional. The remaining four meta columns stay null on disk, preserving the storage saving.The three resulting modes:
populate.meta.fieldsmeta.fields.commit.time.enabledtrue(default)falsefalse(default)populate.meta.fields=falsefalsetruetruetrueWhy a separate boolean instead of a single enum
populate.meta.fields=false, and behave as a NONE reader — they cannot do incremental queries on the table, but they don't produce silent wrong results either.populate.meta.fieldskeep working unchanged; only paths that specifically need commit_time consult the new accessor.Plug points
Config + accessors (
hudi-common/hudi-client-common):HoodieTableConfig.META_FIELDS_COMMIT_TIME_ENABLEDproperty.isCommitTimeOnlyMetaFieldsMode(),isCommitTimePopulated(),isRecordKeyPopulated()— three named predicates.HoodieWriteConfigpass-throughs +Builder.withMetaFieldsCommitTimeEnabled().HoodieWriteConfig.validate()rejects thepopulate=true+commit.time=truecombination at build time.HoodieTableMetaClient.TableBuilder.setMetaFieldsCommitTimeEnabled()persists the flag onhoodie.propertiesat table init.HoodieSparkSqlWriterwires both fresh-table and bootstrap creation paths.Writer engines:
HoodieAvroParquetWriter,HoodieSparkParquetWriter,HoodieRowCreateHandleeach gain acommitTimeOnlyconstructor overload. WhencommitTimeOnly && !populateMetaFields, they populate_hoodie_commit_timeand the derived seq id; the other four columns stay null. Bloom-filter / record-key index registration is intentionally skipped (the record-key column is not populated).Read path (incremental query rejection):
IncrementalRelationV1/V2,MergeOnReadIncrementalRelationV1/V2now checkisCommitTimePopulated()rather thanpopulateMetaFields()— COMMIT_TIME_ONLY tables are accepted, NONE tables remain rejected with a clearer message.Scope
Impact
hoodie.meta.fields.commit.time.enabled(defaultfalse). Only meaningful whenhoodie.populate.meta.fields=false. Persisted onhoodie.propertiesat table init.Risk Level
low
Additive change with a narrow scope. The default path is untouched. The validation guard rejects the ambiguous combination loudly at writer init. Existing
TestHoodieTableConfigregression coverage (93 tests) passes unchanged.Documentation Update
hoodie.meta.fields.commit.time.enableddocumented via@ConfigPropertyannotation onHoodieTableConfig.Contributor's checklist