fix(config): resolve meta-fields mode at sites that still read the deprecated boolean - #19378
Open
nsivabalan wants to merge 14 commits into
Open
fix(config): resolve meta-fields mode at sites that still read the deprecated boolean#19378nsivabalan wants to merge 14 commits into
nsivabalan wants to merge 14 commits into
Conversation
…pulation on CoW tables
Introduces `hoodie.meta.fields.mode` — a comma-separated list of meta columns to populate
when `hoodie.populate.meta.fields=false`. Allowed tokens are `_hoodie_commit_time` and
`_hoodie_file_name`; any other token is rejected up-front. The mode is immutable at runtime
(settable only at table creation, via hudi-cli, or during upgrade).
Motivation: the existing all-or-nothing `populate.meta.fields=false` disables incremental
queries (needs `_hoodie_commit_time`) and file-level pruning / debugging lookups
(needs `_hoodie_file_name`). This lets a table opt into either or both without paying for
the remaining three meta columns.
Resulting modes:
- populate.meta.fields=true → ALL (default; mode list ignored)
- populate.meta.fields=false, mode="" → NONE
- populate.meta.fields=false, mode=<subset> → selective (COMMIT_TIME_ONLY /
FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME)
- populate.meta.fields=true, mode=<subset> → rejected at writer init
Scope in this patch:
- Spark writer path (Avro + Row): HoodieAvroParquetWriter, HoodieSparkParquetWriter,
HoodieRowCreateHandle take a Set<String> meta-fields mode and populate the columns
selectively; bloom filter / record-key index registration stays skipped when record key
is not populated.
- Incremental read path: CoW IncrementalRelationV1/V2 accept the new mode via
isCommitTimePopulated(); MoR incremental relations keep the strict populate=true guard
until the MoR log-write path is updated in a follow-up.
Fail-fast at writer init:
- Unknown token in the mode list (parser throws).
- populate.meta.fields=true combined with a non-empty mode (ambiguous).
- MERGE_ON_READ + non-empty mode (log-write path not yet wired; follow-up).
- Non-Spark engine + non-empty mode (Flink RowData / Java-client not yet wired; follow-up).
Runtime immutability:
- HoodieWriterUtils.validateTableConfig now explicitly rejects a null-on-disk → non-empty
transition on the mode property (existing loop only flagged the mismatch when the on-disk
value was non-null, which let this silent-drop path slip through).
Tests:
- TestHoodieMetaFieldsMode (hudi-hadoop-common) — accessor coverage for every combination
plus unknown-token rejection and whitespace tolerance.
- TestHoodieWriteConfigMetaFieldsMode (hudi-client-common) — builder + validate() coverage.
- TestMetaFieldsMode (hudi-spark) — end-to-end write / re-read for every mode plus the
three rejection paths (populate=true+mode, unknown token, MoR+mode).
…enum Addresses review feedback on apache#19205: - Fold (populateMetaFields boolean, comma-separated mode) into a single MetaFieldsMode enum (ALL / NONE / COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME). The enum is the sole argument passed through every writer factory and writer constructor. populate.meta.fields stays on the config surface for backward compat and is auto-derived from the enum. - On-disk representation is now the enum name (e.g. "COMMIT_TIME_ONLY") instead of a comma-separated token list. Existing tables without the property continue to fall back to ALL or NONE based on the legacy hoodie.populate.meta.fields boolean. - Drop _hoodie_commit_seqno population from the selective write path — only _hoodie_commit_time is needed for incremental queries. (Review comments on HoodieRowCreateHandle:193, HoodieSparkParquetWriter:100, HoodieAvroParquetWriter:103.) - Extend the IncrementalRelation error message to state that hoodie.meta.fields.mode is physical-storage and cannot be flipped by changing write options — existing tables must be recreated to change it. - StreamSync.initializeEmptyTable() was missing the mode pass-through — wired via setMetaFieldsModeFromString. All other non-test initTable call sites either use fromProperties() (which now routes the mode through) or intentionally don't touch meta fields (CLI table create, MDT, sample writes). Test coverage additions: - TestMetaFieldsMode (hudi-spark): end-to-end assertions read parquet back and verify which meta columns are populated / null per mode. Covers row-writer path (bulk_insert with row.writer.enable=true, default) AND non-row-writer path (insert with row.writer.enable=false). Adds inline-clustering coverage for all 5 modes verifying that clustered files preserve the mode's column population semantics. - TestHoodieStreamerMetaFieldsMode (hudi-utilities): parameterized end-to-end test through the HoodieStreamer entrypoint covering all 5 modes plus MoR+selective rejection.
Previously the populateMetaFields=false branch of HoodieDatasetBulkInsertHelper projected the five Hudi meta columns as Alias(Literal(UTF8String.EMPTY_UTF8, StringType), name). A non-null Literal derives a non-null Alias, so the resulting StructField had nullable=false. Under ALL mode this was harmless because HoodieRowCreateHandle.writeRow unconditionally fills every meta slot with a non-null value. Once selective / NONE modes started leaving the opted-out columns null on the row (per hoodie.meta.fields.mode), the non-nullable StructType led SparkToParquetSchemaConverter to declare the columns as `required binary` on disk, and any subsequent read failed with ParquetDecodingException: could not read bytes at offset 0. Switch the stub to Alias(Literal.create(null, StringType), name) so the projected StructField is nullable and the physical parquet column is written as OPTIONAL. Handle-level population (writeRow / writeRowNoMetaFields / writeRowSelectiveMetaFields) is unchanged — the stub value is never observed on disk; the fix only affects the parquet schema. Fixes all row-writer + clustering selective-mode failures in TestMetaFieldsMode (commitTimeOnly / fileNameOnly / commitTimeAndFileName / none, plus clustering variants). 18/18 tests pass locally.
…elds modes TableSchemaResolver.getTableSchema() and getTableSchema(String timestamp) decided whether to include the five Hudi meta columns in the projected schema based on populateMetaFields(). Under the selective modes introduced in this PR (COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME), populateMetaFields() returns false but the opted-in meta columns are still physically present in every Parquet file. The old logic silently stripped those columns from the read schema, so downstream consumers (notably the incremental relations, whose range filter projects _hoodie_commit_time) saw no meta columns at all and produced zero-row reads. Switch the decision to "include meta fields whenever the table's mode is not NONE." That preserves the previous behaviour for the two originally supported modes (ALL -> true, NONE -> false) and correctly turns the flag on for the three new selective modes. No schema change on disk; this only changes what the read side projects.
…e populate.meta.fields Addresses review feedback on apache#19205: composing hoodie.populate.meta.fields and hoodie.meta.fields.mode to determine meta-column population was confusing, and the two-input resolution silently discarded an explicit mode whenever populate.meta.fields=true. hoodie.meta.fields.mode is now the single control. MetaFieldsMode.fromConfig (boolean, String) is replaced by resolve(String, boolean): an explicit mode is the answer outright, and the deprecated boolean is consulted only when the mode is absent, so tables written before the property existed keep resolving to ALL / NONE exactly as before. - Deprecate POPULATE_META_FIELDS (config property, write-config builder setter, and table builder setter). Full call-site migration is a follow-up; the 55 existing populateMetaFields() callers keep working because the accessor now derives from the mode via toLegacyPopulateMetaFields(). - Drop the populate=true + mode mutual-exclusion check from HoodieWriteConfig.validate() — the combination is no longer ambiguous. - Persist an explicit mode verbatim in hoodie.properties, including ALL / NONE. This also closes a hole where a selective mode supplied without an explicit populateMetaFields boolean skipped both cross-validation guards and was then silently ignored at read time. - setMetaFieldsModeFromString now routes through MetaFieldsMode.parse so bad values report the allowed set instead of a bare "No enum constant".
…lds path HoodieRowCreateHandle#writeRowSelectiveMetaFields gated _hoodie_file_name on shouldPreserveHoodieMetadata, copying the source row's value during clustering. Because clustering replaces the file it reads from, the rewritten records ended up pointing at a file that no longer serves them. The ALL-mode path (writeRow) has always written the new file name unconditionally, even when shouldPreserveHoodieMetadata is set — only _hoodie_commit_time and _hoodie_commit_seqno are preserved there. Match that behavior. _hoodie_commit_time still honors the preserve flag. Reported in review of apache#19205.
TestMetaFieldsMode.populateTrueWithSelectiveModeIsRejected asserted the mutual-exclusion check that was removed when hoodie.meta.fields.mode became the source of truth: populate.meta.fields=true together with a selective mode used to be rejected as ambiguous, and now the mode simply wins. Rewritten as selectiveModeWinsOverLegacyPopulateTrue, which asserts the new contract end to end — the table resolves to COMMIT_TIME_ONLY and only _hoodie_commit_time is populated on disk. Caught by CI on spark3.4 / spark3.5; the hudi-spark module does not test-compile locally on this branch due to an unrelated pre-existing break in TestMergeIntoHoodieTableCommand.
nsivabalan
force-pushed
the
meta_fields_mode_streamer_guard
branch
2 times, most recently
from
July 27, 2026 04:02
2f796c1 to
aa7a9c4
Compare
Addresses danny0405's P1 on apache#19205. hoodie.meta.fields.mode and hoodie.populate.meta.fields were persisted independently, so hoodie.properties could record a boolean that contradicts the mode — e.g. a table written COMMIT_TIME_ONLY while still saying populate.meta.fields=true. Pre-1.3.0 readers ignore the mode property entirely and would treat such a table as ALL. For NONE that is actively unsafe: an older incremental reader is allowed to run against all-null commit times and silently returns no rows rather than failing. Whenever an explicit mode is supplied, the legacy boolean is now written from it (ALL -> true, every other mode -> false) rather than from whatever the caller passed: - HoodieTableMetaClient.TableBuilder — the chokepoint for every table-creation path, including HoodieSparkSqlWriter and StreamSync, which pass both setters. - HoodieWriteConfig.Builder#withMetaFieldsMode — so a write config handed to table creation cannot carry a contradicting boolean either. With no mode supplied, only the legacy boolean is recorded, preserving pre-1.3.0 behavior on disk. Tests: TestHoodieTableMetaClient covers all four builder combinations (selective / NONE / ALL / legacy-only); TestMetaFieldsMode asserts the persisted boolean end to end for selective, NONE and ALL; TestHoodieWriteConfigMetaFieldsMode adds an ordering case so a later withPopulateMetaFields cannot widen a selective config.
…tion Addresses danny0405's second P1 on apache#19205. validateAgainstTableProperties compared only the legacy populateMetaFields() booleans. Those collapse every selective mode to false, so a writer resolving to NONE passed the check against a persisted COMMIT_TIME_ONLY table — both sides read false. The writer then produced files with null _hoodie_commit_time while the table still advertised COMMIT_TIME_ONLY, so incremental queries accepted the table and silently missed those rows. Meta-field population is physical, so the writer must agree with what the table records. The check now compares MetaFieldsMode directly and reports both values. Note this is a mismatch check, not a widening check: NONE against COMMIT_TIME_ONLY is a narrowing that the previous guard, and a widening-only guard, would both let through. Reachable from any direct BaseHoodieWriteClient / SparkRDDWriteClient user; the Spark datasource path was already covered by HoodieWriterUtils.validateTableConfig. Tests: TestBaseHoodieWriteClient gains a case reproducing exactly that scenario (verified to fail against the previous boolean-only guard) plus a matching-mode case covering both the default ALL writer and a selective writer.
…case-insensitive Three review comments from danny0405 on apache#19205: - HoodieSparkParquetWriter's boolean constructor had no callers at all — the only construction site (HoodieSparkFileWriterFactory) already passes a MetaFieldsMode. Removed. - HoodieAvroParquetWriter's boolean constructor is still used by SparkHelpers.scala plus four tests, so it stays, but is now marked @deprecated pointing at the MetaFieldsMode overload since it cannot express the selective modes. - MetaFieldsMode.parse now upper-cases before valueOf, so a hand-edited hoodie.properties or a write option does not have to match the enum's casing. TestHoodieMetaFieldsMode covers lower, mixed and padded forms.
Addresses danny0405's review comment on apache#19205: every caller was repeating the same two-line dance — read hoodie.meta.fields.mode, read the deprecated boolean, pass both — which duplicated the property keys and the precedence rule at each site. MetaFieldsMode.resolve(HoodieConfig) now encapsulates that. Both HoodieTableConfig and HoodieWriteConfig extend HoodieConfig, so one overload serves the write config and both writer factories. HoodieTableConfig.getMetaFieldsMode deliberately keeps the two-argument form: its fallback must read the raw property, because populateMetaFields() on that class is itself derived from the mode. Commented in place so the asymmetry is not mistaken for an oversight.
Addresses danny0405's review comment on apache#19205: the new table property had no upgrade or downgrade handling. Upgrade (9 -> 10): version 9 tables predate the property and resolve to ALL / NONE from the deprecated populate.meta.fields boolean. The upgrade now records that derived value explicitly, so an upgraded table describes its meta-field layout the same way a freshly created version 10 table does rather than relying on the legacy fallback at every read. Behavior is unchanged either way. Downgrade (10 -> 9): version 9 does not understand the property, so it is dropped — but populate.meta.fields is deliberately left exactly as it stands. ALL and NONE tables therefore round-trip unchanged, since those are precisely the two states the legacy boolean can express. Selective modes cannot be expressed in version 9; the downgrade is still allowed, and logs a warning that the table will behave as NONE to version 9 readers and that incremental queries relying on the selective mode will stop returning rows. Already-written files keep their populated meta columns either way. The mode lookup in the downgrade handler is best-effort: dropping the property is what matters, and some callers drive the change set without a helper. Tests: new TestNineToTenUpgradeHandler covers both derived values and asserts the upgrade never rewrites the legacy boolean; TestTenToNineDowngradeHandler now asserts the mode is deleted while populate.meta.fields is left untouched.
Comparing the resolved MetaFieldsMode with strict equality rejected a
combination that has always been legal: a writer that resolves to NONE
against an ALL table. Many callers build a HoodieWriteConfig without
restating the table's meta-field settings, so this broke writes across
the Spark, Java and spark-client engines.
Split the guard in two:
- Widening is always rejected. Enabling a meta column now would leave
earlier commits without it, and readers cannot tell the two apart.
- Any disagreement is rejected when the writer *explicitly* sets
hoodie.meta.fields.mode. That still catches an explicit NONE against
a COMMIT_TIME_ONLY table, which would write null commit times while
the table advertises COMMIT_TIME_ONLY and make incremental queries
silently miss those rows.
A writer that never mentions the mode is left alone: writing fewer meta
columns than the table advertises cannot make a reader believe in data
that is absent.
Adds MetaFieldsMode.isWiderThan for the asymmetric comparison, and tests
for both new branches.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…precated boolean An audit of all 103 populateMetaFields() / POPULATE_META_FIELDS references turned up sites that read the deprecated boolean directly instead of resolving hoodie.meta.fields.mode. With a selective mode set and the boolean left at its true default, these resolve the wrong way. Several are reachable today on the Spark CoW path this PR enables. Record-key reads (NPE under selective modes — the meta-column branch is taken, then the null _hoodie_record_key ordinal is dereferenced): - HoodieSparkRecord#wrapIntoHoodieRecordPayloadWithKeyGen - HoodieAvroIndexedRecord#wrapIntoHoodieRecordPayloadWithKeyGen Writer factories bypassing the mode (would stamp all five meta columns and enable the bloom filter — the opposite of the requested mode): - HoodieSparkFileWriterFactory Lance + Vortex paths - HoodieAvroFileWriterFactory HFile path (its getBoolean could also NPE on unbox when neither property is set) Schema reconstruction asking "are meta columns on disk?" with an is-record-key-populated predicate. Selective modes write the meta columns as physical nullable columns, so the correct test is mode != NONE — otherwise the rebuilt schema omits columns the file actually has: - SparkValidatorUtils (fed straight into spark.read().schema(...).parquet(...)) - PartitionStatsIndexer, SparkMetadataWriterUtils Guards bypassable by setting only the mode: - AutoRecordKeyGenerationUtils (auto-generated keys would be computed then discarded on a table that does not populate _hoodie_record_key) - DataSourceOptions ENABLE_ROW_WRITER infer fn (row writer stayed on, silently ignoring COMBINE_BEFORE_INSERT — the exact case its comment warns about) - BaseDatasetBulkInsertCommitActionExecutor (picked a partitioner that sorts on null _hoodie_record_key / _hoodie_partition_path) Bootstrap table creation dropped the mode entirely, unlike HoodieSparkSqlWriter and StreamSync which pair it with the legacy boolean: - BootstrapExecutorUtils, BootstrapExecutor Also replaces the all-or-nothing irreversibility guard in BaseHoodieWriteClient#validateAgainstTableProperties with a mode-lattice check. Both sides of the old condition collapsed to mode == ALL, so NONE -> COMMIT_TIME_ONLY and FILE_NAME_ONLY -> COMMIT_TIME_AND_FILE_NAME slipped through silently. MetaFieldsMode#isWiderThan rejects any transition that adds a populated column; narrowing is still allowed. Deliberately unchanged: the MoR incremental-relation guards, which are stricter than isCommitTimePopulated() on purpose because HoodieAppendHandle does not yet honor the mode. Relaxing them would turn a loud failure into silent data loss. The remaining ~40 references are semantically equivalent today and are left for the deprecation follow-up.
nsivabalan
force-pushed
the
meta_fields_mode_streamer_guard
branch
from
July 27, 2026 15:35
aa7a9c4 to
6b3d885
Compare
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Stacked on #19205 — please review and merge that one first.
While scoping the
hoodie.populate.meta.fieldsdeprecation follow-up requested in review of #19205, I audited all 103populateMetaFields()/POPULATE_META_FIELDSreferences insrc/main. Most are semantically identical today and only misname intent (those are the rename sweep, still to come). But a dozen sites read the deprecated boolean directly rather than resolvinghoodie.meta.fields.mode— and with a selective mode set while the boolean sits at itstruedefault, they resolve the wrong way. Several are reachable on the Spark CoW path #19205 enables.This PR fixes those.
Summary and Changelog
Record-key reads → NPE. Under a selective mode these take the meta-column branch and then dereference the null
_hoodie_record_keyordinal:HoodieSparkRecord#wrapIntoHoodieRecordPayloadWithKeyGenHoodieAvroIndexedRecord#wrapIntoHoodieRecordPayloadWithKeyGenWriter factories bypassing the mode. Would stamp all five meta columns and enable the bloom filter — the opposite of the requested mode:
HoodieSparkFileWriterFactoryLance + Vortex paths (the parquet path beside them was already migrated)HoodieAvroFileWriterFactoryHFile path — itsgetBoolean(rather thangetBooleanOrDefault) could also NPE on unbox when neither property is setSchema reconstruction asking the wrong question. These ask "are the meta columns on disk?" using an is-record-key-populated predicate. Selective modes write the meta columns as physical nullable columns, so the correct test is
mode != NONE— the predicateTableSchemaResolveralready uses. Otherwise the rebuilt schema omits columns the file actually has:SparkValidatorUtils— that schema feeds straight intospark.read().schema(...).parquet(...)PartitionStatsIndexer,SparkMetadataWriterUtilsGuards bypassable by setting only the mode:
AutoRecordKeyGenerationUtils— auto-generated keys computed, then discarded on a table that does not populate_hoodie_record_keyDataSourceOptionsENABLE_ROW_WRITERinfer fn — row writer stayed on, silently ignoringCOMBINE_BEFORE_INSERT, the exact case its own comment warns aboutBaseDatasetBulkInsertCommitActionExecutor— picked a partitioner sorting on null_hoodie_record_key/_hoodie_partition_pathBootstrap dropped the mode entirely.
BootstrapExecutorUtilsandBootstrapExecutorcalledsetPopulateMetaFields(...)without the pairedsetMetaFieldsModeFromString(...)thatHoodieSparkSqlWriterandStreamSynchave.Irreversibility guard was all-or-nothing. In
BaseHoodieWriteClient#validateAgainstTableProperties, both sides of!tableConfig.populateMetaFields() && writeConfig.populateMetaFields()collapse tomode == ALL, soNONE → COMMIT_TIME_ONLYandFILE_NAME_ONLY → COMMIT_TIME_AND_FILE_NAMEslipped through silently. Replaced withMetaFieldsMode#isWiderThan, which rejects any transition that adds a populated column. Narrowing stays allowed — later commits simply leave the column null.Deliberately not changed
MergeOnReadIncrementalRelationV1/V2) are stricter thanisCommitTimePopulated()on purpose:HoodieAppendHandledoes not yet honor the mode, so relaxing them would convert a loud failure into silent data loss.populateMetaFields()is semantically identical today. Those becomeisRecordKeyPopulated()in the rename sweep.HoodieTableMetadataUtil:1094/1138— col stats drops_hoodie_commit_timeunder selective modes even though it is on disk. Fixing it changes the MDT col-stats column set, which makes already-built partitions inconsistent with new ones; it needs an index-version gate and belongs in its own patch.Impact
MetaFieldsMode#isWiderThan.Risk Level
low
Each change narrows a predicate that was resolving incorrectly for selective modes. Default (
ALL) and legacy (NONE) behavior is unchanged, sinceresolve()returns the same value for both when no mode property is set.Documentation Update
None needed — no new user-facing config.
Contributor's checklist