feat(trino): resolve merge-required columns from the table schema

[voonhous]

Follow-up to the RFC-105 migration (#18837, merged); rebased on master.

Describe the issue this Pull Request addresses

Closes #19249

The file-group reader can require columns the query projection does not carry: meta columns like _hoodie_commit_time, delete markers, or the full table schema when a CUSTOM merger is not projection compatible. The connector hand-built a VARCHAR handle for the meta-column case and threw NOT_SUPPORTED for everything else, so non-projection-compatible mergers could not be used at all.

Summary and Changelog

• Merge-required columns are now typed from the table schema's avro types (HudiUtil.toColumnHandle) instead of being synthesized as VARCHAR. The HOODIE_META_COLUMNS guard and the NOT_SUPPORTED error are gone.
• When the resolved merger is not projection compatible, the read projection expands to the full table schema so the base and log reads both carry every merge column. The decision mirrors FileGroupReaderSchemaHandler.generateRequiredSchema, including the pre-v9 merge-config inference (a 0.x table with a custom payload class infers CUSTOM mode with no persisted strategy id).
• The base read also requests the remaining merge-mandatory columns (_hoodie_is_deleted, _hoodie_operation, the custom delete key, record-key fields when hoodie.populate.meta.fields=false), and the reader context now fails loudly if the base projection ever misses a column the file-group reader demands, instead of merging with nulls.
• SynthesizedColumnHandler is replaced by PrefilledColumnValues, a thin adapter over trino-hive's getPrefilledColumnValue. This intentionally goes past the issue's "no behavioral change" acceptance: $partition renders keys in partition order instead of hash order, the \N null marker is handled, more partition-key types work, and $file_modified_time renders in the JVM default zone as the Hive connector does instead of the previous pinned UTC (same instant, different rendered zone).
• MOR splits with log files but no base file fail with a clear NOT_SUPPORTED error instead of a bare NoSuchElementException.
• Under hudi.parquet.use-column-names=false, a projected column absent from a base file (schema-on-write evolution never rewrites old files) now null-fills exactly like the name-based path, instead of failing hard (previously an autoboxing NPE). Column renames under full schema evolution (hoodie.schema.on.read.enable) stay out of scope on both paths: [FEATURE] hudi-trino: resolve parquet columns through the internal schema for full schema evolution (hoodie.schema.on.read.enable) #19381.
• A table that resolves to CUSTOM merge mode without a persisted hoodie.record.merge.strategy.id (possible at table version 8 or later) fails with an actionable HUDI_BAD_DATA error instead of an NPE inside hudi-common's merger resolution.
• Merge-required columns the metastore does not carry but the Avro table schema does (e.g. hive sync with omit_metadata_fields=true) are recovered on the merge read path from the already-resolved table schema -- including columns a CUSTOM merger declares mandatory, by asking the same resolved merger -- at no extra I/O (pinned by the file-operation-count tests).

New tests: TestHudiNonProjectionCompatibleMerger runs narrow-projection MOR queries that only merge correctly if both the base and log side supply the never-projected merge_rank column (one key wins on each side); unit coverage for the new helpers and for the pre-v9 inference at table versions 6, 8 and 9. Review rounds added: a payload-class-only table version 6 fixture read end to end, fixtures whose metastore omits a merge-required column (the ordering field, and a merger-declared column) pinning the schema recovery end to end, and a shared initializer base (AbstractMergerHudiTablesInitializer) replacing three near-identical fixture copies.

Impact

Custom mergers no longer have to override isProjectionCompatible() / getMandatoryFieldsForMerging(). Note that payload-class tables resolve HoodieAvroRecordMerger, which is not projection compatible, so such tables read the full table schema on MOR splits with log files -- same as Spark, and inherent to payload-based merging. Projection-compatible and base-file-only reads are unchanged.

Risk Level

low -- confined to hudi-trino; the projection-compatible fast path keeps its existing test coverage.

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[voonhous]

@wombatu-kun Can you please help to review this? I have a few more PRs to get the Trino integrations landed in a good shape.

I'd like to get this track of work done by this week.

Thank you!

[hudi-bot]

CI report:

• b98144b Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[voonhous]

@wombatu-kun Can i request for another round of review please? For the comments addressed and if you have no further comments, can you please help to "resolve" them so they are collapsed?

[wombatu-kun]

Verified the CUSTOM-branch guards are pinned: without the isNullOrEmpty half the null-id test does reach createValidRecordMerger's leading equals(), and the strategy id the unresolvable-merger test uses is neither the payload-based uuid nor one any configured merger declares, so neither test is vacuous. No further comments from me - all my threads are resolved.