feat(flink): enable LSM reader for Flink metadata table

[cshuo]

Describe the issue this Pull Request addresses

Flink metadata tables did not use the LSM-tree storage layout or HoodieLsmFileGroupReader, missing the sorted-run merge benefits available for native log file slices.

This change enables LSM storage and reads for newly created Flink v10+ metadata tables while preserving classic reader behavior for legacy/mixed log slices and reusable-buffer workloads.

Fixes #19372.

Summary and Changelog

• Configure newly created Flink v10+ metadata tables with the lsm_tree storage layout.
• Use HoodieLsmFileGroupReader for LSM file slices containing only native logs.
• Retain HoodieFileGroupReader for default layouts, mixed/legacy log slices, and reusable log-record buffer scenarios.
• Verify Flink metadata table layout, native log creation, and consistent FILES reads with reuse enabled and disabled.
• Use the current writer version when testing compaction after table-version upgrade.

Impact

• Functional impact: New Flink v10+ metadata tables use LSM storage and sorted-merge reads where applicable. Existing layouts and classic reusable-reader behavior remain supported.
• Maintainability: Reader selection is centralized in HoodieBackedTableMetadata.
• Extensibility: The layout-based selection allows future LSM reader improvements without changing public APIs.

Risk Level

Medium. This changes the storage layout and read path for new Flink v10+ metadata tables. The risk is mitigated by fallback to the classic reader for legacy, mixed-log, and reuse scenarios, with tests covering layout initialization, native logs, metadata reads, and compaction upgrade compatibility.

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-agent]

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for working on this! This PR enables the LSM storage layout and HoodieLsmFileGroupReader for newly created Flink v10+ metadata tables, falling back to the classic reader for reuse/mixed/legacy slices. One edge case around instant-range filtering on the LSM read path is worth a close look in the inline comment before this lands. Please take a look at any inline comments, and this should be ready for a Hudi committer or PMC member to take it from here. One minor duplication nit in the reader branch; everything else is clean.

[hudi-bot]

CI report:

• 32f0844 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[hudi-agent]

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for working on this! The PR enables the LSM storage layout and HoodieLsmFileGroupReader for new Flink v10+ metadata tables, adds instant-range filtering of native log files, and centralizes reader selection in HoodieBackedTableMetadata. One correctness concern worth double-checking is flagged inline around how the LSM reader honors the key/prefix filter for native log records on point/prefix-lookup partitions. Please take a look at the inline comment, and this should be ready for a Hudi committer or PMC member to take it from here.