test(trino): add a Trino E2E testcontainers pipeline for the RFC-105 connector - #19217
Draft
voonhous wants to merge 15 commits into
Draft
test(trino): add a Trino E2E testcontainers pipeline for the RFC-105 connector#19217voonhous wants to merge 15 commits into
voonhous wants to merge 15 commits into
Conversation
voonhous
marked this pull request as draft
July 7, 2026 11:35
voonhous
force-pushed
the
trino-e2e-testcontainers
branch
2 times, most recently
from
July 7, 2026 11:56
8eac2c2 to
6b4b617
Compare
…-105)
Squash of the 55-commit review history. The connector implementation moves
into the Hudi repo as the hudi-trino module (published as
org.apache.hudi:hudi-trino), targeting Trino SPI 481 on JDK 25, behind the
opt-in hudi-trino Maven profile. The Trino repo keeps only a thin trino-hudi
shim depending on the published artifact. Removes hudi-trino-bundle and the
trino{base,coordinator,worker} docker modules.
Co-authored-by: vamsikarnika <vamsikarnika@gmail.com>
Co-authored-by: Aditya Goenka <ad1happy2go@gmail.com>
… parquet reader MOR tables written at table version 10 store delta-log blocks as native parquet files (RFC-103). Reading one calls HoodieIOFactory.getFileFormatUtils(PARQUET).readFooter(...) to pull the log-block header out of the parquet footer, but HudiTrinoIOFactory stubbed getFileFormatUtils out, so any realtime read of a v10 MOR table failed with "FileFormatUtils not supported in HudiTrinoIOFactory" (surfaced by TestHudiCustomMerger and TestHudiCustomMergerEndToEnd, which write their tables at the default version). Add HudiTrinoParquetFileFormatUtils, implementing readFooter with Trino's own parquet reader (MetadataReader.readFooter -> FileMetadata.getKeyValueMetaData), mirroring ParquetUtils.readFooter semantics, and return it from HudiTrinoIOFactory.getFileFormatUtils for PARQUET; non-parquet native log formats now throw a clear message. Record decoding already flows through Trino's page-source reader, so the footer read was the only missing piece. No new dependency is added (hudi-hadoop-common stays excluded).
voonhous
force-pushed
the
trino-e2e-testcontainers
branch
3 times, most recently
from
July 10, 2026 14:23
b6ab7f3 to
3a85205
Compare
The reader context ignored its path/schema arguments and re-served the pre-built base page source, so RFC-103 native (parquet) log records were never read and v10 MOR reads returned base-only values. Build a Trino parquet page source on demand for native log files, projected on the file-group reader's requiredSchema with predicate pushdown off, via a factory injected from HudiPageSourceProvider. Synthesize VARCHAR handles for Hudi meta columns the file-group reader adds to requiredSchema beyond the connector projection (e.g. _hoodie_commit_time); all meta columns are UTF8 strings on disk and parquet columns are resolved by name, so the placeholder ordinal is never read (refactor tracked by apache#19249). Custom mergers must be projection compatible: a non-projection-compatible merger makes the file-group reader demand the full table schema, whose unprojected data columns cannot be typed at this layer; fail such reads with a clear NOT_SUPPORTED error and declare isProjectionCompatible() on the test mergers, which merge on their declared mandatory fields only. Also adapt the TestHudiSmokeTest footer-read call site to the parameterized createPageSource helper.
voonhous
force-pushed
the
trino-e2e-testcontainers
branch
5 times, most recently
from
July 15, 2026 09:19
88dfe79 to
0a59401
Compare
voonhous
changed the title
voonhous
changed the title
…K 25 compile Master commit b0a2563 declares Lombok in the root pom's annotationProcessorPaths (a parallel-build deflake). hudi-trino inherits that path but compiles on JDK 25, where Lombok 1.18.36 crashes javac with ExceptionInInitializerError: com.sun.tools.javac.code.TypeTag :: UNKNOWN -- this broke the hudi-trino CI on every PR merge ref once that commit landed. hudi-trino uses no annotation processors, so set proc=none and override the inherited processor path away.
Fixes apache#19249, including the scope update in issue comment 4937143911; relates to the base read path of apache#18837. The file-group reader can require merge columns the query projection does not carry: Hudi meta fields like _hoodie_commit_time, delete markers, record-key data columns, a custom merger's mandatory fields, or the whole table schema when a CUSTOM merger is not projection compatible. buildRequiredColumnHandles previously hand-synthesized a HIVE_STRING/VARCHAR handle behind a HOODIE_META_COLUMNS guard and threw NOT_SUPPORTED for anything else, and the base-file read silently returned projection-only records that made such mergers see nulls. Log side: - Every required-schema field carries its real Avro type from the table schema, so unresolved handles are typed directly from the field (NativeLogicalTypesAvroTypeBlockHandler + HiveTypeTranslator) via the new HudiUtil.toColumnHandle; the synthesis, guard and exception are gone. trino-hive-formats moves from runtime to compile scope. Base side: - HudiPageSourceProvider mirrors the file-group reader's decision: HudiUtil.resolveMergeModeAndStrategyId applies the same version-gated inference as hudi-common (merge mode below table version 9, strategy id below version 8), and when the resolved CUSTOM merger is not projection compatible (interface default; also every payload-based table, since the payload strategy resolves HoodieAvroRecordMerger), the read projection expands to every table-schema column via HudiUtil.appendMissingSchemaColumns. Matches Spark; the extra I/O is inherent to such mergers. - getMergeRequiredColumnHandles predicts every column FileGroupReaderSchemaHandler.getMandatoryFieldsForMerging can demand: ordering columns under the inferred (not raw) merge mode, _hoodie_is_deleted and _hoodie_operation unconditionally (dropped when not table columns), the custom delete key when both hoodie.payload.delete.field and .marker are set, and the record-key data columns when hoodie.populate.meta.fields is false. - The base read reuses the pre-built page source, so its projection MUST cover requiredSchema: HudiTrinoReaderContext now guards that invariant and fails with HUDI_SCHEMA_ERROR naming any missing columns instead of merging with nulls. MOR splits with log files but no base file fail with a clear NOT_SUPPORTED instead of a bare NoSuchElementException. Cleanups in the same area: - Replace SynthesizedColumnHandler with PrefilledColumnValues, a thin per-split adapter over trino-hive's HiveUtil.getPrefilledColumnValue. Fixes $partition rendering multi-key partitions in hash order, handles the hive null marker (backslash-N) in partition values, adds previously unsupported partition-key types, and deletes SynthesizedColumnStrategy plus the hand-rolled appendPartitionKey dispatch. - Remove dead code (HudiTrinoReaderContext.colToPosMap and dataHandles, HudiPageSource.readerContext); make fields private final. - remapColumnIndicesToPhysical throws a diagnosable TrinoException instead of an autoboxing NPE for columns absent from the parquet file schema, and lowercases with Locale.ROOT. Tests: - TestHudiNonProjectionCompatibleMerger + NonProjectionCompatibleRankMerger (no isProjectionCompatible/getMandatoryFieldsForMerging overrides) + initializer: narrow-projection MOR queries that never project merge_rank, laid out so one key's winning rank is on the log record and the other's on the base record; sum(value)=199 discriminates the correct merge from base-only (110) and newest-wins (103). - Unit coverage: toColumnHandle, appendMissingSchemaColumns, usesNonProjectionCompatibleMerger, resolveMergeModeAndStrategyId (table versions 6/8/9), merge-required column prediction, PrefilledColumnValues (incl. the $partition ordering fix), and the typed missing-column remap error (both remap tests). - Existing projection-compatible merger tests stay as fast-path regression coverage; stale MaxRankRecordMerger javadoc refreshed.
…nd commit-time ordering Fixes apache#18898 (follow-up from the RFC-105 review of HudiTrinoReaderContext.getRecordMerger): the merge-mode dispatch had no end-to-end coverage where the merger choice matters. Adds runtime-written v10 MOR tables and snapshot-read tests for: - Deletes under EVENT_TIME_ORDERING (mor_deletes): hard deletes -- which at v10 are native delete log files read back through the connector's own getFileRecordIterator with the synthetic delete-log schema, a previously untested path enabled by the required-column resolution this branch stacks on -- plus _hoodie_is_deleted soft deletes, an OBSOLETE soft delete and an OBSOLETE update (lower ordering value) that must LOSE against the base row. - COMMIT_TIME_ORDERING (mor_commit_time): a lower-ordering update that must WIN (latest write), the exact mirror of the event-time case, discriminating OverwriteWithLatestMerger from event-time merging. - Payload-driven semantics without any hudi.record-merger-impls property: AWSDmsAvroPayload (v10-translated to COMMIT_TIME + delete-key props; Op='D' log records delete rows at merge time), OverwriteNonDefaultsWithLatestAvroPayload (v10-translated to PARTIAL_UPDATE_MODE=IGNORE_DEFAULTS; null update columns keep stored values), and a user-defined SummingTestPayload riding the payload-based CUSTOM strategy whose merged value (old + new) proves the payload's combineAndGetUpdateValue executed at read time. Test rows are written with the pass-through HoodieAvroPayload (not a BaseAvroPayload) so delete-flagged rows land as data records and every merge decision happens at read time from the table config. Bugs found by these tests and fixed here: - HudiUtil.mergeRequiredColumnNames read the custom delete key/marker from raw table props, but v9+ table creation persists them PREFIXED (hoodie.record.merge.property.*). The file-group reader strips the prefix via getTableMergeProperties before DeleteContext reads the plain keys, so its required schema included the delete column while the connector's base-read prediction missed it and the base-read projection guard failed every narrow projection on such tables. Now resolved the same way the reader does: table merge properties first, raw props as fallback. Unit-tested in TestHudiMergeRequiredColumns. - buildColumnHandles matched predicted merge columns against metastore data columns case-sensitively; the prediction carries the table schema's casing (AWSDms hardcodes 'Op') while metastore names are lowercased, so the handle was never built. Matching is now case-insensitive. - buildRequiredColumnHandles emitted projection handles with the lowercased metastore name on the log projection, but hudi-common reads merge fields off log records by the table schema's exact field name (Avro lookups are case-sensitive), so e.g. the DMS delete marker was never seen on log records. Handles are now re-labeled with the required-schema field's exact name; parquet columns resolve case-insensitively either way. - count(*) over a MoR split with log files failed with "Failed to construct Avro schema": the query projects no columns, and HudiPageSource's serializer fed the empty projection to AvroHiveFileUtils.determineSchemaOrThrowException, which rejects an empty column list. HudiUtil.constructSchema now short-circuits an empty projection to an empty record schema; the page source already counts positions fine with zero channels. HudiUtilTest's empty-columns test now pins the empty-record schema instead of the former exception. - Payload-based merging read garbage on the summing table ([, -44] instead of [k1, 109]): records handed to hudi-common carried a schema reconstructed from Hive metastore types (every column a nullable union, fields in projection order), while the file-group reader tracks its required schema for those records. BaseAvroPayload round-trips the newer record through Avro binary with the tracked schema, so the structural mismatch misaligned the binary decode. HudiTrinoReaderContext now builds the log-side AND base-side serializers with requiredSchema itself, via a new HudiAvroSerializer constructor that takes the record schema explicitly and maps page channels to record fields BY NAME (the base projection's order can differ from the required schema's field order; containment holds in both directions through the existing base-read guard and generateRequiredSchema always containing the requested schema). This also aligns record layouts with the positions hudi-common derives from the tracked schema (partial-update merging, _hoodie_operation lookup). - (hudi-common, engine-agnostic) The file-group reader passed its raw input props to initRecordMerger, the MERGE_TYPE lookup and the schema handler instead of the ConfigUtils.getMergeProps view, which layers the table's prefixed merge properties (hoodie.record.merge.property.*) in as plain keys -- a regression from commit 19961e7 replacing the previous in-place putAll with a copy. DeleteContext therefore missed the translated DMS delete key/marker, the delete column was not a mandatory merge field, and Op='D' log records merged as regular data updates instead of deletes on snapshot reads. Fixed in HoodieFileGroupReader and HoodieLsmFileGroupReader alike by passing this.props downstream. Also re-ports a shim-lineage guard (trino repo commit 78209f83b50) that the RFC-105 migration squash predates: HudiAvroSerializer's default constructor now maps channels past hidden (synthesized, split-prefilled) columns instead of relying on serialize() never being called with them; with identity positions a hidden column would shift every later value into the wrong field and overrun the record. Also replaces the TODO(apache#18898) in getRecordMerger with a pointer to the new suites.
…ning Fixes apache#19279: with the connector default hudi.metadata-enabled=true, any query on a table whose metadata table (MDT) has uncompacted delta commits failed with 'Native HFILE log files are not supported by the Hudi Trino connector'. Since MDT compaction fires only every hoodie.metadata.compact.max.delta.commits (default 10) deltas, actively written tables spend most of their life in that state; the E2E testcontainers pipeline had to set hudi.metadata-enabled=false to pass. - HudiTrinoIOFactory.getFileFormatUtils now returns hudi-common's HFileUtils for HFILE. The native-log read path (HoodieNativeLogFileReader) needs only its readFooter, and HFileUtils is hadoop-free (it decodes via hudi-io's native HFile reader), so it is safe in the Trino runtime -- unlike parquet, whose ParquetUtils lives in the hadoop-excluded hudi-hadoop-common and needs the Trino-native HudiTrinoParquetFileFormatUtils. The reader-factory side (newHFileFileReader) was already implemented; it is how compacted MDT base HFiles are read today. MDT reads never touch HudiTrinoReaderContext. - Guard the partition-stats pruning call in HudiBackgroundSplitLoader.getPartitionInfos: any pruning failure now logs and proceeds unpruned, mirroring the metastore fallback the MDT partition listing directly above it already has. Index pruning is an optimization and must never fail the query. The per-file-slice indexes (column-stats, record-level, secondary) already swallow failures via their future+catch pattern. - Guard the MDT-backed file-system-view load in HudiSnapshotDirectoryLister: on failure, close the partial view and fall back to a direct-listing view built over an MDT-disabled table metadata. This was the second unguarded synchronous MDT read. Tests: UncompactedMetadataHudiTablesInitializer writes a partitioned COW table with an ENABLED metadata table (column-stats + partition-stats indexes) and compact.max.delta.commits=100 -- the inverse of the zip fixtures' compact.max.delta.commits=1, whose always-compacted MDTs are why nothing caught this -- leaving native HFILE log deltas in the files/column_stats/ partition_stats MDT partitions. MDT HFILE writing is native (hudi-io HFileWriterImpl); the 'requires hbase' note in older initializers is stale. TestHudiUncompactedMetadataTable covers MDT-backed listing, MDT-on vs MDT-off result equality, partition-stats pruning (the exact crash path) and column-stats file skipping over the uncompacted stats. The E2E follow-up (separate branch trino-e2e-testcontainers): drop the hudi.metadata-enabled=false workaround from the Trino catalog so the pipeline runs MDT-on.
Revives the testcontainers Trino harness from the stale branch, adapted:
- TrinoService + ITTestTrino{Smoke,StockTicks,CustomType}, gated on a new
docker-compose 'trino' profile (-Dcompose.profiles=trino) so existing
hive-sync CI rows are unaffected
- docker/trino: Dockerfile (FROM trinodb/trino:481, bakes the plugin dir
built from the Trino repo's thin trino-hudi shim + etc configs),
overlay-aware entrypoint (jar-bearing TRINO_PLUGIN_DIR mount overrides the
baked plugin for fast iteration), build_image.sh
- trinocoordinator service (profile-gated) in the spark402 compose pair
The harness from the previous commit could not actually run: nothing in the repo could assemble the Trino plugin directory (the RFC-105 shim is not yet released upstream), the stock-ticks seed fixture was never committed, and no workflow activated the trino compose profile. - docker/trino/shim: standalone maven project mirroring the upstream trinodb/trino plugin/trino-hudi shim (parent io.trino:trino-root:481 from Maven Central, packaging trino-plugin, depends on org.apache.hudi:hudi-trino). Carries its own thin HudiPlugin class because trino-maven-plugin's service descriptor generator only scans the module's own classes; never deployed (maven.deploy.skip). Build with package, never install. - docker/trino/Dockerfile + overlay-entrypoint.sh: preserve the stock image's plugin/hudi/hdfs loader jars at /opt/hudi-hdfs-lib and re-attach them when the baked or overlaid plugin dir lacks them (shim-assembled dirs always do; io.trino:trino-hdfs:zip is not on Maven Central, it only ships in the server tarball / base image). - docker/demo/sparksql-stock-ticks-trino.commands: the COW + MOR seed fixture ITTestTrinoStockTicks references, jdbc-mode hive sync like the existing custom-type fixtures. - .github/workflows/hudi_trino_e2e.yml: JDK 17 reactor build, JDK 25 connector + shim build, local image build via build_image.sh, then ITTestTrino* against the spark402 compose stack with -Dcompose.profiles=trino. No trinodb/trino checkout and no DockerHub publish anywhere in the flow. - docker/trino/.gitignore: anchor plugin/ to /plugin/ so it does not swallow the shim's io/trino/plugin source package. - READMEs: local E2E flow + overlay fast loop; drop stale hudi-trino-plugin path comments.
workflow_dispatch cannot target a workflow that does not exist on the default branch yet. Drop this commit before merging upstream.
Trino 481's SpiDependencyChecker rejects provided scope for org.locationtech.jts:jts-core: it is not part of the 481 SPI surface (trino-spi 481 depends only on jackson-annotations, slice, and the opentelemetry api/api-incubator/common/context set). jts-core now flows into the assembled plugin dir at its transitive scope. The upstream 482-SNAPSHOT shim declares it provided because the SPI gains jts there.
All three ITTestTrino* classes failed with 'execInContainer can only be used while the Container is running': the coordinator dies at startup and nothing captured why (compose teardown discards the container). - ITTestBaseTestcontainers: stream trinocoordinator logs into the test output via Slf4jLogConsumer when the trino profile is active, so a startup crash leaves its root cause in the failsafe report. - TrinoService.waitUntilReady: fail fast with a pointed message when the container is no longer running instead of exhausting 18 retries against a dead container (saves ~3 min per class). - hudi_trino_e2e.yml: assert the assembled plugin dir is populated and carries the services jar; add a standalone smoke-boot of the built image (docker run + SELECT 1 via the CLI, boot log dumped on failure) so image-level boot failures surface ~30 min before the IT step.
The standalone smoke-boot caught the coordinator crash root cause: 'Unrecognized VM option GCLockerRetryAllocationCount=32'. The flag was carried over from Hudi's JDK 11/17 CI OOM workaround, but the GCLocker was removed in modern JDKs, so the JDK 25 JVM inside trinodb/trino:481 refuses to start and the container dies before any Trino code runs -- which is exactly the 'container is not running' failure all three ITTestTrino* classes hit.
The E2E run reached real query execution (ITTestTrinoSmoke fully green, the stock-ticks fixture seeds and syncs correctly) but every table read fails with 'Failed to generate splits for default.<table>' and only the one-line message survives into the report. - TrinoService: run the CLI with --debug so query failures carry the full server-side stack into the assertion message. - hudi_trino_e2e.yml: on failure, print the failsafe *-output.txt tails (where surefire redirects the streamed trinocoordinator logs).
The release licensing check (scripts/release/validate_source_copyright.sh) requires every non-excluded file to contain the ASF header, and an empty .gitkeep cannot. The file only keeps docker/trino/empty-overlay in git as the compose default plugin-overlay mount source; its content is ignored at runtime (the overlay entrypoint only applies an overlay that contains jars), so the header plus a purpose note satisfies the check without touching the shared release script.
voonhous
force-pushed
the
trino-e2e-testcontainers
branch
from
July 15, 2026 16:49
bc8840f to
22bd120
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #19298 (-> #19295 -> #19288 -> #18837) -- only the E2E harness/pipeline commits are new here. Merges last: the pipeline needs the connector fixes below it to pass.
Describe the issue this Pull Request addresses
The RFC-105 connector (#18837) had no end-to-end coverage against a real Trino server. This adds a testcontainers E2E pipeline: the full demo stack (HDFS/Hive/Spark) plus a Trino 481 coordinator running the connector built at HEAD, with tables seeded by Spark using DEFAULT configs and queried through the Trino CLI.
Seeding with defaults leaves the metadata table with uncompacted HFILE deltas, so every query here exercises MDT-on reads -- this pipeline is what surfaced #19279, and it now doubles as that fix's acceptance test (the catalog runs with
hudi.metadata-enabledat its defaulttrue).Summary and Changelog
TrinoService+ three test classes (~30 tests) --ITTestTrinoSmoke(plugin/metastore/CLI liveness),ITTestTrinoStockTicks(the classic COW + MOR_rodemo queries),ITTestTrinoCustomType(BLOB inline and out-of-line shapes, VECTOR, VARIANT on Spark 4.x composes).docker/trino: Trino 481 image with the plugin assembled from the connector at HEAD via the in-repo shim, catalog/JVM configs; compose files gain a profile-gatedtrinocoordinator..github/workflows/hudi_trino_e2e.yml: builds Hudi artifacts (JDK 17) and the connector (JDK 25), bakes and smoke-boots the image, then runsITTestTrino*under thetrinocompose profile.pull_requesttrigger stays).Impact
Test/CI only -- no connector or runtime changes.
Risk Level
none
Documentation Update
none
Contributor's checklist