test(trino): add a Trino E2E testcontainers pipeline for the RFC-105 connector #19217
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| name: Hudi Trino E2E | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - master | ||
| - 'release-*' | ||
| # TEMPORARY: validate the workflow on the dev branch; drop this line | ||
| # (and the commit that added it) before merging upstream. | ||
| - trino-e2e-testcontainers | ||
| paths: | ||
| - 'hudi-trino/**' | ||
| - 'docker/trino/**' | ||
| - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*' | ||
| - 'docker/demo/sparksql-stock-ticks-trino.commands' | ||
| - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**' | ||
| - '.github/workflows/hudi_trino_e2e.yml' | ||
| pull_request: | ||
| branches: | ||
| - master | ||
| - 'release-*' | ||
| paths: | ||
| - 'hudi-trino/**' | ||
| - 'docker/trino/**' | ||
| - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*' | ||
| - 'docker/demo/sparksql-stock-ticks-trino.commands' | ||
|
Contributor
There was a problem hiding this comment.
|
||
| - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**' | ||
| - '.github/workflows/hudi_trino_e2e.yml' | ||
| workflow_dispatch: | ||
|
|
||
| concurrency: | ||
| group: hudi-trino-e2e-${{ github.ref }} | ||
| cancel-in-progress: ${{ !contains(github.ref, 'master') && !contains(github.ref, 'release-') }} | ||
|
|
||
| env: | ||
| MVN_ARGS: -e -ntp -B -V -Dgpg.skip -Djacoco.skip -Pwarn-log | ||
|
Contributor
There was a problem hiding this comment. This drops the |
||
| SCALA_PROFILE: -Dscala-2.13 -Dscala.binary.version=2.13 | ||
| COMPOSE_PREFIX: docker-compose_hadoop340_hive2310_spark402 | ||
|
|
||
| jobs: | ||
| trino-e2e: | ||
| # Testcontainers E2E for the RFC-105 native trino-hudi connector: builds | ||
| # hudi-trino at HEAD, assembles the plugin dir via the in-repo shim | ||
| # (docker/trino/shim, standing in for the not-yet-released upstream | ||
| # trinodb/trino plugin/trino-hudi shim), bakes it into a local | ||
| # apachehudi/hudi-trino_481 image, and runs ITTestTrino* against the | ||
| # spark402 compose stack (the only pair with the trinocoordinator service). | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v5 | ||
| - name: Free disk space | ||
| run: | | ||
| sudo rm -rf /usr/share/dotnet | ||
| sudo rm -rf /usr/local/lib/android | ||
| sudo rm -rf /opt/ghc | ||
| sudo rm -rf /usr/local/share/boost | ||
| docker system prune --all --force --volumes | ||
| - name: Pre-pull compose images (fails fast if not published) | ||
| run: | | ||
| # Surface a missing sparkadhoc image before the long Maven install. The | ||
| # remaining stack images are pulled by docker-compose at test time; the | ||
| # trino image is built locally below, never pulled. | ||
| docker pull apachehudi/hudi-hadoop_3.4.0-hive_2.3.10-sparkadhoc_4.0.2:latest | ||
| - name: Set up JDK 17 | ||
| uses: actions/setup-java@v5 | ||
| with: | ||
| java-version: '17' | ||
| distribution: 'temurin' | ||
| architecture: x64 | ||
| cache: maven | ||
| - name: Build and install Hudi artifacts (JDK 17) | ||
| # Full reactor: the compose containers mount the workspace and the tests | ||
| # use bundles staged by the -Pintegration-tests build (e.g. | ||
| # docker/hoodie/hadoop/hive_base/target/hoodie-spark-bundle.jar). | ||
| run: | ||
| mvn clean install -T 2 $SCALA_PROFILE -Dspark4.0 -Dflink1.20 -Pintegration-tests -DskipTests=true -Ddocker.compose.skip=true $MVN_ARGS | ||
| - name: Set up JDK 25 | ||
| uses: actions/setup-java@v5 | ||
| with: | ||
| java-version: '25' | ||
| distribution: 'temurin' | ||
| cache: maven | ||
| - name: Build hudi-trino connector (JDK 25) | ||
| # No trinodb/trino checkout needed: the unpublished Trino test-jars sit | ||
| # behind the off-by-default hudi-trino-tests profile and packaging | ||
| # resolves entirely from Maven Central. | ||
| run: | ||
| mvn $MVN_ARGS -Phudi-trino -pl hudi-trino install -Dmaven.test.skip=true | ||
| - name: Assemble trino-hudi plugin dir via in-repo shim (JDK 25) | ||
| # package, NOT install: installing would shadow the real | ||
| # io.trino:trino-hudi release coordinates in the local m2. | ||
| run: | ||
| mvn $MVN_ARGS -f docker/trino/shim/pom.xml clean package -DskipTests | ||
| - name: Build apachehudi/hudi-trino_481 image | ||
| run: | | ||
| docker/trino/build_image.sh --plugin-dir docker/trino/shim/target/trino-hudi-481 | ||
| # Sanity: the shim must have produced a populated plugin dir with a | ||
| # service descriptor jar, or Trino cannot load the plugin at boot. | ||
| echo "plugin dir jar count: $(ls docker/trino/shim/target/trino-hudi-481 | wc -l)" | ||
| ls docker/trino/shim/target/trino-hudi-481/*services*.jar | ||
| - name: Smoke-boot the Trino image standalone | ||
| # Catches image-level boot failures (plugin load errors, bad etc/ config) | ||
| # ~30 min before the IT step would, with the full boot log on screen. | ||
| # --hostname trinocoordinator makes the baked discovery.uri self-resolve. | ||
| run: | | ||
| docker run -d --name trino-smoke --hostname trinocoordinator \ | ||
| apachehudi/hudi-trino_481:latest | ||
| ok="" | ||
| for i in $(seq 1 18); do | ||
| if [ "$(docker inspect -f '{{.State.Running}}' trino-smoke)" != "true" ]; then | ||
| echo "trino-smoke container died during startup" >&2 | ||
| break | ||
| fi | ||
| if docker exec trino-smoke trino --server localhost:8080 \ | ||
| --execute "SELECT 1" >/dev/null 2>&1; then | ||
| ok=1; echo "Trino answered SELECT 1 (attempt $i)"; break | ||
| fi | ||
| sleep 10 | ||
| done | ||
| if [ -z "$ok" ]; then | ||
| echo "==== trino-smoke boot log ====" | ||
| docker logs trino-smoke 2>&1 | tail -200 | ||
| docker rm -f trino-smoke >/dev/null 2>&1 || true | ||
| exit 1 | ||
| fi | ||
| docker rm -f trino-smoke | ||
| - name: Set up JDK 17 (restore for the IT run) | ||
| # setup-java resets JAVA_HOME on each call; hudi-integ-test needs 17. | ||
| uses: actions/setup-java@v5 | ||
| with: | ||
| java-version: '17' | ||
| distribution: 'temurin' | ||
| architecture: x64 | ||
| - name: Run Trino E2E ITs (JDK 17) | ||
| run: | | ||
| # -DskipITs=false overrides the spark4.0 profile's skipITs=true default | ||
| # (see root pom.xml). -Dcompose.profiles=trino starts the profile-gated | ||
| # trinocoordinator service; without it every ITTestTrino* class skips. | ||
| mvn verify $SCALA_PROFILE -Dspark4.0 -Pintegration-tests \ | ||
| -pl hudi-integ-test \ | ||
| -DskipITs=false \ | ||
| -Ddocker.compose.skip=true \ | ||
| -Dit.test='ITTestTrino*' \ | ||
| -Dcompose.profiles=trino \ | ||
| -Dspark.docker.compose.prefix=$COMPOSE_PREFIX \ | ||
| $MVN_ARGS | ||
| - name: Dump failsafe test outputs on failure | ||
| # Surefire redirects test stdout (incl. the streamed trinocoordinator | ||
| # boot/query logs) into per-class output files; print their tails so | ||
| # server-side failures are readable straight from the workflow log. | ||
| if: failure() | ||
| run: | | ||
| for f in hudi-integ-test/target/failsafe-reports/*-output.txt; do | ||
|
Contributor
There was a problem hiding this comment.
|
||
| [ -f "$f" ] || continue | ||
| echo "===== $f (last 400 lines) =====" | ||
| tail -n 400 "$f" | ||
| done | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| // Self-contained COW + MOR stock-ticks seed for the Trino E2E tests | ||
| // (ITTestTrinoStockTicks). Mirrors the docker/demo/trino-batch1.commands data | ||
| // shape without the Kafka/streaming pipeline, which integ2 does not exercise. | ||
| // ts stays STRING so Trino's CSV_UNQUOTED output matches the test's exact | ||
| // row assertion: GOOG,2018-08-31 10:29:00,6330,1230.5,1230.5 | ||
| spark.sql(""" | ||
| CREATE TABLE stock_ticks_cow ( | ||
| symbol STRING, | ||
| ts STRING, | ||
| volume LONG, | ||
| open DOUBLE, | ||
| close DOUBLE, | ||
| dt STRING | ||
| ) USING hudi | ||
| PARTITIONED BY (dt) | ||
| LOCATION '/user/hive/warehouse/stock_ticks_cow' | ||
| TBLPROPERTIES ( | ||
| 'primaryKey' = 'symbol', | ||
| 'preCombineField' = 'ts', | ||
| 'hoodie.datasource.hive_sync.enable' = 'true', | ||
| 'hoodie.datasource.hive_sync.database' = 'default', | ||
| 'hoodie.datasource.hive_sync.table' = 'stock_ticks_cow', | ||
| 'hoodie.datasource.hive_sync.jdbcurl' = 'jdbc:hive2://hiveserver:10000/', | ||
| 'hoodie.datasource.hive_sync.mode' = 'jdbc', | ||
| 'hoodie.datasource.hive_sync.partition_fields' = 'dt', | ||
| 'hoodie.datasource.hive_sync.partition_extractor_class' = 'org.apache.hudi.hive.MultiPartKeysValueExtractor', | ||
| 'hoodie.datasource.hive_sync.username' = 'hive', | ||
| 'hoodie.datasource.hive_sync.password' = 'hive' | ||
| ) | ||
| """) | ||
|
|
||
| spark.sql("INSERT INTO stock_ticks_cow VALUES ('GOOG', '2018-08-31 10:29:00', 6330, 1230.5, 1230.5, '2018-08-31')") | ||
| spark.sql("select symbol, ts, volume, open, close from stock_ticks_cow").show(10, false) | ||
| println("STOCK_TICKS_COW_SETUP_SUCCESS") | ||
|
|
||
| // MOR variant: identical schema and seed row. 'type' = 'mor' makes hive sync | ||
| // register stock_ticks_mor_ro / stock_ticks_mor_rt; the initial insert writes | ||
| // parquet base files, so the _ro view already sees the row. | ||
| spark.sql(""" | ||
| CREATE TABLE stock_ticks_mor ( | ||
| symbol STRING, | ||
| ts STRING, | ||
| volume LONG, | ||
| open DOUBLE, | ||
| close DOUBLE, | ||
| dt STRING | ||
| ) USING hudi | ||
| PARTITIONED BY (dt) | ||
| LOCATION '/user/hive/warehouse/stock_ticks_mor' | ||
| TBLPROPERTIES ( | ||
| 'type' = 'mor', | ||
| 'primaryKey' = 'symbol', | ||
| 'preCombineField' = 'ts', | ||
| 'hoodie.datasource.hive_sync.enable' = 'true', | ||
| 'hoodie.datasource.hive_sync.database' = 'default', | ||
| 'hoodie.datasource.hive_sync.table' = 'stock_ticks_mor', | ||
| 'hoodie.datasource.hive_sync.jdbcurl' = 'jdbc:hive2://hiveserver:10000/', | ||
| 'hoodie.datasource.hive_sync.mode' = 'jdbc', | ||
| 'hoodie.datasource.hive_sync.partition_fields' = 'dt', | ||
| 'hoodie.datasource.hive_sync.partition_extractor_class' = 'org.apache.hudi.hive.MultiPartKeysValueExtractor', | ||
| 'hoodie.datasource.hive_sync.username' = 'hive', | ||
| 'hoodie.datasource.hive_sync.password' = 'hive' | ||
| ) | ||
| """) | ||
|
|
||
| spark.sql("INSERT INTO stock_ticks_mor VALUES ('GOOG', '2018-08-31 10:29:00', 6330, 1230.5, 1230.5, '2018-08-31')") | ||
|
Contributor
There was a problem hiding this comment.
|
||
| spark.sql("select symbol, ts, volume, open, close from stock_ticks_mor").show(10, false) | ||
| println("STOCK_TICKS_MOR_SETUP_SUCCESS") | ||
|
|
||
| // Debug aid: proves stock_ticks_mor_ro / stock_ticks_mor_rt got registered. | ||
| spark.sql("show tables").show(100, false) | ||
| println("STOCK_TICKS_TRINO_SETUP_SUCCESS") | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # Transient staging dir populated by build_image.sh (leading slash: must not | ||
| # swallow the shim's io/trino/plugin/ source package under shim/). | ||
| /plugin/ |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only the squash merge button is enabled in
.asf.yaml, so there is no separate commit to drop at merge time and this line ships to master unless it comes off the branch head first. Please strip it before this goes in.