Fix mobile gaming release validation background process cleanup for Java 21 compatibility - #39658
Fix mobile gaming release validation background process cleanup for Java 21 compatibility#39658damccorm wants to merge 8 commits into
Conversation
…ava 21 compatibility In Java 20+, Thread.stop() unconditionally throws UnsupportedOperationException. This caused mobile gaming validation scripts to fail when stopping background injector and leaderboard threads, leaving the child processes running as orphans and causing the nightly snapshot validation workflow to hang and time out at 6 hours. This change introduces Process-based background execution and cleanup in TestScripts to properly terminate background processes across Java versions.
|
Validated with nightly snapshot run: https://github.com/apache/beam/actions/runs/31122841120 (all Java quickstart and mobile gaming validation tasks passed in 31m 34s). |
Updated the default release version and modified the description for the RELEASE input.
| def jvmArgs = "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED" | ||
| t.run """mvn exec:exec -q -Dexec.executable=java \ | ||
| -Dexec.args="${jvmArgs} -cp ${cp} org.apache.beam.examples.WordCount \ | ||
| --inputFile=pom.xml --output=counts --runner=SparkRunner" """ |
There was a problem hiding this comment.
These changes fix:
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project word-count-beam:
An exception occured while executing the Java class. class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x1531e31f)
cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module
which can happen with Java 21 and was getting masked by timeouts
|
Assigning reviewers: R: @liferoad for label build. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| def pb = new ProcessBuilder(shell) | ||
| pb.directory(var.curDir) | ||
| pb.redirectErrorStream(true) | ||
| def proc = pb.start() |
There was a problem hiding this comment.
Could we retain and check background process exit status here? _executeBackground only drains stdout, and stopProcess removes an already-exited process without inspecting its result. DirectRunner keeps leaderboard_DirectRunner_* tables, so failed Injector or LeaderBoard commands can match stale rows and report [SUCCESS]. Reproduced with both background Maven commands exiting 42 while script exited 0.
There was a problem hiding this comment.
Thanks - addressed this + did some other cleanup (running https://github.com/apache/beam/actions/runs/31172410588 to verify I didn't break anything with this)
…ng-java21' into fix-mobile-gaming-java21
…eduplicate _mvn logic, trim and quote classpath
|
Latest validation run succeeded: https://github.com/apache/beam/actions/runs/31172410588 (completed all runner validation tasks in 48m 11s). |
2 participants
Description
In Java 20+,
Thread.stop()unconditionally throwsjava.lang.UnsupportedOperationException(JEP 421).Following the CI upgrade to Java 21 in #39205, the
beam_PostRelease_NightlySnapshotvalidation workflow began timing out at 6 hours becausemobilegaming-java-direct.groovyandmobilegaming-java-dataflow.groovyrelied onThread.stop()to stop background injector and leaderboard threads.When
Thread.stop()threwUnsupportedOperationException, the main test thread crashed and left the backgroundInjectorprocess running indefinitely as an orphan, publishing messages to Pub/Sub and hanging the build until the 6-hour execution timeout was hit.This change:
runBackground,stopProcess, andstopAllBackgroundProcessestoTestScripts.groovyto manage and cleanly terminate child background processes (and their process trees) using Java'sProcessHandleAPIs, with process exit verification.TestScriptsthat callsstopAllBackgroundProcesseswith explicitwaitFor()to guarantee all spawned background processes are terminated before JVM exit.mobilegaming-java-direct.groovyandmobilegaming-java-dataflow.groovyto uset.runBackground/t.stopProcesswith bounded retry loops and fresh BigQuery table cleanup.quickstart-java-spark.groovyandquickstart-java-flinklocal.groovyto usemvn exec:execwith trimmed/quoted classpaths and required JVM--add-opensflags for Java 17/21 compatibility.BeamModulePlugin.groovy,release/build.gradle.kts, and.github/workflows/beam_PostRelease_NightlySnapshot.ymlto properly default toproject.versionon nightly snapshot workflow dispatch runs.Validation
Successful nightly snapshot validation run: https://github.com/apache/beam/actions/runs/31172410588 (completed in 48m 11s, down from timing out at 6h).
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.