[WIP][SS][PYTHON] Stop leaking Python workers and isolation sandboxes on TransformWithState init failure

[jon-gao-db]

What changes were proposed in this pull request?

This patch makes the driver-side TransformWithStateInPySpark pre-initialization runner clean up on every lifecycle path. It moves init() and process() into Utils.tryWithSafeFinally, guaranteeing that stop() runs when initialization or processing fails. It also preserves the original initialization failure when cleanup itself throws by relying on tryWithSafeFinally's suppressed-exception behavior.

The runner's state-server daemon thread is nullable until the end of initialization, so stop() now checks for null before interrupting it. A six-case regression suite covers initialization failure cleanup, repeated failures, suppressed cleanup errors, process failure wrapping, the success path, and stopping before state-server startup.

Why are the changes needed?

StreamingPythonRunner.init() creates the Python worker before it finishes initialization. If initialization fails afterward, the previous code never reached stop(), leaking the worker and its associated resources for the driver's lifetime. Repeated streaming restarts can accumulate these leaked resources and eventually prevent new isolated workers from starting.

Does this PR introduce any user-facing change?

No. Successful execution behavior is unchanged. This only restores cleanup on existing failure paths and preserves the original initialization error instead of allowing cleanup failures or a null-thread error to obscure it.

How was this patch tested?

Added TransformWithStateInPySparkPreInitCleanupSuite with six tests and no real Python worker dependency. Static validation passed with git diff --check and the changed-file line-length check.

The focused Spark test could not run because this checkout does not have sbt installed and its launcher download was unavailable. A Maven test-compile fallback also could not resolve dependencies because the configured Maven mirrors were unreachable from the environment.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (Codex CLI).