[dborowitz]

Note

Stacked on #207 (dborowitz-ci-and-repo-modernization).

Summary

Fixes broken Windows support for local artifact uploads, %dpip / PyPI package installation, and VS Code IDE / Data Agent Kit (DAK) detection, and expands GitHub Actions unit, local Spark, and integration test CI across Linux, macOS, and Windows with multi-region stockout fallbacks.

1. Windows Bug Fixes (pypi_artifacts.py, session.py, ide.py)

• Valid temporary filenames in PyPiArtifacts.write_packages_config (google/cloud/managed_spark_connect/pypi_artifacts.py):
  • Replaces self.__str__() (<google.cloud.managed_spark_connect.pypi_artifacts.PyPiArtifacts object at ...>) with str(uuid.uuid4()) so temporary config filenames do not contain < or >, which are reserved characters on Windows (OSError: [Errno 22] Invalid argument).
• Windows drive-letter artifact path normalization & addArtifact alias (google/cloud/managed_spark_connect/session.py):
  • Adds _normalize_artifact_path to convert local Windows drive-letter paths (C:\..., preserving any #alias archive fragments) into file:///C:/... URIs in ManagedSparkSession.addArtifacts—both for user-supplied artifact paths and for the generated pypi=True config file—so PySpark's ArtifactManager._parse_artifacts does not parse the drive letter as an unsupported URL scheme "c".
  • Aliases ManagedSparkSession.addArtifact = addArtifacts so singular session.addArtifact(...) calls (such as from %dpip install) go through ManagedSparkSession.addArtifacts rather than falling through to the unpatched base SparkSession.addArtifact.
• Windows VS Code InnoSetup <commit[:10]> layout in IDE detection (google/cloud/managed_spark_connect/ide.py):
  • Updates _find_product_json to check <exe_dir>\<commit[:10]>\resources\app\product.json (used by Windows VS Code 1.96+ InnoSetup installers where Code.exe sits at the install root and resources\app\product.json sits under the 10-hex-character commit prefix directory) so process-tree IDE and DAK detection works on Windows.
  • Adds Windows capture fixtures (p4_vscode_windows.json and p4_antigravity_ide_windows.json, including the Windows .venv\Scripts\python.exe launcher redirector hop in the process chain) and skips POSIX-only /proc and POSIX path replay tests on Windows (tests/unit/test_ide.py).

2. Cross-Platform Unit, Local Spark, & Integration Test CI (.github/workflows/tests.yaml, .github/workflows/integration-tests.yaml)

• Expands tests.yaml and integration-tests.yaml strategy.matrix across os: ["ubuntu-latest", "macos-latest", "windows-latest"] with defaults.run.shell: bash.
• Configures cross-platform uv cache paths and JAVA_HOME: ${{ env.JAVA_HOME_17_X64 || env.JAVA_HOME_17_ARM64 }} so ARM64 macos-latest runners locate Java 17 for tests/local/.
• Moves concurrency in integration-tests.yaml to the workflow level so all 3 OS matrix legs run in parallel without cancelling each other, and replaces GNU-only date -d in /gcbrun authorization with portable jq fromdateiso8601 comparisons.

3. Multi-Region Stockout Fallbacks (tests/integration/conftest.py, tests/integration/test_session.py)

• Expands _candidate_regions in CI across ("us-central1", "us-east1", "us-west1") when test_subnet == "default" and rotates the starting region per OS (linux → us-central1, darwin → us-east1, win32 → us-west1) so concurrent OS runners spread their initial 3-VM cluster load across regions while falling back on 503 ServiceUnavailable.
• Releases the shared connect_session early via isolated_session_state once no remaining tests need it, and uses a get_session_controller_client fixture for isolated session lifecycle tests.

Verification

• CI: Unit tests, local Spark tests, and integration tests pass on ubuntu-latest, macos-latest, and windows-latest.
• Manual testing on Windows: Tested end-to-end in a Jupyter notebook inside VS Code on Windows with Data Agent Kit (googlecloudtools.datacloud), verifying IDE/DAK detection (enabled and disabled states), creating a ManagedSparkSession, clicking the rendered dataproc-resource: session link to open the session directly in VS Code, installing packages via %dpip install, and querying via %%sparksql.

[gemini-code-assist]

Code Review

This pull request improves integration test reliability and resource utilization by introducing platform-based region rotation for CI jobs to distribute load, and dynamically terminating shared Spark sessions early when no subsequent tests require them. It also updates session lifecycle tests to support region fallbacks. The reviewer feedback correctly points out that importing helper functions directly from conftest.py is a pytest anti-pattern, and suggests refactoring _get_session_controller_client into a registered pytest fixture to be cleanly injected into the test functions.

[dborowitz]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces Windows compatibility improvements, including support for Windows InnoSetup VS Code layouts, converting Windows drive-letter paths to file:// URIs for PySpark artifacts, and rotating fallback regions by platform in CI to distribute load. Feedback suggests using a context manager with os.scandir in ide.py to prevent unclosed directory handles on Windows, and adding a defensive check in session.py to handle cases where the active session UUID is None before writing package configurations.

[gemini-code-assist]

Using os.scandir without a context manager can lead to unclosed directory handles, which is particularly problematic on Windows where file/directory locking is strict. Additionally, we can filter for directories and hex commit prefixes during the scan to avoid sorting non-matching entries, improving both safety and performance.

Suggested change

	try:
	entries = sorted(os.scandir(exe_dir), key=lambda e: e.name)
	except OSError:
	entries = []
	for entry in entries:
	if _is_hex_commit_prefix(entry.name):
	path = os.path.join(entry.path, "resources", "app", "product.json")
	product = _read_product_json(path)
	if product is not None:
	commit = _str_or_none(product.get("commit"))
	if commit and commit.lower().startswith(entry.name.lower()):
	return path, product, False
	try:
	with os.scandir(exe_dir) as it:
	entries = sorted(
	[e for e in it if e.is_dir() and _is_hex_commit_prefix(e.name)],
	key=lambda e: e.name,
	)
	except OSError:
	entries = []
	for entry in entries:
	path = os.path.join(entry.path, "resources", "app", "product.json")
	product = _read_product_json(path)
	if product is not None:
	commit = _str_or_none(product.get("commit"))
	if commit and commit.lower().startswith(entry.name.lower()):
	return path, product, False

[gemini-code-assist]

If self._active_s8s_session_uuid is None (e.g., if the session is not active or has been stopped), calling write_packages_config will result in a TypeError inside os.path.join. Adding a defensive check here prevents a cryptic crash and provides a clear error message.

            if not self._active_s8s_session_uuid:
                raise ManagedSparkConnectException(
                    "Cannot add PyPI artifacts: no active Managed Spark session."
                )
            artifacts = PyPiArtifacts(set(artifact))
            logger.debug("Making addArtifact call to install packages")
            self.addArtifact(
                _normalize_artifact_path(
                    artifacts.write_packages_config(
                        self._active_s8s_session_uuid
                    )
                ),
                file=True,
            )