Warn when a standalone pyspark dependency collides with databricks-connect

[rugpanov]

Problem

databricks-connect vendors its own pyspark — it ships the pyspark/ package
tree rather than depending on the standalone distribution. A project that also
declares a standalone pyspark dependency resolves it into the same namespace, the
two overwrite each other, and the environment then fails to start a session,
surfacing to users as a cryptic Java or protobuf gencode error long after
setup-local reported success.

The merge-warning system compared the user's pins against the env constraints only,
and the environment deliberately does not pin pyspark, so this collision was never
flagged.

What

• Add warning code W_STANDALONE_PYSPARK_CONFLICT.
• isPysparkDep: PEP 503 name match for the standalone pyspark requirement.
• standalonePysparkWarnings: report the collision once, independent of the pinned
  version (it is a coexistence conflict, not a version one), emitted only when the
  environment manages databricks-connect.
• Wire it into detectMergeWarnings between the databricks-connect warnings and
  the constraint conflicts.

Testing

• go test ./libs/localenv/ (unit tests + regenerated merge-warnings acceptance
  goldens).
• go vet ./libs/localenv/.

This pull request and its description were written by Isaac.

[eng-dev-ecosystem-bot]

Integration test report

Commit: 34667be

Run: 31820299316

	Env	🟨​KNOWN	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
🟨​	aws linux	3	1	4	286	1146	6:56
🟨​	aws windows	3	1	4	288	1144	7:13
🟨​	azure linux	3	1	4	285	1146	8:18
🟨​	azure windows	3	1	4	287	1144	8:23
💚​	gcp linux		1	5	286	1146	6:35
💚​	gcp windows		1	5	288	1144	5:30

8 interesting tests: 4 SKIP, 3 KNOWN, 1 RECOVERED

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🟨​	TestFetchRepositoryInfoAPI_FromRepo	🟨​K	🟨​K	🟨​K	🟨​K	🙈​S	🙈​S
🟨​	TestFetchRepositoryInfoAPI_FromRepo/root	🟨​K	🟨​K	🟨​K	🟨​K
🟨​	TestFetchRepositoryInfoAPI_FromRepo/subdir	🟨​K	🟨​K	🟨​K	🟨​K

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
5:44	aws windows	TestAccept
5:27	azure windows	TestAccept
5:22	gcp windows	TestAccept
3:04	azure linux	TestAccept
2:56	gcp linux	TestAccept
2:55	aws linux	TestAccept

[anton-107]

Reviewed as part of the three-PR stack (this, #6277, and databricks/databricks-vscode#2121). I reproduced the underlying bug rather than reasoning from the description.

Verified empirically

Cloning a real working databricks-connect 18.0.6 venv and layering standalone pyspark 4.2.0 on top reproduces the failure this stack is chasing:

ImportError: cannot import name 'PythonUDFEnvironment' from
'pyspark.sql.connect.expressions'   ← raised at `from databricks.connect import ...`

The premise behind the warning holds: databricks-connect ships the pyspark package tree (its top_level.txt lists pyspark) but registers no pyspark distribution. Confirmed on 13.3.12, 14.3.19, 17.3.8 and 18.0.6 — so the collision is real and the diagnosis is right.

Test coverage is good: the three new unit tests cover ordering, constraints-only, and the report-once dedup, and the acceptance goldens exercise warnings[] end to end.

Finding — constraints-only gating disagrees with #6277

This warning gates on the mode (c.DatabricksConnect != ""); #6277's hard fail gates on what is actually in the venv (dbcVer != "" && pysparkVer != ""). The mismatch is reachable.

The comment here says "in constraints-only mode the env installs no databricks-connect, so a standalone pyspark is harmless." But mergeDatabricksConnect is a no-op on an empty value, not a deletion (merge.go:351) — so a project that already pins databricks-connect itself keeps that pin, and uv installs it. That yields:

constraints-only + project's own dbconnect pin + standalone pyspark
  → merge:    warning SKIPPED   (gated on mode)
  → validate: HARD FAIL E_VALIDATE  (gated on venv contents)

The user gets no warning at merge and then an unexplained hard failure at validate. Because dbcPin == "" on that path, they also aren't told which databricks-connect is involved.

Consider keying this warning on databricks-connect being present by any route rather than on the mode, so the two PRs agree. #6277's comment already reasons that way ("Keyed on both packages actually being present in the venv, not on the mode"); this one doesn't. No test covers constraints-only + a user's own dbconnect pin.

Smaller notes

• isPysparkDep duplicates isDatabricksConnectDep verbatim except for the compared literal — two copies of the PEP 508 name extraction plus PEP 503 normalization. One isDepNamed(entry, name string) bool with both as thin wrappers would avoid the drift.
• standalonePysparkWarnings returns a slice but can only ever return 0 or 1 element (it returns on first match). (Warning, bool) would state the contract honestly — though the plural name matches the sibling dbconnectWarnings, so this is a judgment call.
• detectMergeWarnings is only called on the non-greenfield path. That is right for the override warnings — greenfield has nothing of the user's to override — but a standalone pyspark is not an override, it's a property of the dependency list. Almost certainly nothing to flag today, since greenfield renders a fresh pyproject with no user deps; flagging it only because the reasoning that justifies the placement doesn't extend to this new warning.

Testing

I could not run the Go suites here: the repo requires Go 1.26, the local toolchain is 1.24, and the module proxy is blocked in my environment. I verified the logic by reading it and by reproducing the underlying package behaviour in Python — not by executing the tests. Taking the reported go test ./libs/localenv/ pass at face value.

Approving — a warning is advisory and cannot fail a run, so the gating question above is safe to settle in a follow-up. Worth resolving before #6277 merges, though, since that PR turns this same condition into a hard error.

[rugpanov]

Thanks for the review and the empirical repro. Addressed in the latest push.

Finding — constraints-only gating disagreed with #6277. Fixed: the warning now gates on c.DatabricksConnect != "" || len(dbconnectPins(survivors)) > 0 instead of the mode alone, so a project that pins its own databricks-connect in constraints-only mode (kept, since mergeDatabricksConnect is a no-op on an empty value) now gets the warning at merge — matching #6277's venv-based hard fail. Added TestStandalonePysparkFiresInConstraintsOnlyWhenUserPinsDBConnect.

Nit — duplicated name extraction. Deduped isDatabricksConnectDep/isPysparkDep into one isDepNamed(entry, name), with both as thin wrappers.

Other nits. Left standalonePysparkWarnings returning a slice and the non-greenfield-only placement as-is, for consistency with the sibling dbconnectWarnings — happy to change if you feel strongly.