Proposal: add Spark 4.1 support

[billpratt]

Motivation

PyDeequ currently supports the Spark 3.5 line, while the upstream Deequ project publishes a compatible Spark 4.1 artifact: com.amazon.deequ:deequ:2.0.18-spark-4.1. Supporting this artifact would let PyDeequ run on Spark 4.1 while retaining Spark 3.5 support.

This is intended as generic Apache Spark support; it does not add platform-specific configuration or documentation.

Relationship to #283

This proposal is designed to follow the Spark 3.5 upgrade in #283. The intended final mapping would be:

{
    "3.5": "com.amazon.deequ:deequ:2.0.21-spark-3.5",
    "4.1": "com.amazon.deequ:deequ:2.0.18-spark-4.1",
}

I would coordinate rebasing/merge order with #283 to avoid overlapping changes in dependency metadata, CI, and documentation.

Proposed design

1. Add an exact SPARK_VERSION mapping for 4.1 to deequ:2.0.18-spark-4.1.
2. Expand the optional PySpark dependency range to allow the Spark 4.1 line.
3. Make the Py4J/Scala collection bridge work with both Scala 2.12 (Spark 3.5) and Scala 2.13 (Spark 4.1):
   • use scala.collection.JavaConverters, which is available in both lines;
   • create empty Scala sequences through the existing sequence-conversion helper rather than calling Seq.empty() through Py4J.
4. Add Spark 4.1 CI coverage with a compatible Python/Java runtime.
5. Add focused configuration and runtime tests, plus documentation for selecting Spark 4.1 via SPARK_VERSION=4.1.
6. Update package constraints and lock data so a Spark 4.1 installation receives PySpark-compatible pandas and NumPy versions.

Compatibility expectations

• A process selects one Deequ artifact based on its Spark runtime; it does not load Spark 3.5 and 4.1 artifacts together.
• Spark 3.5 behavior remains supported and continues to select its matching Deequ artifact.

Feedback requested

Would maintainers prefer this as a follow-up PR after #283 merges, or as a coordinated PR that incorporates/rebases onto #283's Spark 3.5 changes?

[github-actions]

Thanks for this detailed proposal! This is a well-scoped enhancement request and the validation results (162 passed, 1 skipped, 22 xfailed against PySpark 4.1.2) are encouraging.

We'd welcome a PR for this. Given the dependency on #283, please coordinate as a follow-up after #283 merges to keep the changes clean. The proposed design looks sound — particularly the JavaConverters compatibility shim for Scala 2.12/2.13 and the updated SPARK_TO_DEEQU_COORD_MAPPING.

A few things to ensure in the PR:

• The pyproject.toml PySpark version upper bound is updated to allow <5.0.0
• CI matrix includes a Spark 4.1 + Java 17 job
• SPARK_VERSION env var docs are updated to list 4.1 as a supported value

Looking forward to the PR!

This has also been flagged for our maintainer team to review.

---

Generated by AI (model: us.anthropic.claude-sonnet-4-6, prompt: ce48b3a9) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow

[billpratt]

fyi @sudsali. I'm working on a PR for Spark 4.1 support. I'd like to coordinate with #283.

[github-actions]

Thanks for the heads-up! To coordinate cleanly:

1. Wait for #283 to merge first — it updates the Spark 3.5 Deequ artifact version. Rebasing your Spark 4.1 PR on top of that will avoid conflicts in configs.py and pyproject.toml.

2. Key changes needed in your PR (on top of #283):

   • configs.py: add "4.1": "com.amazon.deequ:deequ:2.0.18-spark-4.1" to SPARK_TO_DEEQU_COORD_MAPPING
   • pyproject.toml: bump PySpark upper bound to <5.0.0
   • scala_utils.py: replace scala.collection.JavaConversions with scala.collection.JavaConverters for Scala 2.13 compatibility (Spark 4.1 ships Scala 2.13)
   • CI matrix: add a Spark 4.1 + Java 17 job
   • Docs: add 4.1 to the supported SPARK_VERSION values

3. The JavaConverters change is the most critical — JavaConversions was removed in Scala 2.13, so to_scala_seq and related helpers in scala_utils.py will break on Spark 4.1 without it.

Feel free to tag the maintainers once #283 is merged and your PR is ready for review. Looking forward to it!

---

Generated by AI (model: us.anthropic.claude-sonnet-4-6, prompt: 67928783) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow

[sudsali]

@billpratt I have merged in #283

[billpratt]

@sudsali thanks. #287 is now ready for review.