Skip to content

test: enable SPARK-32509 DPP test for Spark 4.0#4100

Closed
andygrove wants to merge 1 commit intoapache:mainfrom
andygrove:enable-spark32509-spark4
Closed

test: enable SPARK-32509 DPP test for Spark 4.0#4100
andygrove wants to merge 1 commit intoapache:mainfrom
andygrove:enable-spark32509-spark4

Conversation

@andygrove
Copy link
Copy Markdown
Member

Which issue does this PR close?

Partially addresses #4045.

Rationale for this change

The SPARK-32509 test in DynamicPartitionPruningSuite was disabled across all Spark versions under issue #4045. Re-running it on Spark 4.0.1 with Comet enabled now passes in all 6 concrete suite variants (V1/V2/V2Filter × AEOff/AEOn).

What changes are included in this PR?

Removes the IgnoreComet annotation from the SPARK-32509 test in dev/diffs/4.0.1.diff. The companion SPARK-34637 test referenced by the same IgnoreComet annotation remains ignored under #4045 since it still fails on Spark 4.0.1 with AQE enabled.

The test was checked on Spark 3.5.8 as well but still fails there in DynamicPartitionPruningV1SuiteAEOn, so the change is scoped to Spark 4.0 only.

How are these changes tested?

Locally ran the test against the patched Spark 4.0.1 source with ENABLE_COMET=true ENABLE_COMET_ONHEAP=true:

sql/testOnly org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOff \
             org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn \
             org.apache.spark.sql.DynamicPartitionPruningV2SuiteAEOff \
             org.apache.spark.sql.DynamicPartitionPruningV2SuiteAEOn \
             org.apache.spark.sql.DynamicPartitionPruningV2FilterSuiteAEOff \
             org.apache.spark.sql.DynamicPartitionPruningV2FilterSuiteAEOn \
  -- -z "SPARK-32509"

All 6 variants pass.

@andygrove andygrove closed this Apr 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant