Skip to content

Spark: Verify 4.0.5 RC3 - #18340

Draft
manuzhang wants to merge 6 commits into
apache:mainfrom
manuzhang:spark-4.0.5-rc1
Draft

manuzhang wants to merge 6 commits into
apache:mainfrom
manuzhang:spark-4.0.5-rc1

Conversation

@manuzhang

@manuzhang manuzhang commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

This draft verifies Iceberg against Spark 4.0.5 RC3 (b472a3aa3c4b71b5a6ff068ef396f757bd662e59) from Apache staging orgapachespark-1539. Maven version remains 4.0.5; staging is restricted to that version. Official tag, POM coordinates, artifact checksums and embedded revision are verified. Existing regressions are retained. New mixed-type subset-key joins cover SPARK-60203 with AQE/skew and partial clustering variants. Existing computed-key ordering and checkpointed-scan tests remain. Current PR head: 61f07d1e6d1c602731f86887242ede94176f6467.

Issues: SPARK-60203, SPARK-59905.

Test plan and results:

  • Compare new mixed-key and computed-ordering cases against an SPJ-disabled baseline in LOCAL/DISTRIBUTED planning; assert non-leading key projection and relevant plan behavior. Focused run: 8 cases, 0 failures/errors. Use isolated candidate dependencies and assert the exact runtime revision.
  • Prior RC2 comparison: 8 cases, 2 expected regression failures (two mixed-key type failures; partial-clustering variants and ordering cases passed). Exact prior runtime revision and published checksums verified; environmental/setup failures are excluded.
  • Final-head local JDK21 validation: 145 tests, 145 passed, 0 skipped, 0 failures/errors. Full SPJ 80 and row-level SPJ 8, focused checkpoint/restart, filter/catalog checks, assemblies, checkstyle and formatting passed.
  • Current PR check rollup: 42 success (observed 2026-10-22T19:47:44.428356+00:00). All currently reported checks are terminal.
  • Spark CI core/extensions checkout verified at merge b6138dbcf0cc78eed467b6d21f9398289ef2f8a8, containing this exact PR head. All ten Spark matrix jobs passed.
  • zizmor completed with success at 2026-10-22T19:44:23Z (original run, attempt 1); no runner/security configuration changes.
  • Scope: default local checkpoint-manager integration; S3A/abortable close-failure injection and Spark async progress-tracking failure paths are not tested. Iceberg async planning is distinct. No merge or release vote.

AI Disclosure

  • Model: GPT-6
  • Platform/Tool: Codex
  • Human Oversight: unreviewed
  • Prompt Summary: Update the existing candidate verification draft; generate targeted mixed-key/order regressions, verify fresh artifacts, run compatibility/checkpoint/SPJ validation, and follow exact-head CI.

Use Spark 4.0.5 RC1 staging artifacts to verify Iceberg compatibility.

Generated-by: Codex
@github-actions github-actions Bot added the build label Oct 1, 2026
Verify runtime filtering on trailing partition keys, binary key equality, and chained joins with undeclared keys to detect Spark 4.0.4 regressions.

Generated-by: Codex
@github-actions github-actions Bot added the spark label Oct 1, 2026
Use the RC2 staging repository for the existing Spark 4.0.5 compatibility tests.

Generated-by: Codex
@manuzhang manuzhang changed the title Spark: Verify 4.0.5 RC1 Spark: Verify 4.0.5 RC2 Oct 13, 2026
Verify the two RC2 fixes to catch lost rows after checkpointing and incorrect broadcast join ordering.

Generated-by: Codex
Resolve Spark 4.0.5 from RC3 staging for compatibility validation.

Generated-by: Codex
@manuzhang manuzhang changed the title Spark: Verify 4.0.5 RC2 Spark: Verify 4.0.5 RC3 Oct 22, 2026
Compare AQE subset-key joins and ordering against a disabled-SPJ baseline, asserting the relevant partition projection path.

Generated-by: Codex

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant