docs: Documentation updates in preparation for 0.16 release#4244
Open
andygrove wants to merge 17 commits intoapache:mainfrom
Open
docs: Documentation updates in preparation for 0.16 release#4244andygrove wants to merge 17 commits intoapache:mainfrom
andygrove wants to merge 17 commits intoapache:mainfrom
Conversation
…examples Update user-facing docs ahead of the 0.16 release: - Promote Spark 4.1 from experimental to fully supported across the installation page, compatibility guide, and Gluten comparison; keep 4.2 listed as experimental. Spark 4.1.1 now runs in CI under both JDK 17 and 21. - Restructure the Supported Operating Systems section in the installation guide to make clear that published Maven jars cover Linux only and that macOS users must build from source. Drop the Intel macOS claim since Apple Silicon is the only macOS variant exercised in CI. - Flip seven recently-added math expressions to supported in the contributor-guide tracking page: acosh, asinh, atanh, cbrt, degrees, pi, radians. - Move spark.comet.scan.enabled to the testing category and rewrite its description to reflect that it is intended for Comet's own test suites only. Remove the corresponding mention from the data sources page. - Replace hard-coded Comet versions in the Iceberg and Kubernetes guides with the \$COMET_VERSION placeholder used elsewhere, and drop the redundant spark.sql.extensions=...CometSparkSessionExtensions conf from the Iceberg examples (CometPlugin registers it automatically).
andygrove
changed the title
…t scans PR apache#4011 added non-AQE DPP and PR apache#4112 added AQE DPP with broadcast reuse for native Parquet scans, but the docs still claimed AQE DPP was unsupported. Update the scans compatibility page, the contributor-guide roadmap, and the Iceberg guide accordingly. AQE DPP for Iceberg native scans remains future work, tracked at apache#3510.
# Conflicts: # docs/source/user-guide/latest/iceberg.md # docs/source/user-guide/latest/kubernetes.md
andygrove
commented
May 7, 2026
Comment on lines
+97
to
+99
| .category(CATEGORY_TESTING) | ||
| .doc("Whether to enable native scans. Intended for use in Comet's own test suites to " + | ||
| "selectively disable native scans; not intended for production use.") |
Member
Author
There was a problem hiding this comment.
this is unrelated to 0.16 / spark 4 changes, but we don't want users using this config
andygrove
commented
May 7, 2026
Comment on lines
+206
to
+208
| # corresponding Spark release, so the scala-2.13 profile is not used here. | ||
| ./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-4.0 -DskipTests install | ||
| ./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-4.1 -DskipTests install |
comphead
reviewed
May 7, 2026
| 10.042 | ||
| ] | ||
| } No newline at end of file | ||
| "1": [12.007], |
comphead
reviewed
May 7, 2026
| ```shell | ||
| $SPARK_HOME/bin/spark-shell \ | ||
| --packages org.apache.datafusion:comet-spark-spark4.1_2.13:0.14.0,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1 \ | ||
| --packages org.apache.datafusion:comet-spark-spark3.5_2.12:$COMET_VERSION,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1 \ |
andygrove
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
We are releasing 0.16.0 soon. This is a good time to review/update documentation.
What changes are included in this PR?
spark.comet.scan.enabledto a test-only configANSI Error Propagationto indexdocs/source/contributor-guide/spark_expressions_support.md- many expressions were added recently