Skip to content

fix: align wide-decimal ANSI overflow value with Spark (#5211) - #5276

Open
sam-1112 wants to merge 4 commits into
apache:mainfrom
sam-1112:fix-wide-decimal-overflow-value-5211
Open

fix: align wide-decimal ANSI overflow value with Spark (#5211)#5276
sam-1112 wants to merge 4 commits into
apache:mainfrom
sam-1112:fix-wide-decimal-overflow-value-5211

Conversation

@sam-1112

@sam-1112 sam-1112 commented Aug 5, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Closes #5211.

Rationale for this change

This is a follow-up to #5072 and #5169.

Under ANSI mode, wide-decimal overflow raises NUMERIC_VALUE_OUT_OF_RANGE, but the value reported by Comet did not match Spark.

Spark reports the pre-toPrecision intermediate value using toPlainString. Comet instead formatted the already-rescaled i256 value at the output scale, which introduced a spurious all zero fractional suffix, such as .000000.

For multiplication, Spark also applies MathContext(39, DOWN) before formatting the reported value. Without this step, values with high storage scales, such as Decimal(38, 18), retained a long .000… suffix that Spark does not emit.

What changes are included in this PR?

  • Format ANSI overflow values using the pre-rescale intermediate value at its natural scale, rather than the rescaled output-scale value.
  • For multiplication, apply Spark's MathContext(39, DOWN) when constructing the reported value.
  • Parse NumericValueOutOfRange parameters using java.math.BigDecimal shims so that Scala BigDecimal's default DECIMAL128 MathContext does not rewrite large integer values.
  • Strengthen CometExpressionSuite to verify that the value parameter matches Spark and does not contain a spurious .000000 suffix.

Note: This branch is based on #5169. If #5169 has not been merged yet, please review and merge it first, or update this PR's base branch accordingly.

How are these changes tested?

  • Native unit tests in wide_decimal_binary_expr.rs covering overflow-value formatting for addition and multiplication, including the Decimal(38, 18) multiplication case.
  • Scalar decimal overflow tests in CometExpressionSuite for both legacy and ANSI modes.

@peterxcli

peterxcli commented Aug 6, 2026

Copy link
Copy Markdown
Member

should we change this to draft? as #5169 hasnt been merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wide-decimal overflow reports a different value than Spark

2 participants