Skip to content

[SPARK-58738][SQL] Add parse_command function for SQL statement JSON analysis - #57962

Open
srielau wants to merge 5 commits into
apache:masterfrom
srielau:SPARK-58738
Open

[SPARK-58738][SQL] Add parse_command function for SQL statement JSON analysis#57962
srielau wants to merge 5 commits into
apache:masterfrom
srielau:SPARK-58738

Conversation

@srielau

@srielau srielau commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add a built-in scalar SQL function parse_command(sqlStmt) that parse-only analyzes a SQL statement string and returns a compact JSON description:

  • parse_success and, on failure, a nested STANDARD-format error object (errorClass, messageTemplate, sqlState, messageParameters) so batch evaluation never fails a query on bad SQL
  • Statement classification from ISO/IEC 9075-2:2023 Table 39 (statement_identifier, statement_code, statement_type, statement_class); Spark-only statements use product-specific identifiers with negative implementation-defined codes
  • Multipart table_references and function_references (identifier part sequences)
  • select_list entries with name parts and expression text
  • parameter_markers (named and unnamed_count)
  • as_subquery for CREATE TABLE ... AS SELECT (still Table 39 CREATE TABLE / code 77)

Why are the changes needed?

Dialect-migration and SQL analysis tooling need an efficient, batchable way to verify that translated SQL parses and to extract statement kind / references without requiring catalog objects to exist.

Does this PR introduce any user-facing change?

Yes. New SQL function:

SELECT parse_command('SELECT a FROM t');
-- {"parse_success":true,"statement_identifier":"SELECT","statement_code":21,...}

SELECT parse_command('SELEC');
-- {"parse_success":false,"error":{"errorClass":"PARSE_SYNTAX_ERROR",...}}

How was this patch tested?

Added ParseCommandResultSuite and ParseCommandSuite (catalyst). Ran:

build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.parser.ParseCommandResultSuite org.apache.spark.sql.catalyst.expressions.ParseCommandSuite'

All 13 tests passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor Grok 4.5

…rences

UnresolvedWith keeps CTE definitions in innerChildren, so collectWithSubqueries
missed tables/functions/params inside WITH. Add foreachPlanDeep and broader CTE
and nested-subquery tests.
Classify CompoundBody as BEGIN END (-22) and deep-walk SingleStatement
roots, exception handlers, and simple CASE else bodies so nested refs
are not dropped.
Cover SELECT/DML/DDL, CTEs, scripts, errors, and batch evaluation via
sql-tests inputs with regenerated result and analyzer golden files.
Keep only statement_identifier/code on success, expose error line/position,
and broaden complex SQL, scripting, and non-syntax parser error tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant