[#11590] fix(doris): fix partition parsing for Doris 3.0+ format by jiangxt2 · Pull Request #11732 · apache/gravitino

jiangxt2 · 2026-06-18T05:21:59Z

What changes were proposed in this pull request?

Partition Regex Fix (`DorisUtils.java`)

Added \\s* to PARTITION_INFO_PATTERN to tolerate whitespace between LIST/RANGE and ( in Doris 3.0+ SHOW CREATE TABLE output (PARTITION BY LIST ( vs PARTITION BY LIST()

Integration Tests

CatalogDorisIT (1.2.x): testListPartitionRoundTrip + testMultiColumnListPartitionRoundTrip
CatalogDoris3xIT (3.0.x): testListPartitionRoundTrip + testRangePartitionRoundTrip
CatalogDoris4xIT (4.0.x): testListPartitionRoundTrip + testRangePartitionRoundTrip

All tests verify partition metadata survives the create → Doris → load round-trip.

Fixed: #11590

Does this PR introduce any user-facing change?

No. Partition parsing fix is internal to Gravitino metadata loading.

How was this patch tested?

Unit tests: TestDorisUtils (existing, covers both LIST and RANGE parsing)

Integration tests (Docker, all three Doris versions):

CatalogDorisIT (1.2.x): 19/19 passed
CatalogDoris3xIT (3.0.x): 8/8 passed
CatalogDoris4xIT (4.0.x): 8/8 passed

github-actions · 2026-06-18T09:42:30Z

Code Coverage Report

Overall Project	67.66% `+0.36%`	🟢
Files changed	84.04%	🟢

Module	Coverage
aliyun	1.72%	🔴
api	46.55% `-0.19%`	🟢
authorization-common	85.96%	🟢
aws	42.04%	🟢
azure	2.47%	🔴
catalog-common	9.92%	🔴
catalog-fileset	80.23%	🟢
catalog-glue	66.91%	🟢
catalog-hive	79.42%	🟢
catalog-jdbc-clickhouse	80.55% `+3.68%`	🟢
catalog-jdbc-common	44.22%	🟢
catalog-jdbc-doris	81.6% `+1.58%`	🟢
catalog-jdbc-hologres	54.03%	🟢
catalog-jdbc-mysql	79.23%	🟢
catalog-jdbc-oceanbase	80.91%	🟢
catalog-jdbc-postgresql	82.29%	🟢
catalog-jdbc-starrocks	78.51%	🟢
catalog-kafka	77.01%	🟢
catalog-lakehouse-generic	59.18%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	86.14%	🟢
catalog-lakehouse-paimon	84.25%	🟢
catalog-model	77.72%	🟢
cli	44.49%	🟢
client-java	78.15%	🟢
common	51.45%	🟢
core	82.44% `+0.06%`	🟢
filesystem-hadoop3	77.3%	🟢
flink	0.0%	🔴
flink-common	47.09%	🟢
flink-runtime	0.0%	🔴
gcp	14.12%	🔴
hadoop-auth	68.0%	🟢
hadoop-common	12.7%	🔴
hive-metastore-common	53.29%	🟢
iceberg-common	63.63%	🟢
iceberg-rest-server	74.37% `+2.42%`	🟢
idp-basic	85.71%	🟢
integration-test-common	0.0%	🔴
jobs	66.17%	🟢
lance-common	20.67%	🔴
lance-rest-server	64.84%	🟢
lineage	53.02%	🟢
optimizer	83.17%	🟢
optimizer-api	21.95%	🔴
server	85.97%	🟢
server-common	74.62%	🟢
spark	28.57%	🔴
spark-common	46.01%	🟢
tencent	69.84%	🟢
trino-connector	40.29%	🟢

Files

Module	File	Coverage
api	Index.java	0.0%	🔴
catalog-jdbc-clickhouse	ClickHouseTableOperations.java	83.89%	🟢
	ClickHouseConstants.java	0.0%	🔴
catalog-jdbc-doris	DorisUtils.java	95.41%	🟢
core	TagMetadataObjectRelBaseSQLProvider.java	100.0%	🟢
	TagMetadataObjectRelPostgreSQLProvider.java	100.0%	🟢
	FunctionMetaService.java	100.0%	🟢
	TagManager.java	83.68%	🟢
iceberg-rest-server	IcebergTableOperationExecutor.java	100.0%	🟢
	IcebergRESTUtils.java	87.3%	🟢
	FederatedCatalogWrapper.java	87.02%	🟢
	IcebergTableOperations.java	81.11%	🟢
	CatalogWrapperForREST.java	73.11%	🟢

Copilot

Pull request overview

Improves Apache Doris catalog compatibility (Doris 3.0+ SHOW CREATE TABLE output) by making partition parsing more tolerant to whitespace and more robust for LIST partition definitions, with corresponding unit test updates.

Changes:

Relaxed partition header regex to tolerate whitespace and updated partition column splitting/backtick handling.
Added LIST partition assignment parsing that can handle nested parentheses for multi-column LIST values.
Updated/expanded TestDorisUtils cases and migrated assertions to JUnit Jupiter Assertions.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/utils/DorisUtils.java	Updates partition regex and adds LIST partition assignment parsing logic.
catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/utils/TestDorisUtils.java	Expands partition parsing tests (including Doris 3.0+ spacing) and switches to JUnit Jupiter assertions.

jiangxt2 · 2026-06-23T03:14:40Z

Thanks for the review. After checking each comment against the actual PR diff:

The code uses createTableSql.replace('\, '\) (the char overload), not replaceAll("\\n", " "). No regex engine is involved.
The variable is already named fieldNames in the PR — the typo from the original code was fixed.
The headerPattern uses ([^]+)for the backtick-quoted branch, which accepts any non-backtick character. The\w+branch only applies to unquoted identifiers. Tests cover names likep-2024_07andp-2024.08` inside backticks.
The test does assert extracted partition names — listTransform.assignments()[0].name() is checked against "p1", "p-2024_07", `"p beijing", etc.

flaming-archer · 2026-06-24T07:50:19Z

@Copilot Please trigger the CI workflow.

yuqi1129 · 2026-07-09T03:44:39Z

@jiangxt2
Can you take the time to resolve the conflicts?

…s for Doris 1.2.x/3.0.x/4.0.x Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2 · 2026-07-09T08:08:05Z

Rebased onto latest main to resolve conflicts with #11763, and fixed test issues found during local Docker IT runs.

Changes:

Regex: added \s* between type keyword and ( in PARTITION_INFO_PATTERN to match 3.0+ format where SHOW CREATE TABLE outputs a space (e.g. PARTITION BY LIST (col))
Tests: fixed LIST partition tests on 3.0.x/4.0.x — creating a table with empty PARTITION BY LIST(…)() is not recognized as partitioned on 3.0+; tests now pass initial assignments at create time
Tests: fixed 1.2.x partition name compatibility (removed hyphens/dots)
Tests: corrected assertion in multi-column LIST test (partitioning().length is 1, not 2)

Local Docker IT results:

CatalogDorisIT (1.2.x): 19/19 passed
CatalogDoris3xIT (3.0.x): 8/8 passed
CatalogDoris4xIT (4.0.x): 8/8 passed
TestDorisUtils (unit): all passed

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

jiangxt2 · 2026-07-09T09:27:50Z

  private static final Pattern PARTITION_INFO_PATTERN =
-      Pattern.compile("PARTITION BY \\b(LIST|RANGE)\\b\\((.+)\\)");
+      Pattern.compile("PARTITION BY \\b(LIST|RANGE)\\b\\s*\\((.+)\\)");



Updated the PR body to accurately reflect the actual code changes.

The previous description overstated the scope — it included Copilot suggestions (comma/backtick/bracket-depth parsing improvements) that were never implemented. The actual change is a single-line regex fix (\s* in PARTITION_INFO_PATTERN) plus integration test coverage for partition round-trip across all three Doris versions.

…pace tolerance Add test cases for Doris 3.0+ SHOW CREATE TABLE format where LIST/RANGE keywords are followed by a space before the opening parenthesis. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2 · 2026-07-13T07:14:57Z

Hi @yuqi1129, the CI failures are from TestOceanBaseDatabaseOperations and TestOceanBaseTableOperations — the OceanBase container (oceanbase/oceanbase-ce:4.2.1-lts) failed to start, timing out while waiting for the boot success! log message. Is it possible to re-run just this specific test job manually?

…g for Doris 3.0+ format (#11732) (#12008) **Cherry-pick Information:** - Original commit: cf289fb - Target branch: `branch-1.3` - Status: ✅ Clean cherry-pick (no conflicts) Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com> Co-authored-by: StormSpirit <jiangxt2@vip.qq.com>

yuqi1129 requested a review from Copilot June 18, 2026 12:29

Copilot started reviewing on behalf of yuqi1129 June 18, 2026 12:29 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

jiangxt2 force-pushed the feat/doris-partition branch 2 times, most recently from badba44 to b62a75e Compare June 22, 2026 13:16

This was referenced Jul 6, 2026

[Bug] Lance deregisterTable relies on coincidence (all tables external) for correctness #11893

Closed

[#11590] improvement(doris): upgrade type system for Doris 3.0+/4.0.x compatibility #11763

Merged

[apache#11590] test(doris): add partition round-trip integration test…

4285945

…s for Doris 1.2.x/3.0.x/4.0.x Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2 force-pushed the feat/doris-partition branch from e37a713 to 4285945 Compare July 9, 2026 08:05

yuqi1129 requested a review from Copilot July 9, 2026 09:04

Copilot started reviewing on behalf of yuqi1129 July 9, 2026 09:05 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

[apache#11590] test(doris): add unit tests for partition regex whites…

cb5813e

…pace tolerance Add test cases for Doris 3.0+ SHOW CREATE TABLE format where LIST/RANGE keywords are followed by a space before the opening parenthesis. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

yuqi1129 assigned jiangxt2 Jul 13, 2026

yuqi1129 added the branch-1.3 Automatically cherry-pick commit to branch-1.3 label Jul 13, 2026

yuqi1129 approved these changes Jul 13, 2026

View reviewed changes

yuqi1129 merged commit cf289fb into apache:main Jul 13, 2026
38 of 39 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[#11590] fix(doris): fix partition parsing for Doris 3.0+ format - #11732

[#11590] fix(doris): fix partition parsing for Doris 3.0+ format#11732
yuqi1129 merged 2 commits into
apache:mainfrom
jiangxt2:feat/doris-partition

jiangxt2 commented Jun 18, 2026 •

edited by yuqi1129

Loading

Uh oh!

github-actions Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jiangxt2 commented Jun 23, 2026

Uh oh!

flaming-archer commented Jun 24, 2026

Uh oh!

yuqi1129 commented Jul 9, 2026

Uh oh!

jiangxt2 commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

jiangxt2 Jul 9, 2026

Uh oh!

jiangxt2 commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

jiangxt2 commented Jun 18, 2026 • edited by yuqi1129 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Partition Regex Fix (DorisUtils.java)

Integration Tests

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jiangxt2 commented Jun 23, 2026

Uh oh!

flaming-archer commented Jun 24, 2026

Uh oh!

yuqi1129 commented Jul 9, 2026

Uh oh!

jiangxt2 commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

jiangxt2 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

jiangxt2 commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jiangxt2 commented Jun 18, 2026 •

edited by yuqi1129

Loading

Partition Regex Fix (`DorisUtils.java`)

github-actions Bot commented Jun 18, 2026 •

edited

Loading