[#11748] feat(catalog-hadoop): support Tencent Cloud COS as fileset storage backend (Java side) by whua3 · Pull Request #11713 · apache/gravitino

whua3 · 2026-06-17T08:52:57Z

What changes were proposed in this pull request?

Add Tencent Cloud Object Storage (COS) support for the fileset catalog. The layout follows the existing OSS / S3 / GCS / Azure modules.

bundles/tencent: source module containing
- COSFileSystemProvider: builds a Hadoop CosNFileSystem for the cosn:// scheme.
- COSCredentialsProvider: Hadoop-side provider that bridges Gravitino-vended COSSecretKey credentials to hadoop-cos.
- COSSecretKeyProvider: server-side CredentialProvider that returns long-lived COS Secret Key credentials.
- COSUtils: helper that maps Gravitino properties to Hadoop Configuration.
bundles/tencent-bundle: shaded fat-jar (gravitino-tencent-bundle-<version>.jar) that bundles hadoop-cos and cos_api-bundle with relocated packages.
api: new COSSecretKeyCredential type.
catalogs/catalog-common: COSProperties and COSCredentialConfig.
clients/filesystem-hadoop3-runtime: include bundles/tencent in the GVFS runtime jar.
docs: new fileset-catalog-with-cos.md and a cross-link from fileset-catalog.md.
Tests: TestCOSFileSystemProvider, TestCOSCredentialProvider, and FilesetCOSCatalogIT. The IT is skipped on CI when no real COS credentials are provided, the same way FilesetOSSCatalogIT is.

Why are the changes needed?

#6490 tracks adding Tencent COS support. Without it, users on Tencent Cloud have to either rely on S3-compatible workarounds, which do not work with Gravitino's credential vending, or maintain their own fork.

This PR is the first of a chain of PRs against main:

PR-A (this PR): Java side, with the static cos-secret-key credential.
PR-B: server-side STS credential vending (cos-token).
PR-C: Python GVFS support (COSStorageHandler and Python-side COSSecretKeyCredential).
PR-D: Python COSTokenCredential wiring. Can be folded into PR-C if reviewers prefer fewer PRs.

Part of #6490
Fix: #11748

Does this PR introduce any user-facing change?

Yes, additive only. Existing fileset / catalog behavior is unchanged. New surface:

New fileset location scheme: cosn://<bucket>/<path>.
New fileset catalog properties: cos-region, cos-endpoint (optional, defaults to cos.${region}.myqcloud.com), cos-access-key-id, cos-secret-access-key.
New credential provider value: credential-providers=cos-secret-key (static AK/SK; STS will follow in PR-B).
New deployable artifact: gravitino-tencent-bundle-<version>.jar.
New documentation page: Fileset Catalog with COS.

How was this patch tested?

Unit tests: :bundles:tencent:test, :catalogs:catalog-common:test, :catalogs:catalog-fileset:test, :api:test all pass locally.
Build: :bundles:tencent-bundle:shadowJar, :clients:filesystem-hadoop3-runtime:shadowJar, and assemble for all touched modules pass locally.
Spotless and Checkstyle pass.
FilesetCOSCatalogIT passes locally against a real COS bucket in ap-guangzhou. It is skipped on CI when no credentials are provided.
Manual end-to-end verification on a local Gravitino server with the new bundle jars: created a fileset catalog backed by cosn://, then exercised hadoop fs -ls, -put, -get, -cat, -rm over gvfs://fileset/..., plus schema/fileset CRUD via the REST API and direct cosn://... access.

github-actions · 2026-06-17T10:36:11Z

Code Coverage Report

Overall Project	67.36% `-0.09%`	🟢
Files changed	42.86%	🔴

Module	Coverage
aliyun	1.72%	🔴
api	46.52% `-0.3%`	🟢
authorization-common	85.96%	🟢
aws	26.5% `+15.17%`	🔴
azure	2.47%	🔴
catalog-common	9.92% `-0.47%`	🔴
catalog-fileset	80.23%	🟢
catalog-glue	66.91%	🟢
catalog-hive	79.42%	🟢
catalog-jdbc-clickhouse	80.2%	🟢
catalog-jdbc-common	44.22%	🟢
catalog-jdbc-doris	80.28%	🟢
catalog-jdbc-hologres	54.03%	🟢
catalog-jdbc-mysql	79.23%	🟢
catalog-jdbc-oceanbase	80.91%	🟢
catalog-jdbc-postgresql	82.29%	🟢
catalog-jdbc-starrocks	78.51%	🟢
catalog-kafka	77.01%	🟢
catalog-lakehouse-generic	58.53%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	85.94%	🟢
catalog-lakehouse-paimon	84.25%	🟢
catalog-model	77.72%	🟢
cli	44.51%	🟢
client-java	78.01%	🟢
common	50.17%	🟢
core	82.59%	🟢
filesystem-hadoop3	77.3%	🟢
flink	0.0%	🔴
flink-common	47.09%	🟢
flink-runtime	0.0%	🔴
gcp	14.12%	🔴
hadoop-auth	66.67%	🟢
hadoop-common	12.7% `-0.05%`	🔴
hive-metastore-common	53.29%	🟢
iceberg-common	58.3%	🟢
iceberg-rest-server	73.94%	🟢
idp-basic	85.71%	🟢
integration-test-common	0.0%	🔴
jobs	66.17%	🟢
lance-common	20.81%	🔴
lance-rest-server	64.84%	🟢
lineage	53.02%	🟢
optimizer	83.17%	🟢
optimizer-api	21.95%	🔴
server	85.96%	🟢
server-common	74.62%	🟢
spark	28.57%	🔴
spark-common	46.01%	🟢
tencent	69.84%	🟢
trino-connector	40.29%	🟢

Files

Module	File	Coverage
api	COSSecretKeyCredential.java	0.0%	🔴
aws	WebIdentityTokenSources.java	82.86%	🟢
	AwsIrsaCredentialGenerator.java	37.13%	🔴
catalog-common	COSCredentialConfig.java	0.0%	🔴
	COSProperties.java	0.0%	🔴
hadoop-common	Constants.java	0.0%	🔴
tencent	COSUtils.java	100.0%	🟢
	COSSecretKeyProvider.java	88.89%	🟢
	COSCredentialsProvider.java	80.77%	🟢
	COSFileSystemProvider.java	45.83%	🔴

Copilot

Pull request overview

Adds Tencent Cloud Object Storage (COS) support to the Fileset catalog/GVFS stack, following the existing cloud-provider bundle pattern (AWS/GCP/Aliyun/Azure). This introduces COS-specific filesystem + credential vending integration, a shaded “tencent-bundle” artifact, and accompanying docs/tests.

Changes:

Introduce new COS bundle modules (:bundles:tencent, :bundles:tencent-bundle) and wire them into build/runtime packaging.
Add COS credential type + server/client-side credential provider plumbing for static SecretId/SecretKey vending.
Document COS setup and add COS integration/unit tests.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
settings.gradle.kts	Includes new Tencent bundle modules in the Gradle build.
gradle/libs.versions.toml	Adds `hadoop-cos` version and `hadoop3-cos` dependency coordinate.
docs/fileset-catalog.md	Mentions COS support and bundle jar name.
docs/fileset-catalog-with-cos.md	New end-to-end documentation for COS-backed Fileset catalogs (incl. credential vending).
clients/filesystem-hadoop3-runtime/build.gradle.kts	Pulls `:bundles:tencent` into the Hadoop3 runtime client jar.
catalogs/hadoop-common/src/main/java/org/apache/gravitino/catalog/hadoop/fs/Constants.java	Adds COS-specific Hadoop configuration keys.
catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/integration/test/FilesetCOSCatalogIT.java	Adds COS integration test (gated by env vars).
catalogs/catalog-fileset/build.gradle.kts	Adds `tencent-bundle` shadow jar to test classpath and task deps.
catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/COSProperties.java	Defines COS catalog property keys (region/endpoint/AK/SK).
catalogs/catalog-common/src/main/java/org/apache/gravitino/credential/config/COSCredentialConfig.java	Adds COS credential config wrapper for static AK/SK.
bundles/tencent/src/test/java/org/apache/gravitino/cos/fs/TestCOSFileSystemProvider.java	Unit tests for COS FS provider mapping and credential conf injection.
bundles/tencent/src/test/java/org/apache/gravitino/cos/credential/TestCOSCredentialProvider.java	Unit tests for COS credential provider behavior.
bundles/tencent/src/main/resources/META-INF/services/org.apache.gravitino.credential.CredentialProvider	Registers COS credential provider via ServiceLoader.
bundles/tencent/src/main/resources/META-INF/services/org.apache.gravitino.catalog.hadoop.fs.FileSystemProvider	Registers COS filesystem provider via ServiceLoader.
bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSUtils.java	Helper for selecting a suitable COS credential from vended credentials.
bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSFileSystemProvider.java	Implements COS `cosn://` filesystem provider + default tuning.
bundles/tencent/src/main/java/org/apache/gravitino/cos/fs/COSCredentialsProvider.java	Bridges Gravitino vended COS credentials into `hadoop-cos` auth.
bundles/tencent/src/main/java/org/apache/gravitino/cos/credential/COSSecretKeyProvider.java	Server-side credential provider for static COS AK/SK.
bundles/tencent/build.gradle.kts	Build config for the thin COS integration module.
bundles/tencent-bundle/build.gradle.kts	Shaded fat-jar bundling `hadoop-cos` + dependencies with relocations.
api/src/main/java/org/apache/gravitino/credential/COSSecretKeyCredential.java	New credential type for COS static SecretId/SecretKey.

yuqi1129

Thanks for the COS support — the layout cleanly mirrors the existing OSS/S3/GCS bundles, and the ServiceLoader registration, relocation rules and credential-vending wiring all look correct. A few inline comments below.

One general note (not tied to a line): the PR description mentions "wire cosn:// into FilesetCatalogPropertiesMetadata" and "register the cosn scheme in hadoop-common", but neither file is actually changed that way — providers are auto-loaded via META-INF/services, and hadoop-common only got the two timeout/retry constants. The change itself is fine; just the description is slightly off.

whua3 · 2026-06-18T16:57:27Z

Pushed an updated commit addressing all review comments.

jerryshao · 2026-06-19T16:11:25Z

I suggest you to create subtasks for this epic issue #6490 . And each PR should have a related issue for that dedicated thing. Not tracking all the PRs in one issue.

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 4 comments.

Add Tencent Cloud Object Storage (COS) support for the fileset catalog, following the same layout as the existing OSS / S3 / GCS / Azure modules. This is the first PR in a chain that together addresses apache#6490. Subsequent PRs will add server-side STS credential vending, Python GVFS support, and Python-side STS wiring. The issue will remain open until the final PR in the chain is merged. Modules added or changed: - bundles/tencent: FileSystemProvider for the cosn:// scheme and a CredentialProvider that returns long-lived COS Secret Key credentials. - bundles/tencent-bundle: shaded fat-jar that bundles hadoop-cos and the COS Java SDK with relocated packages. - api: COSSecretKeyCredential type, registered via META-INF/services so that the GVFS client can deserialize credentials returned by the server. - catalogs/catalog-common: COSProperties and COSCredentialConfig. - catalogs/catalog-fileset: make cosn a recognized filesystem provider for the fileset catalog (loaded via the FilesystemProvider SPI). - clients/filesystem-hadoop3-runtime: include bundles/tencent in the GVFS runtime jar so cosn:// works out of the box. - clients/filesystem-hadoop3: add tencent-bundle as a test dependency for the new GVFS COS integration test below. - docs: new fileset-catalog-with-cos.md, plus a cross-link from fileset-catalog.md. Tests: - TestCOSFileSystemProvider and TestCOSCredentialProvider as unit tests. - FilesetCOSCatalogIT as an integration test for the catalog layer. - GravitinoVirtualFileSystemCOSIT as an integration test for the GVFS client-side path against a real COS bucket. All ITs are skipped when COS_* environment variables are not provided, following the same pattern as FilesetOSSCatalogIT / GravitinoVirtualFileSystemOSSIT. Part of apache#6490

whua3 · 2026-06-28T11:04:31Z

there are screenshots of the manual tests.

whua3 · 2026-06-28T11:23:53Z

Additionally, I encountered a problem in the testing phase, so I created an independent PR(#11814) to fix it.

yuqi1129 · 2026-06-28T13:32:54Z

Additionally, I encountered a problem in the testing phase, so I created an independent PR(#11814) to fix it.

Good catch, thanks for your hard work!

…p-catalog # Conflicts: # common/src/main/java/org/apache/gravitino/config/ConfigConstants.java

yuqi1129

A minor one, others LGTM

whua3 · 2026-07-02T04:00:26Z

A minor one, others LGTM

done~

yuqi1129

LGTM

yuqi1129 · 2026-07-02T12:33:37Z

@whua3
Before merging it, I want to make sure that Python GVFS for COS is not supported currently.

whua3 · 2026-07-02T12:43:03Z

@whua3 Before merging it, I want to make sure that Python GVFS for COS is not supported currently.

Yes, Python GVFS does not support COS yet in this PR. I will implement this in PR-C.

yuqi1129 · 2026-07-02T12:48:38Z

@whua3 Before merging it, I want to make sure that Python GVFS for COS is not supported currently.

Yes, Python GVFS does not support COS yet in this PR. I will implement this in PR-C.

I see. Please correct your PR title and modify the issue link. #6490 is the EPIC link, we may seldom use it in the PR.

…ersion to 2.0.0 Two review-feedback follow-ups on the COS fileset catalog work: 1. docs: pad each row of the three property tables in docs/fileset-catalog-with-cos.md to the width of the longest cell (`cos-endpoint`) so that the pipe characters line up in a raw editor. Rendered output is unchanged. 2. version: bump the "Since version" of all newly introduced COS entry points from 1.4.0 to 2.0.0 to match the current release train (main is on 2.0.0-SNAPSHOT): - ConfigConstants.java: replace VERSION_1_4_0 with VERSION_2_0_0 (the constant is only referenced from the new COS credential config) - COSCredentialConfig.java: switch three .version(...) declarations to VERSION_2_0_0 - fileset-catalog-with-cos.md: update every "Since version" table cell and the credential-vending prose to 2.0.0 - fileset-catalog.md: update the "Since 2.0.0" note referencing COS support Part of apache#6490

whua3 · 2026-07-02T14:01:52Z

@whua3 Before merging it, I want to make sure that Python GVFS for COS is not supported currently.

Yes, Python GVFS does not support COS yet in this PR. I will implement this in PR-C.

I see. Please correct your PR title and modify the issue link. #6490 is the EPIC link, we may seldom use it in the PR.

done

yuqi1129 · 2026-07-03T01:15:10Z

@diqiu50
Would you like to take a look?

yuqi1129 · 2026-07-03T06:14:43Z

I'm going to merge this PR, @whua3. Thanks for your hard work. I'm looking forward to more PRs

whua3 · 2026-07-03T07:30:01Z

I'm going to merge this PR, @whua3. Thanks for your hard work. I'm looking forward to more PRs

@yuqi1129 Thanks for reviewing my PRs and providing lots of valuable suggestions.

whua3 mentioned this pull request Jun 17, 2026

[EPIC] Support COS(Tencent cloud storage) for Hadoop catalog #6490

Open

2 tasks

jerryshao requested a review from yuqi1129 June 17, 2026 16:27

yuqi1129 requested a review from Copilot June 18, 2026 02:50

Copilot started reviewing on behalf of yuqi1129 June 18, 2026 02:50 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread docs/fileset-catalog.md Outdated

Comment thread docs/fileset-catalog-with-cos.md

Comment thread docs/fileset-catalog-with-cos.md Outdated

yuqi1129 reviewed Jun 18, 2026

View reviewed changes

whua3 force-pushed the feat/6490-cos-hadoop-catalog branch from 86af744 to 451a93d Compare June 18, 2026 16:56

whua3 mentioned this pull request Jun 20, 2026

[Subtask] Support Tencent Cloud COS as fileset storage backend (Java side) #11748

Closed

yuqi1129 reviewed Jun 22, 2026

View reviewed changes

yuqi1129 requested a review from Copilot June 22, 2026 14:02

Copilot started reviewing on behalf of yuqi1129 June 22, 2026 14:04 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

whua3 force-pushed the feat/6490-cos-hadoop-catalog branch from 451a93d to ffce209 Compare June 28, 2026 10:53

Merge remote-tracking branch 'upstream/main' into feat/6490-cos-hadoo…

17456e5

…p-catalog # Conflicts: # common/src/main/java/org/apache/gravitino/config/ConfigConstants.java

yuqi1129 reviewed Jul 2, 2026

View reviewed changes

Comment thread docs/fileset-catalog-with-cos.md Outdated

Comment thread docs/fileset-catalog-with-cos.md Outdated

whua3 force-pushed the feat/6490-cos-hadoop-catalog branch from b974a89 to 15460ef Compare July 2, 2026 04:06

yuqi1129 previously approved these changes Jul 2, 2026

View reviewed changes

yuqi1129 reviewed Jul 2, 2026

View reviewed changes

Comment thread common/src/main/java/org/apache/gravitino/config/ConfigConstants.java Outdated

whua3 dismissed yuqi1129’s stale review via 9c75eab July 2, 2026 13:11

whua3 force-pushed the feat/6490-cos-hadoop-catalog branch from 15460ef to 9c75eab Compare July 2, 2026 13:11

whua3 changed the title ~~[#6490] feat(fileset): Add Tencent COS support for fileset catalog~~ [#11748] feat(fileset): Add Tencent COS support for fileset catalog Jul 2, 2026

whua3 changed the title ~~[#11748] feat(fileset): Add Tencent COS support for fileset catalog~~ [#11748] feat(catalog-hadoop): support Tencent Cloud COS as fileset storage backend (Java side) Jul 2, 2026

yuqi1129 assigned whua3 Jul 2, 2026

yuqi1129 approved these changes Jul 3, 2026

View reviewed changes

yuqi1129 merged commit 69edf53 into apache:main Jul 3, 2026
35 checks passed

Uh oh!

Conversation

whua3 commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

github-actions Bot commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuqi1129 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

whua3 commented Jun 18, 2026

Uh oh!

jerryshao commented Jun 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

whua3 commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

whua3 commented Jun 28, 2026

Uh oh!

yuqi1129 commented Jun 28, 2026

Uh oh!

yuqi1129 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

whua3 commented Jul 2, 2026

Uh oh!

yuqi1129 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yuqi1129 commented Jul 2, 2026

Uh oh!

whua3 commented Jul 2, 2026

Uh oh!

yuqi1129 commented Jul 2, 2026

Uh oh!

whua3 commented Jul 2, 2026

Uh oh!

yuqi1129 commented Jul 3, 2026

Uh oh!

yuqi1129 commented Jul 3, 2026

Uh oh!

whua3 commented Jul 3, 2026

whua3 commented Jun 17, 2026 •

edited

Loading

github-actions Bot commented Jun 17, 2026 •

edited

Loading

whua3 commented Jun 28, 2026 •

edited

Loading