Support refreshing vended storage credentials for REST catalog tables

[zakariya-s]

Is your feature request related to a problem or challenge?

REST catalogs can vend short-lived storage credentials for accessing table data. Iceberg Java supports refreshing these credentials through cloud-specific refresh endpoints before they expire.

iceberg-rust does not currently refresh per-prefix credentials. Consequently, long-running reads, writes, and maintenance operations can fail once the initial S3 or GCS credentials expire.

Refresh support must preserve per-prefix credential isolation, table-scoped catalog authentication, and support tables referencing multiple storage prefixes or clouds.

This issue tracks the work required across the core storage API, OpenDAL backends, and REST catalog. The initial reference implementation was developed and reviewed in #2932.

Describe the solution you'd like

Add refreshable storage credential support across the REST catalog, FileIO, and storage backends:

• Introduce a backend-independent credential-provider interface for FileIO
• Use the standard AWS and GCS refresh endpoint properties returned by REST catalogs
• Cache credentials per cloud and select the longest matching storage prefix

Tasks

• Security and independent fixes

  • Redact secret-bearing catalog and storage configuration.

• Core credential-provider API [1/6] feat: add refreshable storage credential providers #2976

• OpenDAL S3 support and shared dynamic-credential machinery

  • Adapt credentials into reqsign’s AWS provider.

• OpenDAL GCS support

  • Adapt OAuth2 credentials into reqsign’s Google provider.

• REST credential-refresh provider

  • Fetch and parse credentials from catalog-provided endpoints.
  • Cache credentials independently by cloud.
  • Select the longest matching prefix.
  • Implement cloud-compatible prefetch timing, jitter, and failure backoff.

• REST catalog wiring

  • Construct the provider from table properties and attach it to table FileIO.

Follow-up

Azure/ADLS refresh is not part of the initial S3 and GCS implementation. The remaining upstream OpenDAL capability is tracked by apache/opendal#8030. Azure support can follow once that API is merged and released.

Willingness to contribute

I can contribute to this feature independently