feat(s3): support IAM role and default credential provider chain for s3:// crawling#174
Open
marevol wants to merge 2 commits into
Open
feat(s3): support IAM role and default credential provider chain for s3:// crawling#174marevol wants to merge 2 commits into
marevol wants to merge 2 commits into
Conversation
…s3:// crawling Make accessKey, secretKey, and endpoint optional in S3Client so s3:// crawling can authenticate via the AWS default credential provider chain (IAM role / instance profile / ECS task role / EKS IRSA / environment / profile) instead of requiring static credentials. - When both accessKey and secretKey are blank, no credentials provider is set and the AWS SDK falls back to the default provider chain. - When endpoint is blank, the endpoint is derived from the region for standard Amazon S3 (virtual-hosted style); endpoint override and path-style access are kept for MinIO / S3-compatible storage. - Setting exactly one of accessKey / secretKey fails fast, since a partial pair is almost always a misconfiguration. Existing configurations that set endpoint plus both keys are behavior-preserving. Adds tests for the blank-credentials, partial- credentials, and no-endpoint init paths.
When no endpoint is configured (standard Amazon S3), enable the AWS SDK cross-region access so buckets in any region are reachable without setting "region" to match. This avoids a 301 PermanentRedirect on every object when the default region does not match the bucket's region. Add an optional "crossRegionAccessEnabled" init parameter (default true) to opt out; it is ignored when an endpoint override is set (MinIO/S3-compatible storage). Move the Docker-free init tests to a new S3ClientInitTest and expand coverage (reverse credential XOR, standard S3 with static keys, cross-region enabled/disabled/ignored), asserting the actual cross-region branch selection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
s3://crawler client (S3Client) previously requiredaccessKey,secretKey, andendpoint, throwingCrawlingAccessExceptionwhen any was blank — before the AWS SDK client was even built. This made it impossible to crawls3://targets using an IAM role / instance profile / ECS task role / EKS IRSA, even though AWS SDK v2 supports the default credential provider chain out of the box.This change makes
accessKey,secretKey, andendpointoptional so IAM-role–based crawling works, while keeping existing static-credential / S3-compatible (MinIO) configurations behavior-identical. The same conditional-credentials pattern is already used by the siblingnet.protocol.s3.Handlerand byGcsClient.Behavior
CrawlingAccessException(fail fast on misconfiguration)endpointis blank, the endpoint is derived fromregion(virtual-hosted style).endpointOverride+forcePathStyle(true)are retained whenendpointis set, for MinIO and other S3-compatible storage.regionstill defaults tous-east-1; for standard Amazon S3 it should be set to the bucket's region (there is no automatic region resolution).DefaultCredentialsProvider(IAM role / instance profile / ECS task role / EKS IRSA / environment / profile).Tests
Adds three
S3ClientTestcases covering init with blank credentials (endpoint set), partial credentials (fail fast), and no endpoint / no credentials.builder.build()performs no network I/O and the default credential provider resolves lazily, so these run offline. FullS3ClientTestpasses (10 tests).Note: real IMDS / IAM-role authentication cannot be exercised in CI; the tests verify that client construction succeeds or fails fast, relying on the AWS SDK's documented default-credential-chain behavior.
Notes / follow-ups (not in this PR)
storage://client (StorageClient, MinIO SDK) has the same mandatory-credentials limitation and could get equivalent optional-credential support (MinIO'sIamAwsProvider).fess-docsfile-crawling config guide) could document the IAM-role usage.