feat(low-code): add RateLimitedMultipleTokenAuthenticator component

[Daryna Ishchenko (darynaishchenko)]

Summary

Adds a new declarative authenticator, RateLimitedMultipleTokenAuthenticator, that generalizes source-github's MultipleTokenAuthenticatorWithRateLimiter so it can be reused by any connector during declarative migrations (context: https://github.com/airbytehq/airbyte-internal-issues/issues/16512).

Behavior (per outgoing request, via AuthBase.__call__):

• Requests are classified into named quota pools (quotas) via HttpRequestRegexMatchers; a pool with no matchers is the default (generalizes GitHub's hardcoded "graphql" in path REST-vs-GraphQL split).
• Each token tracks a local per-pool counter, seeded once per token from quota_status_source (e.g. GitHub GET /rate_limit) using remaining_path/reset_path/limit_path dpath extraction. Seeding is lazy (first request) and repeated after an exhaustion wait.
• Active token's counter is decremented; on exhaustion the authenticator rotates through tokens; when all are drained it sleeps until the earliest reset (heartbeat logs, floored at MIN_EXHAUSTION_WAIT = 5s so stale reset timestamps can't busy-loop the quota endpoint, bounded by max_wait_time, else raises transient_error).
• Proactive throttling: once every token drops below max(budget_min_reserve, budget_reserve_fraction * limit), delays of seconds_to_reset / total_remaining (capped at 10s) are injected for graceful degradation instead of a hard wall.

Thread safety (new vs. the source-github implementation, required for the concurrent CDK):

• Token selection + decrement happen atomically under an RLock; _acquire_call returns the token chosen under the lock and the request header is built from that value, so a request is always signed with the token whose counter was decremented (no TOCTOU with concurrent rotation). All sleeps occur outside the lock.
• Double-checked re-seed via a separate _refresh_lock so only one thread refreshes counters after an exhaustion wait.
• ModelToComponentFactory.create_rate_limited_multiple_token_authenticator caches instances keyed on the model definition, so all streams share one counter set (same idea as the shared api_budget).

Example manifest usage (source-github):

authenticator:
  type: RateLimitedMultipleTokenAuthenticator
  tokens: "{{ config['credentials']['personal_access_token'] }}"  # comma-separated PATs, or a list
  auth_method: "token"
  quota_status_source:
    type: QuotaStatusSource
    url: "{{ config.get('api_url', 'https://api.github.com') }}/rate_limit"
  quotas:
    - type: TokenQuota
      name: rest
      remaining_path: ["resources", "core", "remaining"]
      reset_path: ["resources", "core", "reset"]
      limit_path: ["resources", "core", "limit"]
    - type: TokenQuota
      name: graphql
      remaining_path: ["resources", "graphql", "remaining"]
      reset_path: ["resources", "graphql", "reset"]
      matchers:
        - type: HttpRequestRegexMatcher
          url_path_pattern: "/graphql"

Changes:

• declarative_component_schema.yaml: new RateLimitedMultipleTokenAuthenticator, QuotaStatusSource, TokenQuota definitions; added to HttpRequester.authenticator and SelectiveAuthenticator.authenticators unions (models regenerated; new classes applied on top of the checked-in file to avoid unrelated codegen churn).
• airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py: runtime implementation.
• model_to_component_factory.py: create_rate_limited_multiple_token_authenticator with token-string splitting, interpolation, and shared-instance caching.
• Unit tests: seeding/header injection, quota matching, rotation, exhaustion wait/raise, budget throttling, thread safety (no lost decrements across 200 threads; signed token matches decremented token under concurrent rotation), factory token parsing and instance sharing.

Requested by Daryna Ishchenko (darynaishchenko).

Link to Devin session: https://app.devin.ai/sessions/3ced8f6a11d44ed4abc517221294f767
Requested by: Daryna Ishchenko (@darynaishchenko)

Summary by CodeRabbit

• New Features
  • Added a declarative authenticator that rotates across multiple tokens based on quota availability.
  • Added configurable quota pools, request matching, proactive throttling, automatic quota refresh, and reset-aware waiting.
  • Enabled the authenticator in declarative HTTP requester and selective authentication configurations.
  • Added support for configuring tokens, quota status endpoints, headers, and throttling limits.
• Bug Fixes
  • Improved enum handling when generating authentication specifications.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1783347505-rate-limited-multiple-token-authenticator#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1783347505-rate-limited-multiple-token-authenticator

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 150 tests  +23   4 138 ✅ +23   6m 31s ⏱️ - 1m 10s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit dd2b20c. ± Comparison against base commit dd9558c.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

4 153 tests  +23   4 141 ✅ +23   11m 59s ⏱️ + 4m 44s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit dd2b20c. ± Comparison against base commit dd9558c.

♻️ This comment has been updated with latest results.

[coderabbitai]

Warning

Review limit reached

@devin-ai-integration[bot], you've reached your PR review limit, so we couldn't start this review.

Next review available in: 50 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 665a725d-cb4e-4245-82c6-8726291d3288

📥 Commits

Reviewing files that changed from the base of the PR and between 891fb08 and dd2b20c.

📒 Files selected for processing (7)

• airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py
• airbyte_cdk/sources/declarative/declarative_component_schema.yaml
• airbyte_cdk/sources/declarative/models/declarative_component_schema.py
• airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
• airbyte_cdk/sources/declarative/spec/spec.py
• unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py
• unit_tests/sources/declarative/spec/test_spec.py

📝 Walkthrough

Walkthrough

Adds a declarative authenticator that rotates multiple tokens using per-request quota pools, refreshes quota state, waits for resets, and throttles proactively. Declarative schemas, factory construction and caching, public exports, enum serialization, and comprehensive runtime and integration tests are updated.

Changes

Rate-limited multi-token authentication

Layer / File(s)	Summary
Authenticator contracts and declarative registration airbyte_cdk/sources/declarative/auth/__init__.py, airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py, airbyte_cdk/sources/declarative/declarative_component_schema.yaml, airbyte_cdk/sources/declarative/models/declarative_component_schema.py	Defines quota and authenticator models, exposes the new authenticator, and enables it in selective and HTTP requester authentication configurations.
Quota-aware token rotation airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py, unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py	Adds quota seeding, matcher-based pool selection, synchronized token rotation, exhaustion waits, quota refreshes, proactive throttling, and concurrency/configuration tests.
Declarative factory construction and reuse airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py, unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py	Builds the authenticator from declarative models, interpolates settings, supports token lists or delimited strings, and reuses matching authenticator instances through a factory cache.

Advanced authentication spec serialization

Layer / File(s)	Summary
Defensive advanced-auth enum conversion airbyte_cdk/sources/declarative/spec/spec.py	Converts advanced-auth flow and OAuth scope strategy values only when they are enum instances.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Request as PreparedRequest
  participant Auth as RateLimitedMultipleTokenAuthenticator
  participant Quota as quota-status endpoint
  Request->>Auth: submit request for authentication
  Auth->>Quota: fetch quota state for tokens
  Quota-->>Auth: return remaining and reset values
  Auth->>Auth: match quota pool and reserve token
  Auth-->>Request: attach authorization header

Loading

Suggested reviewers: devin-ai-integration[bot]

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main addition of the new RateLimitedMultipleTokenAuthenticator component.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch devin/1783347505-rate-limited-multiple-token-authenticator

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

airbyte_cdk/sources/declarative/spec/spec.py (1)

58-69: 🗄️ Data Integrity & Integration | 🔵 Trivial | 🏗️ Heavy lift

Could we normalize a serialized copy instead of mutating the typed model?

generate_spec() assigns .value strings back into Enum-backed fields, so after the first call self.advanced_auth no longer conforms to its declared model contract. This fixes repeated serialization, but can break callers that reuse advanced_auth and expect AuthFlowType/ScopesJoinStrategy instances. Could you build the output dictionary first and normalize that copy instead, wdyt?

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@airbyte_cdk/sources/declarative/spec/spec.py` around lines 58 - 69, The
generate_spec() serialization flow currently mutates Enum-backed fields on
self.advanced_auth and its nested oauth configuration. Build or obtain a
serialized output copy first, then normalize auth_flow_type and
scopes_join_strategy on that copy only, preserving the typed AuthFlow and OAuth
connector input models for repeated callers.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py`:
- Around line 165-205: The _acquire_call exhaustion path currently applies
_max_wait_time to each sleep rather than the full retry loop. Track a single
exhaustion deadline across repeated reset/refresh iterations, bound each wait by
the remaining time, and raise the existing transient AirbyteTracedException when
that cumulative budget is exhausted; add a regression test covering repeated
zero-quota refreshes.

In `@airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py`:
- Around line 725-726: The shared authenticator registry must be reused by
nested ModelToComponentFactory instances so parent and child streams share
rate-limit counters. Update the parent-stream factory creation paths around
ModelToComponentFactory and the related factory construction at 4514-4526 to
accept and propagate the parent’s _rate_limited_authenticators mapping, rather
than initializing a fresh registry; preserve the existing local initialization
only for root factories.

In `@unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py`:
- Around line 186-211: Update
test_thread_safety_header_token_matches_decremented_token to capture exceptions
raised by each worker in make_call, using the existing thread coordination or an
equivalent mechanism, and assert that the collected exception list is empty
after all threads join. Keep the signed token collection and per-token
remaining-quota assertions unchanged.

---

Nitpick comments:
In `@airbyte_cdk/sources/declarative/spec/spec.py`:
- Around line 58-69: The generate_spec() serialization flow currently mutates
Enum-backed fields on self.advanced_auth and its nested oauth configuration.
Build or obtain a serialized output copy first, then normalize auth_flow_type
and scopes_join_strategy on that copy only, preserving the typed AuthFlow and
OAuth connector input models for repeated callers.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 53a02d6d-190c-4424-9f9f-d4b367840869

📥 Commits

Reviewing files that changed from the base of the PR and between dd9558c and 891fb08.

📒 Files selected for processing (7)

• airbyte_cdk/sources/declarative/auth/__init__.py
• airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py
• airbyte_cdk/sources/declarative/declarative_component_schema.yaml
• airbyte_cdk/sources/declarative/models/declarative_component_schema.py
• airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
• airbyte_cdk/sources/declarative/spec/spec.py
• unit_tests/sources/declarative/auth/test_rate_limited_multiple_token.py

[Anatolii Yatsuk (tolik0)]

Deep review with local reproductions (ran the PR's tests, drove it end-to-end through ConcurrentDeclarativeSource manifests, replicated the model codegen, and compared behavior against source-github's MultipleTokenAuthenticatorWithRateLimiter). The core machinery works and thread safety is done correctly, but two must-fix items remain — see inline comments:

1. High: the shared-instance cache key is poisoned by $parameters propagation, so streams get separate counter sets in realistic manifests (reproduced on this head).
2. Medium: the checked-in generated models file is out of sync with the schema (hand-edited, not regenerated).

Also a medium on max_wait_time not being a cumulative bound (reproduced infinite reseed loop), plus several low-severity robustness notes. The substream-factory registry sharing added in d92aaac was verified working with a parent/child manifest.