Repository navigation
Commit b8989cc
authored
fix(webhooks): stop admission-refusal retry loops and per-retry log writes (#8870)
* fix(execution): tag deterministic admission rejections and throttle blocked-run logs
Usage-limit, suspended-account, and missing-billing-account refusals now carry
stable codes, so surfaces can tell a refusal that holds until a person acts
from a transient one.
Unattended surfaces can opt into throttleErrorLogs: each refusal of a workflow
by the same gate records at most one execution error row per 15-minute window
(Redis SET NX, in-process LRU without Redis, fail-open on Redis errors). A sender
resending a refused delivery no longer writes an execution row, trace archive,
and workspace_files row per attempt. A caller-supplied logging session is
always completed.
* fix(webhooks): acknowledge deterministic admission rejections for Telegram and Slack
Telegram resends a non-2xx update until it succeeds or 24 hours pass, and Slack
disables an app's event subscription once most deliveries fail, so answering a
usage-limit refusal with 402 only loops. Providers opt in with
acknowledgeAdmissionRejections and get an empty 200 with an ignored outcome;
transient refusals (rate limit, concurrency, reservation outage) still fail so
the sender retries. Generic webhooks keep the 402. Polling receives the raw
refusal with its code. Webhook preprocessing throttles its error rows.
* fix(webhooks): skip polls for over-limit payers and back off failing sources
The poll orchestrator checks each workspace's payer once per tick and skips its
webhooks while the payer is over its usage limit: nothing is fetched, marked
seen, or counted as a failure, so items deliver once the payer is back under
the limit and triggers are no longer auto-disabled over billing state.
A webhook with consecutive poll failures waits 2^(n-1) minutes (capped at an
hour) before the next fetch, and a source's Retry-After or FLOOD_WAIT_<n> is
persisted and honored. RSS logs a source's 4xx once at warn, and an admission
refusal mid-poll leaves the remaining items unseen.
* fix(webhooks): stop Slack redelivering to deleted trigger paths and log them once
A POST to a path with no webhook keeps its 404 but now carries
x-slack-no-retry, sent unconditionally so it reveals nothing the 404 does not.
The processor and route lines for an unknown path drop to debug, leaving the
route handler's single client-error line.
* fix(telegram): verify webhook deliveries with a per-webhook secret token
Register a generated secret_token with setWebhook, store it as
providerConfig.secretToken, and reject deliveries whose
X-Telegram-Bot-Api-Secret-Token header does not match (401). Webhooks
registered before this change have no stored secret and stay accepted
until their next deploy registers one. A new registration reuses the
active deployment's secret for the same bot so deliveries keep verifying
during cutover. Drops the stale empty User-Agent warning: the proxy
exempts webhook trigger routes from the empty-UA block.
* refactor(webhooks): declare the Telegram admission opt-in on its handler
* fix(execution): only treat billing and account refusals as deterministic
An unreadable usage ledger fails closed as exceeded; it said nothing about the
payer, yet it was tagged USAGE_LIMIT_EXCEEDED and acknowledged-and-dropped for
Telegram and Slack. It now stays untagged and retryable. Reservation headroom
denials clear as in-flight runs settle, so they leave the deterministic set too.
The blocked-run log claim drops its in-process fallback: usage refusals only
happen on hosted billing deployments, which run Redis, and without Redis every
refusal records its row. A Redis failure logs at debug.
* fix(webhooks): keep a fan-out target's retryable failure visible past a dropped refusal
An acknowledged admission refusal counted as an acknowledgment, so a Slack or
path fan-out answered 200 even when another target failed and needed the sender
to retry. Dropped targets (block missing, acknowledged refusal) now answer 200
only when no other target failed.
* fix(telegram): match the active bot through env-var token references
Active rows store the bot token as authored, often a {{VAR}} reference, while
subscription calls receive it resolved, so the comparison never matched: every
deploy minted a fresh secret (a candidate that never activated left the bot
rejected by the active row), and retiring an old version could delete the
webhook the active version still used. Stored tokens are now resolved against
the background webhook env before comparing.
* fix(webhooks): skip polls only after a recorded refusal and back off source failures
The per-tick payer pre-check read billing attribution and the usage ledger for
every polled workspace, including healthy idle ones. A deterministic admission
refusal of a polled event now records the workspace in Redis for five minutes,
and the orchestrator skips only those workspaces in one MGET; healthy payers
cost no billing reads, and billing-disabled deployments skip the mechanism.
Backoff now follows source fetch failures only, tracked in providerConfig and
stamped from the failed poll's start, so transient item-processing refusals
never back a webhook off. Poll state keys are system-managed so deploy change
detection ignores them.
* refactor(webhooks): route every poller's source failures through one backoff
Every poller's outer catch now records a source failure, so a failing Gmail,
Outlook, IMAP, Drive, Sheets, Calendar, or HubSpot source backs off like RSS
instead of only RSS. markWebhookSuccess clears the backoff in its existing
reset write, the window uses the shared jittered backoff, and the orchestrator
asks a boolean isPollBackedOff.
Smaller cleanups: one isDroppedDispatch predicate for the Slack and path
fan-outs, explicit precedence for a polled refusal's code, a typed RSS refusal
error instead of a flag, Telegram resolves only the stored bot token, and
PollOutcome lives with the polling types.
* chore(webhooks): tighten poll comments and backoff tests
Poll outcome docs describe what a skipped poll actually does, source failures
keep logging the full error object as the pollers did before, the backoff table
pins the clock past the poll start so it proves the window is anchored there,
and the RSS rate-limit test drives a Retry-After header.
* fix(webhooks): stop every poller's batch on a deterministic admission refusal
Only RSS stopped at a refused item; Gmail, Outlook, and IMAP advanced their
cursors past refused emails, and every poller counted the refusals toward
auto-disable. A shared PollAdmissionRefusedError now leaves the idempotency
callback, stops the batch, and returns skipped before any cursor update or
failure count; items that already ran replay as idempotent no-ops.
Source backoff goes back to RSS only, where the rate-limited feed was: the other
pollers' fetch helpers do not carry status or Retry-After, so routing their
failures through it would back off on a guess. The block-missing 404 also tells
Slack not to redeliver.
* fix(webhooks): never replay completed poll work or mask a retryable fan-out failure
A poller stops its batch on a deterministic refusal only while nothing in the
batch has completed; once an item has run, the refusal is an ordinary item
failure, so the poller saves its completed work exactly as before and no
completed event can replay after the idempotency window.
In a multi-target delivery a missing block's no-retry 404 no longer stands in
for a target that failed and needs the sender to retry. A source's Retry-After
is counted from its answer rather than the poll's start. The RSS backoff keys
are cleared by RSS's own state write, so other pollers' success path is
unchanged, and two fields nothing reads are dropped.
* fix(execution): drop the unreachable billing-account admission code
A workspace without a billing account fails inside payer resolution and takes
the retryable attribution-error path; the branch that tagged
BILLING_ACCOUNT_REQUIRED only ran for an attribution with no actor, which
system attribution never produces. The branch goes back to its staging form
and the deterministic set keeps the usage limit and suspended accounts.
* chore(webhooks): key blocked-run claims by gate and centralize the polling utils mock
Two gates that fail without a code (a ban lookup error and a usage lookup
error) no longer share one throttle claim, so neither hides the other's row.
The polling utils module gets one central mock in @sim/testing, replacing the
partial importOriginal mocks and the hand-rolled factory in the table trigger
test.
* fix(telegram): match the active bot with the env the caller resolved its token with
Subscription creation resolves the incoming bot token with the deployer's env,
while cleanup resolves with the background env; the active-row matcher now
uses the same env as its caller, so a bot referenced through a personal
variable still reuses the active secret and is not deleted from under the
active deployment.
Tests: the idempotency service gets one central mock in @sim/testing, used by
every test that mocked it locally, and the new tests import single factory and
mock files instead of the @sim/testing barrel.
* chore(testing): stub IdempotencyService.createWebhookIdempotencyKey in the central mock1 parent b5d3f7b commit b8989cc
40 files changed
Lines changed: 1986 additions & 240 deletions
File tree
- apps/sim
- app/api/webhooks/trigger/[path]
- lib
- billing
- core/admission
- execution
- table
- webhooks
- polling
- providers
- workspace-files/application
- packages/testing/src/mocks
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
703 | 703 | | |
704 | 704 | | |
705 | 705 | | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
706 | 797 | | |
707 | 798 | | |
708 | 799 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
126 | 127 | | |
127 | 128 | | |
128 | 129 | | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
129 | 133 | | |
130 | 134 | | |
131 | 135 | | |
132 | | - | |
| 136 | + | |
133 | 137 | | |
134 | 138 | | |
135 | 139 | | |
| |||
201 | 205 | | |
202 | 206 | | |
203 | 207 | | |
204 | | - | |
| 208 | + | |
205 | 209 | | |
206 | 210 | | |
207 | 211 | | |
| |||
261 | 265 | | |
262 | 266 | | |
263 | 267 | | |
264 | | - | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
265 | 271 | | |
266 | 272 | | |
267 | 273 | | |
268 | 274 | | |
269 | 275 | | |
270 | | - | |
271 | | - | |
| 276 | + | |
| 277 | + | |
272 | 278 | | |
273 | 279 | | |
274 | 280 | | |
| |||
333 | 339 | | |
334 | 340 | | |
335 | 341 | | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
336 | 347 | | |
337 | 348 | | |
338 | 349 | | |
339 | 350 | | |
340 | 351 | | |
341 | 352 | | |
342 | | - | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
343 | 358 | | |
344 | 359 | | |
345 | 360 | | |
| |||
352 | 367 | | |
353 | 368 | | |
354 | 369 | | |
355 | | - | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
356 | 374 | | |
357 | 375 | | |
358 | 376 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
1 | 5 | | |
2 | 6 | | |
3 | | - | |
4 | | - | |
5 | | - | |
6 | | - | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
| 7 | + | |
24 | 8 | | |
25 | 9 | | |
26 | 10 | | |
27 | 11 | | |
28 | 12 | | |
29 | 13 | | |
30 | 14 | | |
| 15 | + | |
| 16 | + | |
31 | 17 | | |
32 | 18 | | |
33 | 19 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
0 commit comments