Skip to content

fix: batch trigger returns stale failed runs when idempotency key points to a dead run - #4818

Closed
Jaimin2687 wants to merge 1 commit into
triggerdotdev:mainfrom
Jaimin2687:fix/batch-trigger-idempotency-key-status-check
Closed

fix: batch trigger returns stale failed runs when idempotency key points to a dead run#4818
Jaimin2687 wants to merge 1 commit into
triggerdotdev:mainfrom
Jaimin2687:fix/batch-trigger-idempotency-key-status-check

Conversation

@Jaimin2687

Copy link
Copy Markdown

What happened

I noticed that batchTrigger and single trigger behave differently when an idempotency key points to a run that already failed.

With single trigger, if a previous run with the same idempotency key ended up in a terminal failure state (CRASHED, SYSTEM_FAILURE, TIMED_OUT, EXPIRED, COMPLETED_WITH_ERRORS, INTERRUPTED), the SDK correctly clears the key and creates a fresh run. This works because IdempotencyKeyConcern.handleExistingRun calls shouldIdempotencyKeyBeCleared(status) before deciding whether to return a cached result.

With batchTrigger, that check was missing entirely. The batch path in BatchTriggerV3Service.#prepareRunData only checked time-based expiration (idempotencyKeyExpiresAt < now), so a failed run would get returned as isCached: true — silently handing back a dead run that will never produce output.

The root cause turned out to be pretty simple: the SQL query backing findRunsByIdempotencyKeys never selected the status column, and the IdempotencyKeyRunMatch type didn't include it. So even if someone wanted to add the check, the data wasn't there.

What this PR does

Three small changes:

  1. Added status to the query and its return typeIdempotencyKeyRunMatch in run-store/src/types.ts now includes status: string, and the raw SQL in PostgresRunStore.findRunsByIdempotencyKeys selects it.

  2. Added the missing status check in the batch path — After the existing expiry check in #prepareRunData, there's now a shouldIdempotencyKeyBeCleared(cachedRun.status) guard that mirrors what the single-trigger path already does. If the cached run is in a failure state, we add it to expiredRunIds (so the key gets cleared) and mint a new run ID.

  3. Added the importshouldIdempotencyKeyBeCleared was already exported from taskStatus.ts, just not imported in the batch service.

✅ Checklist

  • I have followed every step in the contributing guide
  • The PR title follows the convention
  • I ran and tested the code works

Testing

Unit tests — Added 21 tests in batchTriggerIdempotencyStatusCheck.test.ts that cover every TaskRunStatus value against shouldIdempotencyKeyBeCleared:

  • All 6 failure statuses correctly return true (key should be cleared, run re-triggered)
  • All 11 non-failure statuses correctly return false (run stays cached)
  • Explicit edge cases for COMPLETED_SUCCESSFULLY (valid cache), CANCELED (user-intentional), RETRYING_AFTER_FAILURE (still in progress), and EXPIRED

Integration tests — Extended the existing PostgresRunStore.findRunsByIdempotencyKeys.test.ts with testcontainers:

  • Added status assertions to the existing test (catches future regressions if someone removes the column)
  • New test: creates runs in all 6 failure statuses and verifies the query returns the correct status for each
  • New test: verifies COMPLETED_SUCCESSFULLY status is returned correctly

Build verification:

  • pnpm run typecheck --filter webapp
  • pnpm run build --filter @internal/run-store
  • pnpm run format
  • pnpm run lint:fix ✅ (0 warnings, 0 errors)

How to reproduce the bug manually:

// 1. A task that always fails
export const failingTask = task({
  id: "failing-task",
  run: async () => { throw new Error("boom"); },
});

// 2. Batch trigger with idempotency key
await tasks.batchTrigger("failing-task", [
  { payload: {}, options: { idempotencyKey: "key-1" } },
]);
// Wait for the run to fail...

// 3. Trigger again — before this fix, you get the dead run back
await tasks.batchTrigger("failing-task", [
  { payload: {}, options: { idempotencyKey: "key-1" } },
]);
// BEFORE: isCached: true (dead run returned)
// AFTER:  isCached: false (fresh run created)

Changelog

Batch triggers with idempotency keys now correctly re-trigger when a previous run failed, matching the existing single-trigger behavior. Previously, calling batchTrigger with an idempotency key that pointed to a crashed or failed run would silently return the dead run instead of starting a new one.


Screenshots

N/A — backend-only change, no UI impact.

💯

BatchTriggerV3Service.#prepareRunData only checked time-based expiration
on cached idempotency key lookups, but never checked whether the matched
run was in a terminal failure state. The single-trigger path
(IdempotencyKeyConcern.handleExistingRun) correctly calls
shouldIdempotencyKeyBeCleared(status) and re-triggers in that case.

This meant batchTrigger with an idempotency key that pointed at a
CRASHED, SYSTEM_FAILURE, TIMED_OUT, EXPIRED, COMPLETED_WITH_ERRORS, or
INTERRUPTED run would silently return the dead run as isCached: true,
instead of clearing the key and creating a fresh run.

The root cause was twofold:
- findRunsByIdempotencyKeys SQL query did not SELECT the status column
- IdempotencyKeyRunMatch type did not include status

Fix:
- Add status to IdempotencyKeyRunMatch and the backing SQL query
- Add shouldIdempotencyKeyBeCleared guard in #prepareRunData, mirroring
  the single-trigger path

Tests:
- 21 unit tests covering all TaskRunStatus values
- Integration tests verifying status is returned from the query
- Edge case coverage for COMPLETED_SUCCESSFULLY, CANCELED, and
  RETRYING_AFTER_FAILURE (all correctly remain cached)
@changeset-bot

changeset-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 54a6df6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions

Copy link
Copy Markdown
Contributor

Hi @Jaimin2687, thanks for your interest in contributing!

This project requires that pull request authors are vouched, and you are not in the list of vouched users.

This PR will be closed automatically. See https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md for more details.

@github-actions github-actions Bot closed this Aug 28, 2026
@coderabbitai

coderabbitai Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: fdbc1264-05c9-4084-9181-b6bdc61acdc2

📥 Commits

Reviewing files that changed from the base of the PR and between 2e24c01 and 54a6df6.

📒 Files selected for processing (5)
  • apps/webapp/app/v3/services/batchTriggerV3.server.ts
  • apps/webapp/test/batchTriggerIdempotencyStatusCheck.test.ts
  • internal-packages/run-store/src/PostgresRunStore.findRunsByIdempotencyKeys.test.ts
  • internal-packages/run-store/src/PostgresRunStore.ts
  • internal-packages/run-store/src/types.ts

Walkthrough

The run store now returns task run status with idempotency-key matches. The batch trigger uses shouldIdempotencyKeyBeCleared to re-trigger runs in terminal failure states with new child IDs. Tests cover status persistence, returned statuses, and clearable and non-clearable status decisions.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Warning

⚠️ This pull request shows signs of AI-generated slop (redundant_comments). It has been flagged by CodeRabbit slop detection and should be reviewed carefully.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment on lines +468 to +481
// Mirror the single-trigger path (IdempotencyKeyConcern.handleExistingRun):
// if the cached run is in a terminal failure state (CRASHED, SYSTEM_FAILURE,
// TIMED_OUT, EXPIRED, COMPLETED_WITH_ERRORS, INTERRUPTED), clear the
// idempotency key and re-trigger instead of returning the dead run.
if (shouldIdempotencyKeyBeCleared(cachedRun.status as TaskRunStatus)) {
expiredRunIds.add(cachedRun.friendlyId);

return {
id: await this.mintChildFriendlyId(environment, childAnchor, item.options?.region),
isCached: false,
idempotencyKey: item.options?.idempotencyKey ?? undefined,
taskIdentifier: item.task,
};
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Missing server release note for user-facing fix

This user-facing behavior change to batch triggers ships only server code under apps/webapp and internal-packages, with no .server-changes/ note added. The repository guidelines require a .server-changes/ entry for user-facing server-only changes, so this fix will be absent from release notes.

Prompt for agents
CONTRIBUTING.md and AGENTS.md require a .server-changes/ file for user-facing server-only changes (changes under apps/webapp with no package changes). This PR changes observable batchTrigger behavior (failed runs are now re-triggered instead of returned as a dead cached run) and touches only apps/webapp and internal-packages. Add a markdown file under .server-changes/ (e.g. fix-batch-trigger-stale-failed-runs.md) with frontmatter `area: webapp` and `type: fix`, and a one-line user-facing description. See .server-changes/README.md for the exact format.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant