Skip to content

feat(webapp): deployment lifecycle telemetry events - #4778

Merged
myftija merged 13 commits into
mainfrom
feature/tri-13477-deployment-lifecycle-telemetry
Aug 26, 2026
Merged

feat(webapp): deployment lifecycle telemetry events#4778
myftija merged 13 commits into
mainfrom
feature/tri-13477-deployment-lifecycle-telemetry

Conversation

@myftija

@myftija myftija commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Deployments currently leave little analytical trace. This PR makes every deployment emit two analytics events to enable useful queries. It also enables comparing deployments across build paths, CLI versions, runtimes, and orgs.

Where the events come from

 trigger deploy
      │
      ▼
  initialize ─────────────────────────────▶ ✨ deployment.initialized
      │ createdAt
      ▼
   PENDING      waiting for a build slot        ┐
      │ startedAt                               │ queue time
      ▼                                         ┘
  INSTALLING    build server installs deps      ┐
      │ installedAt      (native paths only)    │ install time
      ▼                                         ┘
   BUILDING     the image is built              ┐
      │ builtAt                                 │ building time
      ▼                                         ┘
  DEPLOYING     indexing + registry push        ┐
      │ deployedAt / failedAt / canceledAt      │ deploying time
      ▼                                         ┘
  DEPLOYED · FAILED · TIMED_OUT · CANCELED
      │
      └───────────────────────────────────▶ ✨ deployment.finished

deployment.finished fires exactly once, whichever way the deployment ends, and is backdated to cover the deployment's real lifetime. Not every path visits every state (Depot deploys skip PENDING/INSTALLING, for example) — a phase duration is simply omitted when its state was never entered.

What each event carries

  • Which path built it: depot, native, or native_local_bundle
  • How it ended: status, plus an error class and message when it failed
  • How long each phase took: queue, install, building, deploying, and total — derived from the timestamps above
  • Who and with what: org, project, environment, runtime, CLI version, and how the deploy was triggered (CLI, GitHub, Vercel)

With that, one query gives failure rate per build path, duration percentiles per phase, adoption per CLI version, or a per-org health table.

Fixes that ride along

  • The old deployment.outcome span was silently dropped ~95% of the time (it was subject to trace sampling). The new events opt out of sampling explicitly, so every deployment is counted.
  • The fail/timeout/finalize transitions were racy: a late timeout could overwrite a successful deployment. They now use guarded writes, so exactly one caller wins the terminal transition — and exactly one event is emitted.
  • Canceled deployments previously recorded nothing; they do now.
  • The deployment's CLI version is now stored at initialization (new nullable column), so even deploys that fail early are attributable to a CLI release.
  • Telemetry is flushed on shutdown (the last batch used to be lost on every webapp deploy), and an optional second exporter can mirror just these events into a dedicated dataset.

Replaces the deployment.outcome span with a wide deployment.lifecycle
event emitted once per terminal transition (DEPLOYED/FAILED/TIMED_OUT/
CANCELED), backdated createdAt-to-terminal, carrying build path (depot/
native/local_bundle), per-phase durations derived from the persisted
timestamp chain, error class, org/project/env, runtime, CLI version and
trigger source as attributes. A zero-duration deployment.initialized
event at creation provides the funnel denominator for stuck-deployment
detection.

Events are emitted on ROOT_CONTEXT with the forceRecording attribute:
the previous span was started under the ambient request context, where
the parent-based sampler drops ~95% of traffic before the force-record
check runs. SEMINTATTRS_FORCE_RECORDING is now exported for this.

The fail, timeout and finalize transitions now use guarded updateMany
writes so exactly one caller commits a terminal status and emits the
event; this also stops a late timeout from overwriting DEPLOYED. The
cancel path now emits too (it previously recorded nothing).

Also: cliVersion is stamped onto WorkerDeployment at initialization from
the x-trigger-cli-version header (previously only available post-index
via BackgroundWorker, i.e. null for pre-index failures); an optional
second OTLP exporter (INTERNAL_OTEL_DEPLOYMENT_EVENT_EXPORTER_URL)
mirrors deployment.* spans into a dedicated dataset; the tracer provider
is flushed on SIGTERM/SIGINT so shutdowns stop dropping the last batch.
@changeset-bot

changeset-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 104bcc9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 27 packages
Name Type
trigger.dev Patch
@internal/dashboard-agent Patch
@trigger.dev/build Patch
@trigger.dev/core Patch
@trigger.dev/python Patch
@trigger.dev/react-hooks Patch
@trigger.dev/redis-worker Patch
@trigger.dev/rsc Patch
@trigger.dev/schema-to-json Patch
@trigger.dev/sdk Patch
@trigger.dev/database Patch
@trigger.dev/otlp-importer Patch
@trigger.dev/rbac Patch
@trigger.dev/sso Patch
@internal/clickhouse Patch
@internal/llm-model-catalog Patch
@internal/metrics-pipeline Patch
@internal/redis Patch
@internal/replication Patch
@internal/run-engine Patch
@internal/run-store Patch
@internal/schedule-engine Patch
@internal/tracing Patch
@internal/webhook-engine Patch
@internal/webhook-sources Patch
@internal/testcontainers Patch
@internal/cache Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4fbff98b-4359-45aa-b2bf-b98d672c266d

📥 Commits

Reviewing files that changed from the base of the PR and between 4038e6f and 9cd1c6c.

📒 Files selected for processing (10)
  • apps/webapp/app/env.server.ts
  • apps/webapp/app/v3/deploymentTelemetry.ts
  • apps/webapp/app/v3/services/deployment.server.ts
  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/finalizeDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
  • apps/webapp/app/v3/services/timeoutDeployment.server.ts
  • apps/webapp/app/v3/tracer.server.ts
  • internal-packages/database/prisma/migrations/20260825120000_add_worker_deployment_cli_version/migration.sql
  • internal-packages/database/prisma/schema.prisma
💤 Files with no reviewable changes (1)
  • internal-packages/database/prisma/migrations/20260825120000_add_worker_deployment_cli_version/migration.sql
🚧 Files skipped from review as they are similar to previous changes (9)
  • apps/webapp/app/env.server.ts
  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/finalizeDeployment.server.ts
  • apps/webapp/app/v3/services/deployment.server.ts
  • apps/webapp/app/v3/deploymentTelemetry.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
  • apps/webapp/app/v3/tracer.server.ts
  • apps/webapp/app/v3/services/timeoutDeployment.server.ts
  • internal-packages/database/prisma/schema.prisma

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.

📜 Recent review details
⏰ Context from checks skipped due to timeout. (31)
  • GitHub Check: report
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (19, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (24, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (23, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (20, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (22, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (15, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (21, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (17, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (18, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (14, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (13, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (16, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 24)
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (2, 2)
  • GitHub Check: internal / 🧪 Unit Tests: Internal
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (1, 2)
  • GitHub Check: typecheck / typecheck
  • GitHub Check: Analyze (javascript-typescript)
  • GitHub Check: 🛡️ E2E Auth Tests (full)

Walkthrough

Adds deployment lifecycle telemetry for initialization, success, failure, cancellation, and timeout events. Adds build-path and duration derivation helpers with tests. Persists the CLI version from the request header on deployments. Guards terminal database transitions against concurrent updates. Adds an optional OTLP exporter for deployment.* spans with header parsing and shutdown flushing.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 36.36% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 12 files. (2 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The description explains the telemetry changes and related fixes in detail, but it does not follow the repository template. It omits the issue reference, checklist, Testing section with test steps, Ch… Add the required template sections. Include the issue reference, completed checklist, specific testing steps, a short changelog entry, and screenshots or an explicit indication that screenshots are not applicable.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the main change: deployment lifecycle telemetry events. It is concise and specific.
Full details: Docstring Coverage

Explanation

Docstring coverage is 36.36% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 12 files. (2 skipped: 2 unsupported.)

Full details: Description check

Explanation

The description explains the telemetry changes and related fixes in detail, but it does not follow the repository template. It omits the issue reference, checklist, Testing section with test steps, Changelog section, and Screenshots section.

✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/tri-13477-deployment-lifecycle-telemetry

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
apps/webapp/app/v3/services/failDeployment.server.ts (1)

49-60: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Refresh the deployment before recording its lifecycle.

failedDeployment uses the row loaded before updateMany. A concurrent phase update can change startedAt, installedAt, builtAt, or buildServerMetadata, so recordDeploymentLifecycle may record stale values.

When updatedCount === 1, reload the row with findFirst and guard a missing result. Use the refreshed row for the lifecycle record. The event log uses only shortCode and does not require this reload.

Source: Coding guidelines


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b8a1cfb5-79a0-4d98-882b-9659465a6c46

📥 Commits

Reviewing files that changed from the base of the PR and between 486ec62 and cb09bb1.

📒 Files selected for processing (2)
  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 5 remain after this review.

📜 Review details
⏰ Context from checks skipped due to timeout. (35)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (21, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (16, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (13, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (24, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (23, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (15, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (22, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (19, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (20, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (17, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (18, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (14, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 24)
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (1, 2)
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (2, 2)
  • GitHub Check: typecheck / typecheck
  • GitHub Check: obsmap / 🧪 Unit Tests: Observability Map
  • GitHub Check: fk-cascade-guard / fk-cascade-guard
  • GitHub Check: internal / 🧪 Unit Tests: Internal
  • GitHub Check: runops-guard / runops-guard
  • GitHub Check: code-quality / code-quality
  • GitHub Check: report
  • GitHub Check: Analyze (javascript-typescript)
  • GitHub Check: 🛡️ E2E Auth Tests (full)
🧰 Additional context used
📓 Path-based instructions (10)
New code must target Run Engine V2 through the singleton in `app/v3/runEngine.server.ts`; do not reintroduce V1 execution paths. V1 branches may only reject or finalize gracefully with a clean 4xx.

📄 CodeRabbit inference engine (apps/webapp/CLAUDE.md)

Files:

  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
Never use `request.signal` to detect client disconnects. Use `getRequestAbortSignal()` from `app/services/httpAsyncStorage.server.ts`, which is wired to Express response close events.

📄 CodeRabbit inference engine (apps/webapp/CLAUDE.md)

Files:

  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
For dashboard changes, visually verify the running Remix app with Chrome DevTools MCP, using snapshots, screenshots, interaction, and console-message checks as appropriate.

📄 CodeRabbit inference engine (apps/webapp/CLAUDE.md)

Files:

  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
Add crumbs as you write code — not just when debugging. Mark lines with

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
Use zod for validation in packages/core and apps/webapp

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
Access environment variables through the `env` export of `env.server.ts` instead of directly accessing `process.env`

📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)

Files:

  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
Use function declarations instead of default exports

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
Use types over interfaces for TypeScript

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

Files:

  • apps/webapp/app/v3/services/failDeployment.server.ts
  • apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts
🔇 Additional comments (2)
apps/webapp/app/v3/services/failDeployment.server.ts (1)

44-89: Add crumbs markers to this transition path.

The guarded update and lifecycle emission have no // @Crumbs marker or `#region `@crumbs block. This repeats the existing missing-crumbs finding for neighboring telemetry code.

As per coding guidelines, add crumbs as you write code with // @Crumbs or `#region `@crumbs, then remove them with agentcrumbs strip before merge.

Source: Coding guidelines

apps/webapp/app/v3/services/recordDeploymentLifecycle.server.ts (1)

69-71: 🎯 Functional Correctness

No change needed for TIMED_OUT terminal time. timeoutDeployment.server.ts assigns and persists failedAt before passing timedOutDeployment to recordDeploymentLifecycle, so terminalAt resolves to failedAt.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

coderabbitai[bot]

This comment was marked as resolved.

…w cancel emission

Replaces the standalone DEPLOYMENT_TELEMETRY_ATTRIBUTES.md with short
comments on the DeploymentTelemetryAttributes keys, and chains the
canceled-lifecycle emission through the cancel ResultAsync pipeline
instead of a fire-and-forget promise.
coderabbitai[bot]

This comment was marked as resolved.

@myftija myftija changed the title feat(webapp): deployment lifecycle telemetry events per build path feat(webapp): deployment lifecycle telemetry events Aug 25, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

The webapp stamps cliVersion at deployment initialization from
x-trigger-cli-version, but the CLI only sent that header on two
unrelated endpoints - getHeaders() now includes it everywhere.
Also swap the webapp's regex check for a plain length cap.
@pkg-pr-new

pkg-pr-new Bot commented Aug 26, 2026

Copy link
Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@b515933

trigger.dev

npm i https://pkg.pr.new/trigger.dev@b515933

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@b515933

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@b515933

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@b515933

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@b515933

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@b515933

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@b515933

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@b515933

commit: b515933

@myftija
myftija merged commit 38e78f8 into main Aug 26, 2026
56 checks passed
@myftija
myftija deleted the feature/tri-13477-deployment-lifecycle-telemetry branch August 26, 2026 10:57
@github-actions github-actions Bot mentioned this pull request Aug 26, 2026
myftija added a commit that referenced this pull request Aug 26, 2026
…vents (#4785)

Adds `$trigger.org.slug` and `$trigger.project.name` attributes to the
`deployment.finished` / `deployment.initialized` events (follow-up to
#4778).
ericallam pushed a commit that referenced this pull request Aug 28, 2026
## Summary
4 new features, 12 improvements, 5 bug fixes.

## Improvements
- `trigger.dev deploy` now asks the server whether to build with Depot
or the native build server unless `--native-build`, `--depot-build`, or
`--local-build` is passed, so the native build server can be rolled out
per organization without a CLI change. `--local-bundle` and `--detach`
now require `--native-build`.
([#4803](#4803))
- Add an experimental `--local-bundle` deploy flag that runs the install
and bundling steps on your machine and uploads only the build output;
the image is still built remotely. Useful when your project's install
step needs tooling or credentials that only exist locally.
([#4331](#4331))
- Send the CLI version header on all API requests so deployments are
attributable to a CLI version
([#4778](#4778))
- A message that arrives mid-turn and is not injected into that turn is
now answered as the next turn, instead of being dropped. This is what
the `pendingMessages` docs have always described, and it applies to the
default too: configuring `pendingMessages` without a `shouldInject`
declines every batch, which previously meant every mid-turn message was
lost with no error at either end.
([#4795](#4795))
  
  ```ts
  chat.agent({
  id: "my-chat",
  pendingMessages: {
    onReceived: ({ message }) =>
      logger.info("arrived mid-turn", { id: message.id }),
    // Only interrupt once the agent has started calling tools.
    shouldInject: ({ steps }) => steps.length > 0,
  },
  run: async ({ messages, signal }) =>
    streamText({
      model,
      messages,
      abortSignal: signal,
      // Required for injection. Without it nothing injects, and every
      // mid-turn message is answered as the next turn instead.
      ...chat.toStreamTextOptions(),
    }),
  });
  ```
  
A declined message keeps its place in the queue, so it survives a crash
and is answered by whichever run picks the conversation up. An injected
one is consumed at the moment it is injected, so it is never also
answered as a later turn.
- Browser chats now keep the active turn open across page reloads when
older completion records are replayed.
([#4643](#4643))
- Add `chat.endAndContinue()` so fully hand-rolled custom chat agents
can hand a conversation off to a fresh run on the latest deployed task
version while preserving unconsumed Session input.
([#4647](#4647))
- Custom chat agents now validate and parse client data declared with
`chat.withClientData({ schema })` before passing it to agent code.
([#4646](#4646))

## Bug fixes
- Fixes a case where a chat could silently lose a message. If a message
arrived while the agent was between turns and a stop arrived after it,
the cursor the next boot resumed from could point past that message, so
it was never answered and no error was raised. This affected
`chat.agent`, not just custom agents.
([#4644](#4644))
  
Fixes a recovered answer being cut off. After a crash the agent replays
the message it had not answered yet, but it was replaying the stop that
arrived after that message too, so the turn answering it was aborted the
moment it began. A stop is now only applied to the turn that was live
when it arrived. That holds however the stop got there: sent after the
last completed turn, or sent to a chat whose most recent turn was
completed by an older version of the SDK.
  
One limitation to know about: the recovered answer is persisted
correctly, but a chat page that stayed open across the crash keeps
showing the partial answer it had already received. Reload the page to
see the full recovered answer.
  
Also fixes a retried send being answered twice. When a send was retried
and its idempotency claim was lost, the agent could consume the same
message a second time.
  
Custom agent loops can now inspect pending chat input without consuming
it, and consume one record at a time, with `chat.messages.hasPending()`
and `chat.messages.next()`. Records carry stable identifiers so a
redelivery is recognisable.
  
  ```ts
  if (await chat.messages.hasPending()) {
  const record = await chat.messages.next({ timeoutInSeconds: 0 });
  if (record) handle(record.payload);
  }
  ```
  
`hasPending()` answers for messages alone, so a message sitting behind a
stop, or behind a record this version of the SDK does not recognise,
still reports as pending and is still delivered. Anything the agent has
no consumer for is discarded rather than left where it would make every
message queued behind it undeliverable. `chat.messages.next()` returning
`undefined` means no message became consumable before the timeout.
  
`chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is
safe to resume from, not the sequence of the record the turn answered.
It is held back behind any message still waiting to be handled, so a
value below the record you just handled is expected.
- Fixed a chat agent hanging after an interrupted turn: when a run was
killed mid-answer (out of memory, crash, or eviction) and only the one
message it was answering was still outstanding, the new run never
replied to it. That message is now re-answered on the new run.
([#4768](#4768))
- Fix chat transport discarding the next turn after stopping generation.
`skipToTurnComplete` is now reset when a new message or action is sent,
so a message sent after `stopGeneration` streams normally instead of
leaving the chat stuck in a streaming state.
([#4744](#4744))
- Fixes a message sent while the agent was mid-answer being lost if the
run then crashed. The cursor written at the end of each turn could point
past a message that had arrived during that turn but had not been
answered yet, so the next boot skipped it and no error was raised
anywhere. Such a message is now held until a turn actually takes it.
([#4795](#4795))
  
This also removes the in-memory buffer those messages used to sit in, on
both `chat.agent` and `chat.createSession()`, so a message waiting for
its turn is durable rather than only present in the worker that received
it.

## Server changes

These changes affect the self-hosted Docker image and Trigger.dev Cloud:

- Self-hosted instances can now disable the admin dashboard and user
impersonation entirely. See the self-hosting docs for the new setting.
([#4774](#4774))
- The dashboard has two new themes, Black and White, plus appearance
options for stronger colors and underlined links.
([#4547](#4547))
- Deployment logs no longer jump to the bottom while you are reading
earlier output. Scroll up to pause auto-scroll, and scroll back down or
use the new scroll-to-bottom button in the log header to resume
following.
([#4776](#4776))
- Customize the runs list: show, hide, and reorder columns, and add
smart columns that pull a value straight out of a run's payload,
metadata, or output. Your column choices are saved in the page URL, so
you can share a view, bookmark it, or save it straight to your
favorites.
([#4652](#4652))
- Stop the browser offering to autofill or save environment variable
values as saved credentials.
([#4777](#4777))
- Cut webapp CPU usage by about a quarter on the routes that workers
call most, freeing headroom at the same request rate. Detailed
event-loop blocking traces are no longer recorded by default, because
producing them was itself a large part of that cost.
([#4746](#4746))
- When a runs list or runs.list API request spans too much data to
complete, it now returns a clear, actionable error asking you to narrow
the time range, instead of failing with a generic error.
([#4773](#4773))
- Improved the performance and reliability of the runs list and the
runs.list API, especially for large projects and filtered views.
([#4763](#4763))
- New Vercel connections now get version skew protection turned on
automatically, so each run uses the task version its deployment shipped
with. Automatic atomic deployments are deprecated and no longer offered
when you connect a project, but stay available in your Vercel
integration settings.
([#4741](#4741))
- The Staging branch setting now shows an upgrade prompt on plans that
don't include a Staging environment, instead of looking editable and
then silently doing nothing when saved.
([#4784](#4784))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.5.13

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.13`
## trigger.dev@4.5.13

### Patch Changes

- `trigger.dev deploy` now asks the server whether to build with Depot
or the native build server unless `--native-build`, `--depot-build`, or
`--local-build` is passed, so the native build server can be rolled out
per organization without a CLI change. `--local-bundle` and `--detach`
now require `--native-build`.
([#4803](#4803))
- Add an experimental `--local-bundle` deploy flag that runs the install
and bundling steps on your machine and uploads only the build output;
the image is still built remotely. Useful when your project's install
step needs tooling or credentials that only exist locally.
([#4331](#4331))
- Send the CLI version header on all API requests so deployments are
attributable to a CLI version
([#4778](#4778))
- Updated dependencies:
  - `@trigger.dev/core@4.5.13`
  - `@trigger.dev/build@4.5.13`
  - `@trigger.dev/schema-to-json@4.5.13`
## @trigger.dev/core@4.5.13

### Patch Changes

- `trigger.dev deploy` now asks the server whether to build with Depot
or the native build server unless `--native-build`, `--depot-build`, or
`--local-build` is passed, so the native build server can be rolled out
per organization without a CLI change. `--local-bundle` and `--detach`
now require `--native-build`.
([#4803](#4803))
- Add an experimental `--local-bundle` deploy flag that runs the install
and bundling steps on your machine and uploads only the build output;
the image is still built remotely. Useful when your project's install
step needs tooling or credentials that only exist locally.
([#4331](#4331))
- A message that arrives mid-turn and is not injected into that turn is
now answered as the next turn, instead of being dropped. This is what
the `pendingMessages` docs have always described, and it applies to the
default too: configuring `pendingMessages` without a `shouldInject`
declines every batch, which previously meant every mid-turn message was
lost with no error at either end.
([#4795](#4795))

  ```ts
  chat.agent({
    id: "my-chat",
    pendingMessages: {
      onReceived: ({ message }) =>
        logger.info("arrived mid-turn", { id: message.id }),
      // Only interrupt once the agent has started calling tools.
      shouldInject: ({ steps }) => steps.length > 0,
    },
    run: async ({ messages, signal }) =>
      streamText({
        model,
        messages,
        abortSignal: signal,
        // Required for injection. Without it nothing injects, and every
        // mid-turn message is answered as the next turn instead.
        ...chat.toStreamTextOptions(),
      }),
  });
  ```

A declined message keeps its place in the queue, so it survives a crash
and is answered by whichever run picks the conversation up. An injected
one is consumed at the moment it is injected, so it is never also
answered as a later turn.

- Fixes a case where a chat could silently lose a message. If a message
arrived while the agent was between turns and a stop arrived after it,
the cursor the next boot resumed from could point past that message, so
it was never answered and no error was raised. This affected
`chat.agent`, not just custom agents.
([#4644](#4644))

Fixes a recovered answer being cut off. After a crash the agent replays
the message it had not answered yet, but it was replaying the stop that
arrived after that message too, so the turn answering it was aborted the
moment it began. A stop is now only applied to the turn that was live
when it arrived. That holds however the stop got there: sent after the
last completed turn, or sent to a chat whose most recent turn was
completed by an older version of the SDK.

One limitation to know about: the recovered answer is persisted
correctly, but a chat page that stayed open across the crash keeps
showing the partial answer it had already received. Reload the page to
see the full recovered answer.

Also fixes a retried send being answered twice. When a send was retried
and its idempotency claim was lost, the agent could consume the same
message a second time.

Custom agent loops can now inspect pending chat input without consuming
it, and consume one record at a time, with `chat.messages.hasPending()`
and `chat.messages.next()`. Records carry stable identifiers so a
redelivery is recognisable.

  ```ts
  if (await chat.messages.hasPending()) {
    const record = await chat.messages.next({ timeoutInSeconds: 0 });
    if (record) handle(record.payload);
  }
  ```

`hasPending()` answers for messages alone, so a message sitting behind a
stop, or behind a record this version of the SDK does not recognise,
still reports as pending and is still delivered. Anything the agent has
no consumer for is discarded rather than left where it would make every
message queued behind it undeliverable. `chat.messages.next()` returning
`undefined` means no message became consumable before the timeout.

`chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is
safe to resume from, not the sequence of the record the turn answered.
It is held back behind any message still waiting to be handled, so a
value below the record you just handled is expected.
## @trigger.dev/python@4.5.13

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/sdk@4.5.13`
  - `@trigger.dev/core@4.5.13`
  - `@trigger.dev/build@4.5.13`
## @trigger.dev/react-hooks@4.5.13

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.13`
## @trigger.dev/redis-worker@4.5.13

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.13`
## @trigger.dev/rsc@4.5.13

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.13`
## @trigger.dev/schema-to-json@4.5.13

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.5.13`
## @trigger.dev/sdk@4.5.13

### Patch Changes

- Fixed a chat agent hanging after an interrupted turn: when a run was
killed mid-answer (out of memory, crash, or eviction) and only the one
message it was answering was still outstanding, the new run never
replied to it. That message is now re-answered on the new run.
([#4768](#4768))
- Browser chats now keep the active turn open across page reloads when
older completion records are replayed.
([#4643](#4643))
- Add `chat.endAndContinue()` so fully hand-rolled custom chat agents
can hand a conversation off to a fresh run on the latest deployed task
version while preserving unconsumed Session input.
([#4647](#4647))
- Fix chat transport discarding the next turn after stopping generation.
`skipToTurnComplete` is now reset when a new message or action is sent,
so a message sent after `stopGeneration` streams normally instead of
leaving the chat stuck in a streaming state.
([#4744](#4744))
- Custom chat agents now validate and parse client data declared with
`chat.withClientData({ schema })` before passing it to agent code.
([#4646](#4646))
- Fixes a message sent while the agent was mid-answer being lost if the
run then crashed. The cursor written at the end of each turn could point
past a message that had arrived during that turn but had not been
answered yet, so the next boot skipped it and no error was raised
anywhere. Such a message is now held until a turn actually takes it.
([#4795](#4795))

This also removes the in-memory buffer those messages used to sit in, on
both `chat.agent` and `chat.createSession()`, so a message waiting for
its turn is durable rather than only present in the worker that received
it.

- A message that arrives mid-turn and is not injected into that turn is
now answered as the next turn, instead of being dropped. This is what
the `pendingMessages` docs have always described, and it applies to the
default too: configuring `pendingMessages` without a `shouldInject`
declines every batch, which previously meant every mid-turn message was
lost with no error at either end.
([#4795](#4795))

  ```ts
  chat.agent({
    id: "my-chat",
    pendingMessages: {
      onReceived: ({ message }) =>
        logger.info("arrived mid-turn", { id: message.id }),
      // Only interrupt once the agent has started calling tools.
      shouldInject: ({ steps }) => steps.length > 0,
    },
    run: async ({ messages, signal }) =>
      streamText({
        model,
        messages,
        abortSignal: signal,
        // Required for injection. Without it nothing injects, and every
        // mid-turn message is answered as the next turn instead.
        ...chat.toStreamTextOptions(),
      }),
  });
  ```

A declined message keeps its place in the queue, so it survives a crash
and is answered by whichever run picks the conversation up. An injected
one is consumed at the moment it is injected, so it is never also
answered as a later turn.

- Fixes a case where a chat could silently lose a message. If a message
arrived while the agent was between turns and a stop arrived after it,
the cursor the next boot resumed from could point past that message, so
it was never answered and no error was raised. This affected
`chat.agent`, not just custom agents.
([#4644](#4644))

Fixes a recovered answer being cut off. After a crash the agent replays
the message it had not answered yet, but it was replaying the stop that
arrived after that message too, so the turn answering it was aborted the
moment it began. A stop is now only applied to the turn that was live
when it arrived. That holds however the stop got there: sent after the
last completed turn, or sent to a chat whose most recent turn was
completed by an older version of the SDK.

One limitation to know about: the recovered answer is persisted
correctly, but a chat page that stayed open across the crash keeps
showing the partial answer it had already received. Reload the page to
see the full recovered answer.

Also fixes a retried send being answered twice. When a send was retried
and its idempotency claim was lost, the agent could consume the same
message a second time.

Custom agent loops can now inspect pending chat input without consuming
it, and consume one record at a time, with `chat.messages.hasPending()`
and `chat.messages.next()`. Records carry stable identifiers so a
redelivery is recognisable.

  ```ts
  if (await chat.messages.hasPending()) {
    const record = await chat.messages.next({ timeoutInSeconds: 0 });
    if (record) handle(record.payload);
  }
  ```

`hasPending()` answers for messages alone, so a message sitting behind a
stop, or behind a record this version of the SDK does not recognise,
still reports as pending and is still delivered. Anything the agent has
no consumer for is discarded rather than left where it would make every
message queued behind it undeliverable. `chat.messages.next()` returning
`undefined` means no message became consumable before the timeout.

`chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is
safe to resume from, not the sequence of the record the turn answered.
It is held back behind any message still waiting to be handled, so a
value below the record you just handled is expected.

- Updated dependencies:
  - `@trigger.dev/core@4.5.13`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants