feat(webapp): deterministic grounding facts and org-wide tokens for the dashboard agent - #4796
feat(webapp): deterministic grounding facts and org-wide tokens for the dashboard agent#4796kathiekiwi wants to merge 66 commits into
Conversation
…text Stops the assistant chat crashing to a full-screen error when the code-highlighting chunk fails to load. Retries twice with backoff, then renders plain text instead of throwing.
…d errors Extracts the lazy chunk-load factory as loadStreamdownRenderer so the plain-text fallback path is covered by a test. The fallback still re-raises the original error as an unhandled rejection so the deploy-skew asset-recovery reload can pick it up.
…s their organization
One read-only Lua script reads the base concurrency sets, every CK variant in ckIndex and the runningCounter together, so the run ids behind a queue's running count come with counts from the same snapshot.
Adds slotHolders and slotHolderFacts to the retrieved queue: which runs hold the queue's slots, their phase, and whether Redis membership matches the run's status. Both the Redis and Postgres reads degrade instead of failing.
The holder list can never claim completeness for a CK queue, so the contract now reports only what is provable: truncated when the cap was hit and unlistedRunning for dequeued holders that exist but aren't listed.
Pass slotHolders/holderResolution through from the queue live row when present, and ground the model on how to read them: name the holder when consistent, call out scheduler/run-state mismatches without saying leaked or stale, and never assert an executing run from runningNow alone.
Grounding block now covers partial resolution and unresolved holders, the none branch no longer asserts usage as fact, and the runningNow-mismatch clause fires only when holderResolution is complete.
Adds slotHolderFacts to withLiveState with the same independent gating as slotHolders/holderResolution. Grounding block covers admitted-vs- dequeued phase and prefers slotHolderFacts over comparing runningNow manually.
…ists slotHolders is never exhaustive by contract for per-key concurrency queues, so no field claims completeness. slotHolderFacts gains truncated/unlistedRunning as proof of unlisted holders, and its consistency can be unresolved (counts then unusable). Grounding rules and tests updated to match.
get_repo_info and read_file now surface the run-pinned deployment's dirty flag (built from a tree with uncommitted changes), and the source read ledger tracks it per-sha so evidence canonicalization can caveat citations instead of asserting an exact commit match.
A dirty run-pinned deploy and the clean tracked branch can share a sha. dirtyForSha was last-write-wins, so a later clean read of that sha erased the dirty caveat. Fixed to OR instead of overwrite.
…oundary The dashboard agent's delegated token now carries the organization alongside the environment. For a token with an organization, the request may name any environment in that org — re-authorized against the org and the user's membership — and the token's own environment is only the default. Tokens without one stay env-pinned.
…h alert The unsubscribe route read the environment off the token alone, so an org-wide token could subscribe an alert in a sibling environment but not remove it. It now resolves the environment the same way the other agent routes do, checked against the token's organization, which resolveAgentAlertContext requires its caller to pass.
Curated runs expose a computed wait (queued vs created basis, reliability flag) instead of raw timestamps the model had to subtract itself, mirroring dashboardAgentWatchRunChecks' queue-wait semantics. The webapp run presenter now selects queuedAt and derives queueWaitReliable from the raw status. Trace spans carry their spanId, and span evidence is validated against a per-turn span ledger (mirroring the source-read ledger) so a citation must come from this turn's trace read. Source evidence also gets a code-stamped dirty flag from the same ledger, fed by run-pinned/default snapshot dirtiness. Error groups expose a computed recurredSinceResolve instead of leaving the model to compare resolvedAt/lastSeen dates.
…akdown on get_queue The queue can show headroom while the environment is saturated, so the binding constraint may not be the queue itself. Add envConcurrency (limit, current) to QueueRetrievePresenter, guarded like slotHolders, and pass it through the dashboard agent's get_queue tool alongside the concurrency override breakdown (base/override/overriddenBy/overriddenAt) the route already returns but the tool was dropping.
… not the plain limit current >= limit is not the dequeue gate; it's current >= limit * burstFactor (burstFactor defaults to 2). Add burstFactor to EnvConcurrency and reword the get_queue description so the model reasons from the real gate instead of assuming current == limit means the environment is saturated.
Add optional scope/queueName/limit/current fields so the model knows which queue or env is saturated, instead of guessing from the page. Populated by the webapp from data already graded (no new queries), carried through verbatim by the dashboard-agent tool.
The bare-evidence test asserted the pre-validation contract without ever reading a trace. Now it drives get_run_trace for real before citing the span, plus a new case for a span id no trace read returned this turn.
Dedupe get_queue grounding between the tool description and the system prompt, and tighten verbose investigation/watch phrasing, to bring both prompt.chars ceilings back under budget without dropping any grounding rule.
Fabricates PG+Redis fixtures for S1-S6 and S10 of the dashboard-agent UAT scenarios in the local References/hello-world dev environment.
🦋 Changeset detectedLatest commit: 2381a8a The changes in this PR will be included in the next version bump. This PR includes changesets to release 27 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…roducer absence claims The sweep read as an offer, so the model asked permission instead of sweeping; it's now mandatory in the same turn, with the offer scoped to whatever's beyond it. get_queue's consumerTasks wording licensed "nothing writes to it" almost verbatim — reworded to an observation.
… a trim The mandatory-sweep rewrite lost the ban on pointing the user at the switcher for scopes the agent can read. Restored, and trimmed a few words elsewhere in get_queue's grounding to hold the prompt budget.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| const curated = curateTrace(result.data); | ||
| spanLedger.recordTraceSpans( | ||
| runId, | ||
| curated.spans.map((s) => s.spanId).filter((id): id is string => typeof id === "string") | ||
| ); | ||
| return curated; |
There was a problem hiding this comment.
🟡 Cross-project citations point at the wrong project
The new project/environment override lets a trace read pull spans from a sibling project (recordTraceSpans), but evidence is canonicalized with the turn's own scope in canonicalizeEvidence, so a span cited from that read gets a trigger:// URI naming the current project instead of the one the data came from.
Impact: A citation for cross-project data links to the wrong run or 404s when opened.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Accepted by design: a cross-project citation carries the turn's scope, and run friendly ids are globally unique, so a mis-scoped link 404s rather than resolving to the wrong run — it never mislinks.
…escriptions The system-prompt bullet doesn't bind at decision time; get_run, get_error, get_queue, and correlate_version now carry the imperative in their own descriptions (tools.chars is uncapped). correlate_version also gained project/environment overrides and lost its "dev runs behave this way" claim, which asserted a run's deploy state from a single-environment 404 — the same fabricated-absence bug banned elsewhere. Tool schemas rebuild fresh every turn with no caching or dashboard-override path (unlike the system prompt, which resolves through a managed prompt cached per worker process), so this change is live on the very next call.
…the found environment
Drives a real magic-link login and the dashboard-agent resource route over HTTP, polls until the turn settles, and prints the transcript. Lets UAT scenarios be exercised without a browser.
| if (claims.organizationId) { | ||
| const environmentId = requested.environmentId ?? claims.environmentId; | ||
| if (!environmentId) { | ||
| return { | ||
| ok: false, | ||
| code: "invalid_target", | ||
| error: "Name the environment to use, as `environmentId`.", | ||
| }; | ||
| } | ||
| return { ok: true, environmentId, organizationId: claims.organizationId }; | ||
| } |
There was a problem hiding this comment.
🔍 Org-wide token makes environmentId a default, not a pin
Because the mint now always stamps organizationId, resolveAgentTokenScope always takes the org branch and honors a request-body environment over the token's own. Every agent token is effectively org-wide. Tenant safety rests entirely on downstream re-authorization — authorizeWatchEnvironmentById plus the org checks in resolveAgentAlertContext and assertUserActorEnvironmentAccess. Any route reading these claims that still treats environmentId as an authoritative pin would now accept a body-named environment.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
This is the intended org-wide design (documented in the file header): the org is the authorization boundary and the env is a default; assertUserActorEnvironmentAccess re-authorizes membership + org match downstream on every exchange.
| description: | ||
| "Get the status, timing, cost, and error details for a single run in the current environment, by its run id (run_...).", | ||
| "Get the status, timing, cost, and error details for a single run in the current environment, by its run id (run_...). The `wait` field is the already-computed queue wait (or, when unreliable, time since creation) — never recompute it from createdAt/startedAt. A 404 (in the error message) means this run isn't in the current environment, never that it doesn't exist: you MUST immediately, this same turn, with no permission question, call list_projects and repeat this call with `project`/`environment` set for the current project's other environments and every sibling project's matching environment, then answer naming what you checked.", | ||
| inputSchema: z.object({ | ||
| runId: z.string().describe("The run id, e.g. run_abc123."), | ||
| project: projectOverrideField, | ||
| environment: environmentOverrideField, | ||
| }), | ||
| }); | ||
|
|
||
| export const getRunTraceSchema = tool({ | ||
| description: | ||
| "Get a run's execution trace: the timeline of spans (tasks, waits, attempts) with durations and error flags. Use this to explain why a run failed, retried, or was slow.", | ||
| "Get a run's execution trace: the timeline of spans (tasks, waits, attempts) with durations and error flags. Use this to explain why a run failed, retried, or was slow. Each span's `spanId` is required to cite it as span evidence — only ids returned by this call are citable.", | ||
| inputSchema: z.object({ | ||
| runId: z.string().describe("The run id, e.g. run_abc123."), | ||
| project: projectOverrideField, | ||
| environment: environmentOverrideField, | ||
| }), | ||
| }); | ||
|
|
There was a problem hiding this comment.
🔍 list_errors omits the cross-project override its siblings gained
get_run, get_run_trace, get_error, get_queue, correlate_version, and list_runs all gained project/environment override inputs, but list_errors did not. The omission looks deliberate, but given the prompt's mandatory cross-project sweep instruction, confirm the agent isn't expected to list errors in a sibling project.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Deliberate: the cross-project sweep enumerates via list_runs then resolves with get_error/get_run under an explicit scope; there is no cross-project error-listing step, so list_errors intentionally has no project/environment override.
@trigger.dev/build
trigger.dev
@trigger.dev/core
@trigger.dev/python
@trigger.dev/react-hooks
@trigger.dev/redis-worker
@trigger.dev/rsc
@trigger.dev/schema-to-json
@trigger.dev/sdk
commit: |
| export function activeToolPendingKey(status: string, inFlightTool: string | null): string | null { | ||
| const inFlight = status === "streaming" || status === "submitted"; | ||
| return inFlight ? inFlightTool : null; | ||
| } |
There was a problem hiding this comment.
🔍 Tool-pending deadline can arm on a leftover part while merely submitted
activeToolPendingKey treats submitted as in-flight, so a resumed transcript carrying a stale input-available tool part arms the tool-pending timer before any tool call is issued this turn. The idle (ready) case is gated; submitted is not, so a stuck-tool error can appear for a turn still in its thinking phase.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
A new submitted turn puts a user message last, so inFlightToolName returns null and the tool-pending deadline can't arm on a leftover part; it's additionally gated on status streaming/submitted.
806448e to
5c65e5f
Compare
…rganization /api/v1/projects is identity-only and returns every org the user belongs to, so a sweep could 403 on a same-named foreign-org project and report the sibling as not found without ever reaching it. curateProjects now filters by ctx.organizationId (fails closed to empty when it's missing) and returns ref+name only.
…ganizationId is missing An empty list read as a proven absence the sweep rule would act on; now it's an explicit error, and the fixture asserts that shape.
…ngerprint S3 uat-ck-queue had no BackgroundWorkerTask pointing at it, so get_queue's consumerTasks was empty and the agent's honest "undeployed" answer preempted the invisible-holder scenario. Registers one on the env's current worker (minting a minimal V2 one if the env has none yet). S10's printed ClickHouse INSERT omitted error_fingerprint, so both materialized views (errors_v1, error_occurrences_v1) silently excluded the row - get_error/list_errors could never find it. Also fixes a SQL-string escaping bug where the stack trace's \n broke ClickHouse's JSON parser. Prints the ask-able error_<fingerprint> id.
aae1ac0 to
235cfc3
Compare
… sweep A 403 (no personal dev env in a sibling project) was ending the sweep early. Factored the repeated sweep imperative into one shared const across get_run/get_error/get_queue/correlate_version's descriptions and the system-prompt bullet, both now saying inaccessible scopes are skipped, not stopped on, and reported alongside what was checked.
list_environments 403s cross-project on the delegated token, which the
model read as fatal and stopped the sweep on. It now returns a
structured { inaccessible: true, projectRef } shape, and the sweep
imperative no longer routes the sibling leg through it at all: each
sibling is retried directly with project set and environment
defaulting to the current name, reserving list_environments for the
current project's own other environments.
dda1795 to
e0d1943
Compare
A re-read after a turn settles keyed only on message id, so a user message re-read under a different id than its optimistic copy got appended again at the end of the transcript, after the reply.
…d closing shape
A ban the model could satisfy while still opening with a restated
cause ("the root cause is clear: ..."). Replaced with a format: at
most two sentences, one optional new fact then one offer, never
opening with anything the card already states.
"leaked"/"stale" is now sayable only when both facts are observed this turn (terminal or missing run state, and the scheduler still holding the slot) — never from slotHolderFacts/counters alone, which stays an absolute ban.
…tches A. The sweep only retried siblings at the matching env name, missing a target in a sibling's other environment. It now also retries each accessible project's other environments (list_environments where reachable, else prod/stg/staging guesses that 4xx cleanly) once the matching-env round misses. Kept the system-prompt bullet short and put the detail in the uncapped tool descriptions. B. schedule_watch gains optional project/environment inputs, resolved to a canonical environmentId through the same JWT exchange the data tools use (decoding the minted token's `sub`), and carried on the watch intent as `target`. The default (no-override) path is unchanged and makes no extra network call. Alert-subscription tools stay chat-scoped: an alert has no independent target to override. This closes the loop at the tool/schema layer only — the in-chat confirm flow (webapp) does not yet read the intent's `target` when creating the watch, so a confirmed cross-project watch still needs that follow-up to actually persist against the resolved environment.
watchDraftSchema carries an optional target ({environmentId}), plumbed
from the proposed watch intent through openWatchCard/watchDraftFor and
submitted with the draft. The watch-create route resolves and
re-authorizes draft.target with the same session-based membership gate
as the URL's own environment, and additionally requires it to stay
inside the URL's own organization; absent target, behavior is
unchanged. Queue/target validation now runs against whichever
environment was resolved, since that's the object passed downstream.
| const dismissError = useCallback(() => { | ||
| clearError(); | ||
| setDeadlineError(null); | ||
| firstEventDeadline.sync(null); | ||
| toolPendingDeadline.sync(null); | ||
| }, [clearError, firstEventDeadline, toolPendingDeadline]); |
There was a problem hiding this comment.
🔍 Dismissed bounded-wait error never re-arms during the same turn
dismissError clears the error and calls sync(null) on both deadlines but never bumps attempt, unlike retry. With status and messages unchanged, the deadline effects do not re-run, so a still-running turn stays unbounded after one dismiss. Confirm this is the intended dismiss semantics.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Owner decision — dismiss means 'stop telling me', so it deliberately does not bump the attempt counter (unlike retry, which re-arms). Intended asymmetry.
| export function dashboardAgentTriggerConfig(): { ttl: string; lockToVersion?: string } { | ||
| return { | ||
| ttl: DASHBOARD_AGENT_RUN_TTL, | ||
| ...(env.DASHBOARD_AGENT_VERSION ? { lockToVersion: env.DASHBOARD_AGENT_VERSION } : {}), | ||
| }; | ||
| } |
There was a problem hiding this comment.
🔍 2m run TTL also applies to session continuation runs
dashboardAgentTriggerConfig now always sets ttl: "2m", and sessions reuse the same triggerConfig for continuation runs after termination/end-and-continue. Under worker starvation a continuation run could expire after 2 minutes undequeued, not only an initial run that was never picked up. Confirm this is acceptable for continuations.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
The 2m ttl only expires PENDING, non-executing, unlocked runs; a session continuation run is EXECUTING by the time it's parked on session.in, so the ttl never bites a live continuation.
- Restore process listeners in a finally block so a failed StreamdownRenderer test doesn't leak unhandledRejection listeners into later tests. - Import assertExhaustive from the core/utils subpath instead of the root. - Correct the envConcurrency.current comment: it's the displayed dequeued count, not the gated concurrency count, and can trail it. - Correct the dashboard-agent token comment: environmentId is the default, organizationId is the actual authorization boundary. - Cap the CK-variant scan in slotHoldersOfQueue's Lua so a queue with many concurrency-key variants can't turn a per-request read into an unbounded scan; report the cap via the existing truncated signal. - Carry queuedAt/queueWaitReliable through the run list presenter so list_runs computes wait the same way get_run does. - Print the seed script's ClickHouse fixture command as a heredoc so it survives copy-paste.
- The honesty rule's "single snapshot, never proof of a leak" no longer contradicts get_queue's own leaked/stale exception (both facts observed same-turn); it now points to that grounding instead of re-banning it. - The closing-message rule now also bans mid-sentence restatement of the card, reworded or not, not just opening with it.
1 participant
Grounds the dashboard agent's answers in deterministic platform facts instead of LLM inference, and widens its token to the organization.