diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/protocols/stage-1.md b/docs/design/agent-workflows/projects/agent-multi-modality/protocols/stage-1.md index 3177045129..d15be48c4a 100644 --- a/docs/design/agent-workflows/projects/agent-multi-modality/protocols/stage-1.md +++ b/docs/design/agent-workflows/projects/agent-multi-modality/protocols/stage-1.md @@ -308,3 +308,170 @@ dissolves on the next rebase. - **The catalogs cannot express document or audio support today**, so native document delivery (Stage 2) will need the catalog schema to grow before the gate can ever say yes; the gate's absence-means-unknown rule is what keeps that honest in the meantime. + +## WP4: frontend transport and rendering + +### Implementation decisions worth knowing (beyond the plan) + +- The shared composer package (`RichChatInput`, `SubmitPlugin`, `SendButton` in `agenta-ui`) + gained send-only disabling. The review enumerated all three consumers; the new props default + to today's behavior, so the two non-attachment surfaces are untouched. +- The tray uid doubles as the upload idempotency key (stated in a comment at the generation + site); that identity is what makes retry reuse safe. `generateId()` is used instead of + `crypto.randomUUID`, which is undefined on plain-HTTP dev deployments. +- Send blocks (with a tooltip) until every tray upload settles; a failed upload never falls back + to base64. Removing a chip aborts its in-flight request. +- Static composer limits stay; capability-derived limits remain the Stage 2 item. +- web/oss joined the unit-test loop with an explicit include list; the 13 pre-existing orphaned + test files stay excluded and are tracked in issue #5618. + +### The review, and what it changed + +Fourteen findings, four merge blockers, all fixed before the PR opened: + +1. The acceptance Playwright spec sat one directory outside the collected test root and would + never have run anywhere; it now lives with its siblings, carries the suite's tags, and has no + silent environment skip. +2. Voice messages routed through the upload transport unconditionally, so with voice on and + uploads off a recording died in a permanent error chip; the recorder now uses references only + when the uploads flag is on and keeps its inline path otherwise, preserving the flags' + independence (the seam WP2's protocol had predicted would matter). +3. `size` sat top-level on the file part, where the AI SDK's validation strips it, so every + persisted turn would silently lose the field; it moved into the `providerMetadata.agenta` + envelope beside the id. +4. A hardcoded perception map contradicted the model-capability data the same component already + computes for the voice controls; perception now derives from the shared memoized fact, with + absent-or-unknown meaning the workspace-only notice. + +The remaining ten: an error taxonomy for uploads (cap and quota named, Retry-After honored, +old-backend 404 explained, non-retryable states without a retry button), abort-on-remove, a +double-send guard over the voice path's upload await, delivery notices joined to filenames, +lowercase-pinned id validation matching the adapter, a filename fallback that never renders the +URL tail, no eager full-file downloads on render, junit reporting for the new test setup, restored +comment rationales, and feedback for swallowed Enter presses. + +### Forced routes to double-check + +- **The one-line CI glob addition** (publishing web/oss's junit report) is left uncommitted: the + session's push credential lacks the GitHub `workflow` scope, and a commit touching workflow + files is rejected at push. The vitest gate itself works without it (the runner discovers the + script); only the published report is missing until someone with the scope lands the line. +- **The Fern/OpenAPI regeneration** for typed accessors elsewhere in the sessions domain is a + stated follow-up, deliberately not bundled into this package's diff. + +## The end-to-end QA (dev stack, all four packages deployed, 2026-08-01) + +All four containers were recreated and the complete path driven from a real browser on the QA +account. First round: everything behind the front end passed at wire level (a referenced upload +delivered natively and read back verbatim; a ZIP claimed, materialized, and read from the +workspace by the agent; reload-from-records rendering with real filenames and zero base64 in the +DOM; the over-cap error naming the limit with no retry offer), but every composer upload failed +with 422. The cause was a two-line front-end omission: the shared axios instance defaults to a +JSON content type and silently JSON-serializes a FormData, collapsing the file to an empty +object; the existing multipart callers in the codebase pass the explicit header and the new +transport did not. With the header added, the re-run passed both blocked cases: upload 200 with +a real multipart body, the run request carrying the reference and no base64, the reply reading +the image's text, and the ZIP's named workspace-only notice followed by the agent reading the +archive's marker. + +Observations recorded for follow-up, none blocking: + +- **ZIP-container sniffing mislabels.** A plain `.zip` stores as the Word-document media type + (both share the container signature and the sniffer scores the more specific format higher). + Delivery classification and the notice were still correct; only the stored label is wrong. + Tracked as a classifier refinement. +- **A corrupt image surfaces as a raw provider 400** in the chat rather than a friendly message + (found via a QA-side transcription error, reproduced deliberately). +- **The model may perceive via either channel.** With both the native block and the working copy + present, one run answered through its read tool rather than native vision; the guarantee D1 + promises (the bytes reach the model) held either way. +- Pre-existing and unrelated: the stale provider-key banner, the billing cron's date validation + error, and an approvals-lane behavior ("Deny and send note" answering "not handled") flagged + on PR #5598. + +### Product-owner corrections (2026-08-01) + +Three corrections after the product owner saw the shipped surfaces: + +1. **The in-message delivery notice was removed** (": the model did not perceive this + file. The agent can use it at attachments/…"). Its wording and placement were + implementation-invented. +2. **The chip-level hint was removed too** (the crossed-eye indicator with "The model may not + perceive this file…"). The first release therefore ships no notice UI at all; decision D6's + "visible notice" is deferred to a future surface the product owner designs. The + `attachment_delivery` events keep flowing and persisting unchanged, and the front end parses + and ignores them, so that future surface needs no wire changes. The upload error messages + were reviewed and kept. +3. **The per-turn attachment count rose from 5 to 100** (front-end constant and the runner's + `AGENTA_ATTACHMENTS_MAX_PER_TURN` default). The old 5 was the dark-shipped composer default + with no motivating constraint; provider count caps are far higher. Because the per-session + count quota was also 100, one full turn would have consumed it, so the session count quota + rose to 1,000; the 256 MB per-session byte quota and the 20-pending cap stay the real + bounds. The design docs' matrix numbers were updated in the same pass. + +## The external-review fix wave (2026-08-01) + +After the train went up for review, three sources produced findings: a Codex sweep across the +implementation PRs (11 comments), CodeRabbit's open threads, and CodeRabbit's stack-wide review +body. The product owner adjudicated every item one by one; the accepted set landed as one wave of +lane-scoped fix commits (api 56bc0b6526 + 973ade3965, runner 208fb31218 + 020b56dea6, sdk +df5321bc8c, services bf070f7309, frontend aa1cb47a7f, docs 5cd13b49c2 and 64c52a8b04 with +212d6470a2 and 200c360c60). Every thread got a reply naming its commit; addressed threads were +resolved. + +Decisions worth restating, with their reasons: + +- **A failed attachment claim stays non-fatal** (CodeRabbit wanted the result consumed). A swept + attachment already degrades to a "no longer available" mention on cold replay; failing the turn + over reference bookkeeping would trade a graceful loss for a hard one. Documented at the call + site. +- **The stale-after-24h staged upload is a recorded v1 limitation.** The failure needs a tab left + open for a day with a staged file; the fix (revalidate at send) is medium-sized and deferred. +- **The reference contract rose to 100 ids** to match the 100-file turn the product owner set; + the mismatch (composer 100, claim contract 50) would have silently unreferenced files on large + turns, which the sweep would then delete. +- **Legacy inline images now respect supplied model capabilities.** The image-assuming literal + applies only when the resolver sent nothing, preserving legacy behavior exactly where legacy + behavior is the only information available. +- **The "fresh user content" rule became one shared predicate** (text, attachments, or inline + media) used by turn authority, resume classification, and freshness; the three had drifted + before and each drift was a bug. +- **The denied-tool corruption found during QA was root-caused to the client-tool dispatcher** + (a denied part auto-settled as "not handled by this client", erasing the denial from history). + Fixed with tests in PR #5630, stacked above WP4 because the owning files predate the train. +- **Two CodeRabbit "Major" flags on the built-ins open-questions doc stay open questions** + (unconfined local sandbox, tool-name collisions). They are product decisions the doc already + records; the feature is dev-only behind a flag. + +Skipped with rebuttals on the threads: response-model count computation (breaks the file's +uniform convention), nginx body-size scoping and a purpose enum (low value against churn), and a +provider-independent inline base64 ceiling (only one provider needs a cap today). + +## The rebase onto release/v0.107.0 (2026-08-01, evening) + +Release branch `release/v0.107.0` was cut from main (which had meanwhile absorbed 106.2). The +whole workspace retargeted onto it. The 106.2 release had split `AgentConversation.tsx` into +hooks and views and shipped its own upload plumbing, so the frontend commits could not replay +textually; each conflicted commit was resolved by porting its INTENT into the new structure, +one commit at a time, preserving history-faithful intermediate states (one mid-history test +failure existed in the original branch at that same point and healed when the next commit +replayed, exactly as in the original). + +Where things moved: the composer attachment machinery now lives in +`hooks/useComposerAttachments.ts` (upload-on-attach, staged files, settle rule, scoped clear), +paste/drag guards partly in `components/AgentComposerDock.tsx`, message rendering behind +`components/AgentTurn.tsx`. The base's own defect fixes (unreadable-file hold, multi-tab +follow, turn memoization) survived unchanged; the port passes `undefined` instead of an empty +map where a fresh-Map prop would have defeated the base's turn memo. + +Verification on the new base: zero rebase casualties. Runner 1389, API 1597 + 304 DB-backed, +SDK 705, services 101, web suites green; the one real overlap (the sandbox-error rename) +merged with the back-compat alias intact. One latent train bug surfaced by the DB-backed run +(the approval-resume lane typed `SessionInteractionData.request` but an older DAO test still +compared a raw dict) was fixed on its lane. + +Two additions rode the retarget: `feat/agent-file-uploads-default-on` (PR #5633) makes the +uploads flag opt-out per the product owner's rollout call, and `fix/drive-folder-drop-walk` +(PR #5635, closes #5626) adds the directory walk the drawer lacked, stacked on the multipart +transport fix (#5625). The Fern spec on the rebased stack is byte-identical to the one the +committed client was generated from, so no regeneration was needed. diff --git a/docs/design/agent-workflows/projects/agent-multi-modality/status.md b/docs/design/agent-workflows/projects/agent-multi-modality/status.md index bdada05db7..596ec5e3c0 100644 --- a/docs/design/agent-workflows/projects/agent-multi-modality/status.md +++ b/docs/design/agent-workflows/projects/agent-multi-modality/status.md @@ -38,8 +38,8 @@ See [README.md](README.md). In short: [context.md](context.md) for the plain sto | Stage | Scope | State | | --- | --- | --- | -| 0 | Close the silent-failure gap: gate the ungated paste and drag path on `NEXT_PUBLIC_AGENT_FILE_UPLOADS` | not started (optional) | -| 1 | First user-visible release: the attachment resource and storage, the record-schema extension, the runner's resolve-materialize-and-deliver seam for images, structured capability errors, the minimum security and limits work (per the settled matrix in [design.md](design.md), including the gateway raise to 32 MB), and the front-end transport and reference wiring | not started | +| 0 | Close the silent-failure gap: gate the ungated paste and drag path on `NEXT_PUBLIC_AGENT_FILE_UPLOADS` | in review (PR #5604) | +| 1 | First user-visible release: the attachment resource and storage, the record-schema extension, the runner's resolve-materialize-and-deliver seam for images, structured capability errors, the minimum security and limits work (per the settled matrix in [design.md](design.md), including the gateway raise to 32 MB), and the front-end transport and reference wiring | in review as four stacked PRs (#5607 API, #5615 runner, #5617 SDK, #5619 front end), end-to-end QA green on the dev stack; the trail is in [protocols/stage-1.md](protocols/stage-1.md) | | 2 | The audio release: turn on the voice UI, dictation as the only audio-to-text, recordings on the D6 workspace-only path (D14); the document plan (blocked on adapter work), the capability-alias rollout, derived front-end limits | not started (documents blocked on adapter work) | | 3 | Findability polish and cleanup: "Shared by you" origin, reference-counting cleanup refinement, read-only credential scope, verify the edit-then-find flow | not started | @@ -73,10 +73,12 @@ in [plan.md](plan.md) Stage 2. ## Next actions -- Start implementation: Stage 1 of [plan.md](plan.md), optionally preceded by Stage 0. -- Decide whether Stage 0 ships on its own or folds into Stage 1. -- Build the Stage 1 upload route against the settled matrix ([design.md](design.md), "The - media-type, validation, and limits matrix"), including the compose gateway raise to 32 MB. +- Review and merge the Stage 1 train bottom-up (#5604, #5607, #5598, #5615, #5597, #5617, + #5619), then flip `NEXT_PUBLIC_AGENT_FILE_UPLOADS` in production as the rollout's fifth act. +- Follow-ups recorded in [protocols/stage-1.md](protocols/stage-1.md): the CI report glob line + (needs a workflow-scoped push), the Fern client regeneration, and the zip-container + classifier refinement. +- Then Stage 2 of [plan.md](plan.md). ## Artifacts diff --git a/hosting/docker-compose/oss/env.oss.dev.example b/hosting/docker-compose/oss/env.oss.dev.example index 87c393a762..a995d43832 100644 --- a/hosting/docker-compose/oss/env.oss.dev.example +++ b/hosting/docker-compose/oss/env.oss.dev.example @@ -119,6 +119,8 @@ AGENTA_RUNNER_DEFAULT_SANDBOX_PROVIDER=local # the runner flag off, a turn arrives with no history at all. # AGENTA_SESSIONS_RECONSTRUCT=false # NEXT_PUBLIC_SESSIONS_LAST_MESSAGE_ONLY=false +# Agent chat attachments (dev default on; production flips separately) +NEXT_PUBLIC_AGENT_FILE_UPLOADS=true # AGENTA_RECORDS_DURABLE=false # Smart truncation preserves the structure of a record whose body exceeds the API size # cap (higher-fidelity reconstruction). Still opt-in, default off. diff --git a/web/oss/src/components/AgentChatSlice/AgentConversation.tsx b/web/oss/src/components/AgentChatSlice/AgentConversation.tsx index 0a17e33c45..3ebcc73935 100644 --- a/web/oss/src/components/AgentChatSlice/AgentConversation.tsx +++ b/web/oss/src/components/AgentChatSlice/AgentConversation.tsx @@ -28,7 +28,8 @@ import CopiedToast from "@/oss/components/TemplateStrip/components/CopiedToast" import {describeAccepted} from "./assets/attachments" import {CONTENT_VISIBILITY_ENABLED} from "./assets/conversationLayout" -import {filesToParts} from "./assets/files" +import {filesToInlineParts, filesToParts} from "./assets/files" +import {runWithInFlightSubmit} from "./assets/inFlightSubmit" import {isEmptyAssistantTurn, isVisiblePart} from "./assets/messageParts" import {messageText, sideEffectingToolsInRange} from "./assets/rewind" import {ignoreStreamRejection} from "./assets/runError" @@ -107,20 +108,6 @@ const AgentConversation = ({ const composer = useComposerDraft({sessionId, richInputRef, revealPlayedRef}) - // Pending attachments for this session + the whole-panel drop target. - const attachments = useComposerAttachments({sessionId}) - const { - uploadsEnabled, - files, - viewingUid, - setViewingUid, - limits, - atMax, - attachmentsSettled, - isDragging, - addFiles, - } = attachments - // What the transcript should do next (follow / arm a pin / show the pill), shared by the // producers below (send, queue release, history adoption) and whichever scroll engine is // active. Declared here so every producer can state intent without touching the DOM. @@ -222,15 +209,27 @@ const AgentConversation = ({ // component and its logic stay wired up; flip this to `true` to bring the UI back. const showContextBudget = false - /** - * Whether the selected model can actually take audio in. `null` means the catalog does not - * say — treated as unknown, never as "no", so a missing field can't quietly demote voice. - * Drives which voice mode leads; it never refuses an attachment (design decision D6). - */ - const audioPerceivable = useMemo(() => { - const modalities = modalitiesForModel(harnessCapabilities, modelKey.harness, modelKey.model) - return modalities ? modalities.includes("audio") : null - }, [harnessCapabilities, modelKey.harness, modelKey.model]) + const modelModalities = useMemo( + () => modalitiesForModel(harnessCapabilities, modelKey.harness, modelKey.model), + [harnessCapabilities, modelKey.harness, modelKey.model], + ) + // Voice defaults follow the model's audio modality; unknown stays workspace-only, matching + // the runner's rule. + const audioPerceivable = Boolean(modelModalities?.includes("audio")) + + // Pending attachments for this session + the whole-panel drop target. + const attachments = useComposerAttachments({sessionId}) + const { + uploadsEnabled, + files, + viewingUid, + setViewingUid, + limits, + atMax, + attachmentsSettled, + isDragging, + addFiles, + } = attachments // Playground-native onboarding: the hero, Create-agent / Continue-in-IDE, the template strip // and the optimistic first turn. Every value is inert outside the onboarding playground. @@ -401,33 +400,11 @@ const AgentConversation = ({ useVirtuoso, }) - const handleSubmit = async (text: string, extraFiles: File[] = []) => { - const trimmed = text.trim() - const fileObjs = [ - ...files - .map((f) => f.originFileObj as File | undefined) - .filter((f): f is File => Boolean(f)), - ...extraFiles, - ] - if (!trimmed && fileObjs.length === 0) return - if (!attachmentsSettled) return - let fileParts: FileUIPart[] | undefined - if (fileObjs.length) { - const {parts, unreadable} = await filesToParts(fileObjs) - // Hold the send rather than quietly dropping bytes the user staged, and say which file - // failed through the same inline channel the other attachment refusals use. - if (unreadable.length) { - attachments.setRejections( - unreadable.map((f) => ({ - name: f.name, - reason: "couldn't be read — remove it and attach it again", - })), - ) - attachments.setAttachmentsOpen(true) - return - } - fileParts = parts - } + const finishSubmit = ( + trimmed: string, + fileParts: FileUIPart[] | undefined, + consumedUids: string[], + ) => { // Glide to the bottom; the min-h-full active turn makes that show the new question at the top // with the answer streaming below. Park during the glide, follow again on settle. Clear any // prior "stopped" marker — it's resolved by asking again. @@ -438,9 +415,59 @@ const AgentConversation = ({ // The message left the composer — drop its persisted draft (and any pending capture). composer.clearDraft() onboardingChat.consumeTemplateProvenance() - attachments.clearAttachments() + attachments.clearAttachments(consumedUids) } + // A voice take awaits its upload, so the guard keeps a second send from starting meanwhile. + const inFlightSubmitRef = useRef(false) + const handleSubmit = (text: string, extraFiles: File[] = []) => + runWithInFlightSubmit(inFlightSubmitRef, async () => { + const trimmed = text.trim() + if (!trimmed && files.length === 0 && extraFiles.length === 0) return + if (!attachmentsSettled) return + const stagedUids = files.map((file) => file.uid) + + if (!uploadsEnabled) { + // Voice and upload flags are independent; this seam preserves the inline recorder path. + const inlineFiles = [ + ...files + .map((file) => file.originFileObj as File | undefined) + .filter((file): file is File => Boolean(file)), + ...extraFiles, + ] + let fileParts: FileUIPart[] | undefined + if (inlineFiles.length) { + const {parts, unreadable} = await filesToInlineParts(inlineFiles) + // Hold the send rather than quietly dropping bytes the user staged, and say which + // file failed through the same inline channel the other attachment refusals use. + if (unreadable.length) { + attachments.setRejections( + unreadable.map((file) => ({ + name: file.name, + reason: "couldn't be read — remove it and attach it again", + })), + ) + attachments.setAttachmentsOpen(true) + return + } + fileParts = parts + } + finishSubmit(trimmed, fileParts, stagedUids) + return + } + + // A take sent outright never entered the tray, so it uploads here before the send. + const uploadedExtras = extraFiles.length + ? await attachments.uploadExtraFiles(extraFiles) + : [] + if (!uploadedExtras) return + const outboundFiles = [...files, ...uploadedExtras] + const fileParts = outboundFiles.length + ? filesToParts(outboundFiles, sessionId) + : undefined + finishSubmit(trimmed, fileParts, stagedUids) + }) + handleSubmitRef.current = handleSubmit const handleRewind = useCallback( @@ -515,6 +542,7 @@ const AgentConversation = ({ { + if (!sessionId || !attachmentId) return null + try { + // Axios (not Fern): Fern JSON-parses response bodies, mangling binary payloads. + const response = await axios.get(attachmentContentUrl(sessionId, attachmentId), { + responseType: "blob", + }) + return response.data as Blob + } catch { + return null + } +} + +/** Attachment blobs are dropped as soon as the last renderer unmounts. */ +export const attachmentBlobQueryFamily = atomFamily( + ({sessionId, attachmentId}: {sessionId: string; attachmentId: string}) => + atomWithQuery(() => ({ + queryKey: ["sessions", "attachment-blob", sessionId, attachmentId], + queryFn: () => fetchAttachmentBlob({sessionId, attachmentId}), + enabled: Boolean(sessionId && attachmentId), + staleTime: Infinity, + gcTime: 0, + refetchOnWindowFocus: false, + })), + (a, b) => a.sessionId === b.sessionId && a.attachmentId === b.attachmentId, +) + +export function useAttachmentObjectUrl( + sessionId: string | null | undefined, + attachmentId: string | null | undefined, +): {url: string | null; isPending: boolean; failed: boolean} { + const query = useAtomValue( + attachmentBlobQueryFamily({sessionId: sessionId ?? "", attachmentId: attachmentId ?? ""}), + ) + const blob = query.data ?? null + const url = useMemo(() => (blob ? URL.createObjectURL(blob) : null), [blob]) + useEffect(() => { + return () => { + if (url) URL.revokeObjectURL(url) + } + }, [url]) + return {url, isPending: query.isPending, failed: !query.isPending && !blob} +} + +/** Try the direct content URL first, then fall back to an authenticated axios blob. */ +export function useAttachmentMediaSrc( + sessionId: string | null | undefined, + attachmentId: string | null | undefined, +): {src: string | null; isPending: boolean; failed: boolean; onError: () => void} { + const directUrl = + sessionId && attachmentId ? attachmentContentUrl(sessionId, attachmentId) : null + const [mode, setMode] = useState<"direct" | "blob">(directUrl ? "direct" : "blob") + + useEffect(() => { + setMode(directUrl ? "direct" : "blob") + }, [directUrl]) + + const blobQuery = useAtomValue( + attachmentBlobQueryFamily({ + sessionId: mode === "blob" ? (sessionId ?? "") : "", + attachmentId: mode === "blob" ? (attachmentId ?? "") : "", + }), + ) + const blob = mode === "blob" ? (blobQuery.data ?? null) : null + const blobUrl = useMemo(() => (blob ? URL.createObjectURL(blob) : null), [blob]) + useEffect(() => { + return () => { + if (blobUrl) URL.revokeObjectURL(blobUrl) + } + }, [blobUrl]) + + return { + src: mode === "direct" ? directUrl : blobUrl, + isPending: mode === "blob" && blobQuery.isPending, + failed: mode === "blob" && !blobQuery.isPending && !blob, + onError: () => setMode("blob"), + } +} diff --git a/web/oss/src/components/AgentChatSlice/assets/attachmentTransport.test.ts b/web/oss/src/components/AgentChatSlice/assets/attachmentTransport.test.ts new file mode 100644 index 0000000000..4ee78ee1f4 --- /dev/null +++ b/web/oss/src/components/AgentChatSlice/assets/attachmentTransport.test.ts @@ -0,0 +1,188 @@ +// @vitest-environment node +// jsdom's File has no `text()`, and this suite asserts on the multipart body it builds. +import type {AxiosProgressEvent} from "axios" +import {CanceledError} from "axios" +import {beforeEach, describe, expect, it, vi} from "vitest" + +import axios from "@/oss/lib/api/assets/axiosConfig" + +import {AttachmentUploadError, uploadAttachment} from "./attachmentTransport" + +vi.mock("@/oss/lib/api/assets/axiosConfig", () => ({ + default: {post: vi.fn()}, +})) + +vi.mock("@/oss/lib/helpers/api", () => ({ + getAgentaApiUrl: vi.fn(() => "https://api.example.test"), +})) + +const attachmentId = "0198f489-8c20-7000-8000-000000000001" +const idempotencyKey = "0198f489-8c20-7000-8000-000000000002" +const response = { + count: 1, + attachment: { + attachment_id: attachmentId, + filename: "notes.txt", + media_type: "text/plain", + size: 5, + created_at: "2026-07-31T12:00:00Z", + }, +} + +describe("uploadAttachment", () => { + beforeEach(() => { + vi.clearAllMocks() + }) + + it("posts the multipart fields and returns a validated response", async () => { + vi.mocked(axios.post).mockResolvedValue({data: response}) + const file = new File(["hello"], "notes.txt", {type: "text/plain"}) + + await expect( + uploadAttachment({file, sessionId: "session-1", idempotencyKey}), + ).resolves.toEqual(response) + + const [url, body, config] = vi.mocked(axios.post).mock.calls[0] + expect(url).toBe("https://api.example.test/sessions/attachments") + expect(config?.params).toEqual({session_id: "session-1"}) + expect(body).toBeInstanceOf(FormData) + const form = body as FormData + expect(form.get("idempotency_key")).toBe(idempotencyKey) + const uploaded = form.get("file") as File + expect(uploaded.name).toBe("notes.txt") + await expect(uploaded.text()).resolves.toBe("hello") + }) + + it("reports upload progress", async () => { + vi.mocked(axios.post).mockResolvedValue({data: response}) + const onProgress = vi.fn() + + await uploadAttachment({ + file: new File(["hello"], "notes.txt"), + sessionId: "session-1", + idempotencyKey, + onProgress, + }) + + const config = vi.mocked(axios.post).mock.calls[0][2] + config?.onUploadProgress?.({loaded: 1, total: 4} as AxiosProgressEvent) + expect(onProgress).toHaveBeenCalledWith(25) + }) + + it("forwards the abort signal and preserves cancellation", async () => { + const controller = new AbortController() + vi.mocked(axios.post).mockImplementation( + (_url, _body, config) => + new Promise((_resolve, reject) => { + config?.signal?.addEventListener("abort", () => { + reject(new CanceledError("canceled")) + }) + }), + ) + + const upload = uploadAttachment({ + file: new File(["hello"], "notes.txt"), + sessionId: "session-1", + idempotencyKey, + signal: controller.signal, + }) + controller.abort() + + await expect(upload).rejects.toBeInstanceOf(CanceledError) + expect(vi.mocked(axios.post).mock.calls[0][2]?.signal).toBe(controller.signal) + }) + + it("turns a network failure into a retryable user-safe error", async () => { + vi.mocked(axios.post).mockRejectedValue(new Error("socket path and token details")) + + const upload = uploadAttachment({ + file: new File(["hello"], "notes.txt"), + sessionId: "session-1", + idempotencyKey, + }) + + await expect(upload).rejects.toMatchObject({ + name: "AttachmentUploadError", + message: "Couldn't upload the file. Try again.", + retryable: true, + }) + }) + + it.each([ + [413, "text/plain", "This file exceeds the 10.0 MB document limit."], + [422, "text/plain", "This file isn't valid."], + [429, "text/plain", "This session's attachment quota is full."], + [404, "text/plain", "This backend does not support attachments yet."], + ])("maps HTTP %s to a short non-retryable error", async (status, mediaType, message) => { + vi.mocked(axios.post).mockRejectedValue({ + isAxiosError: true, + response: {status, headers: {}}, + }) + + const upload = uploadAttachment({ + file: new File(["hello"], "notes.txt", {type: mediaType}), + sessionId: "session-1", + idempotencyKey, + }) + + await expect(upload).rejects.toMatchObject({message, retryable: false}) + }) + + it("honours Retry-After for an upload already in flight", async () => { + vi.mocked(axios.post).mockRejectedValue({ + isAxiosError: true, + response: {status: 409, headers: {"retry-after": "7"}}, + }) + + const upload = uploadAttachment({ + file: new File(["hello"], "notes.txt"), + sessionId: "session-1", + idempotencyKey, + }) + + await expect(upload).rejects.toMatchObject({ + message: "This file is already uploading. Retry in 7s.", + retryable: true, + retryAfterSeconds: 7, + }) + }) + + it("logs schema drift and throws a controlled retryable error", async () => { + const consoleError = vi.spyOn(console, "error").mockImplementation(() => undefined) + vi.mocked(axios.post).mockResolvedValue({ + data: {...response, attachment: {...response.attachment, attachment_id: "not-a-uuid"}}, + }) + + const upload = uploadAttachment({ + file: new File(["hello"], "notes.txt"), + sessionId: "session-1", + idempotencyKey, + }) + + await expect(upload).rejects.toBeInstanceOf(AttachmentUploadError) + expect(consoleError).toHaveBeenCalledWith( + "[uploadAttachment] Validation failed:", + expect.any(Object), + ) + consoleError.mockRestore() + }) + + it("rejects uppercase attachment ids emitted outside the canonical server contract", async () => { + const consoleError = vi.spyOn(console, "error").mockImplementation(() => undefined) + vi.mocked(axios.post).mockResolvedValue({ + data: { + ...response, + attachment: {...response.attachment, attachment_id: attachmentId.toUpperCase()}, + }, + }) + + const upload = uploadAttachment({ + file: new File(["hello"], "notes.txt"), + sessionId: "session-1", + idempotencyKey, + }) + + await expect(upload).rejects.toBeInstanceOf(AttachmentUploadError) + consoleError.mockRestore() + }) +}) diff --git a/web/oss/src/components/AgentChatSlice/assets/attachmentTransport.ts b/web/oss/src/components/AgentChatSlice/assets/attachmentTransport.ts new file mode 100644 index 0000000000..19ae0dbb82 --- /dev/null +++ b/web/oss/src/components/AgentChatSlice/assets/attachmentTransport.ts @@ -0,0 +1,131 @@ +import {safeParseWithLogging} from "@agenta/entities/shared" +import {isAxiosError, isCancel} from "axios" +import {z} from "zod" + +import axios from "@/oss/lib/api/assets/axiosConfig" +import {getAgentaApiUrl} from "@/oss/lib/helpers/api" + +import {DEFAULT_ATTACHMENT_LIMITS, formatBytes, kindForType} from "./attachments" + +const CANONICAL_UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/ + +const sessionAttachmentResponseSchema = z.object({ + count: z.number().int().nonnegative(), + attachment: z.object({ + // The server emits lowercase ids, and the adapter deliberately rejects uppercase variants. + attachment_id: z.string().regex(CANONICAL_UUID), + filename: z.string(), + media_type: z.string(), + size: z.number().int().nonnegative(), + created_at: z.string(), + }), +}) + +export type SessionAttachmentResponse = z.infer + +export class AttachmentUploadError extends Error { + readonly retryable: boolean + readonly retryAfterSeconds?: number + + constructor( + message = "Couldn't upload the file. Try again.", + { + retryable = true, + retryAfterSeconds, + }: {retryable?: boolean; retryAfterSeconds?: number} = {}, + ) { + super(message) + this.name = "AttachmentUploadError" + this.retryable = retryable + this.retryAfterSeconds = retryAfterSeconds + } +} + +const retryAfterSeconds = (value: unknown): number | undefined => { + const parsed = Number(Array.isArray(value) ? value[0] : value) + return Number.isFinite(parsed) && parsed >= 0 ? Math.ceil(parsed) : undefined +} + +const errorForResponse = (error: unknown, file: File): AttachmentUploadError => { + if (!isAxiosError(error) || !error.response) return new AttachmentUploadError() + + switch (error.response.status) { + case 413: { + const kind = kindForType(file.type || "application/octet-stream") + const limit = formatBytes(DEFAULT_ATTACHMENT_LIMITS.maxBytes[kind]) + const label = kind === "other" ? "file" : kind + return new AttachmentUploadError(`This file exceeds the ${limit} ${label} limit.`, { + retryable: false, + }) + } + case 422: + return new AttachmentUploadError("This file isn't valid.", {retryable: false}) + case 429: + return new AttachmentUploadError("This session's attachment quota is full.", { + retryable: false, + }) + case 409: { + const retryIn = retryAfterSeconds(error.response.headers?.["retry-after"]) + if (retryIn !== undefined) { + return new AttachmentUploadError( + `This file is already uploading. Retry in ${retryIn}s.`, + {retryAfterSeconds: retryIn}, + ) + } + return new AttachmentUploadError("This upload conflicts with an earlier file.", { + retryable: false, + }) + } + case 404: + return new AttachmentUploadError("This backend does not support attachments yet.", { + retryable: false, + }) + default: + return new AttachmentUploadError() + } +} + +export async function uploadAttachment({ + file, + sessionId, + idempotencyKey, + onProgress, + signal, +}: { + file: File + sessionId: string + idempotencyKey: string + onProgress?: (percent: number) => void + signal?: AbortSignal +}): Promise { + const form = new FormData() + form.append("file", file, file.name) + form.append("idempotency_key", idempotencyKey) + + try { + // Axios (not Fern): the Fern client uses fetch, which can't stream upload progress. + // The explicit header matters: the shared instance defaults to application/json, and + // axios then JSON-serializes the FormData, collapsing the File to {}. + const response = await axios.post(`${getAgentaApiUrl()}/sessions/attachments`, form, { + params: {session_id: sessionId}, + headers: {"Content-Type": "multipart/form-data"}, + signal, + onUploadProgress: (event) => { + if (onProgress && event.total) { + onProgress(Math.round((event.loaded / event.total) * 100)) + } + }, + }) + const validated = safeParseWithLogging( + sessionAttachmentResponseSchema, + response.data, + "[uploadAttachment]", + ) + if (!validated) throw new AttachmentUploadError() + return validated + } catch (error) { + if (signal?.aborted || isCancel(error)) throw error + if (error instanceof AttachmentUploadError) throw error + throw errorForResponse(error, file) + } +} diff --git a/web/oss/src/components/AgentChatSlice/assets/attachments.test.ts b/web/oss/src/components/AgentChatSlice/assets/attachments.test.ts new file mode 100644 index 0000000000..0f1bfde2e6 --- /dev/null +++ b/web/oss/src/components/AgentChatSlice/assets/attachments.test.ts @@ -0,0 +1,44 @@ +import {describe, expect, it} from "vitest" + +import {DEFAULT_ATTACHMENT_LIMITS, kindForType, validateIncoming} from "./attachments" + +const MB = 1024 * 1024 + +const file = (name: string, type: string, size: number): File => ({name, type, size}) as File + +describe("attachment validation", () => { + it("accepts an unrecognized media type through the other bucket", () => { + const archive = file("bundle.zip", "application/zip", 1) + + expect(kindForType(archive.type)).toBe("other") + expect(validateIncoming([archive], 0)).toEqual({accepted: [archive], rejections: []}) + }) + + it("enforces the 10 MB cap for the other bucket", () => { + const archive = file("bundle.zip", "application/zip", 10 * MB + 1) + + const result = validateIncoming([archive], 0) + + expect(result.accepted).toEqual([]) + expect(result.rejections).toEqual([ + { + name: "bundle.zip", + reason: "is too large (10.0 MB) · max 10.0 MB for other files", + }, + ]) + }) + + it("enforces the per-message count cap", () => { + const image = file("photo.png", "image/png", 1) + + const result = validateIncoming([image], DEFAULT_ATTACHMENT_LIMITS.maxCount) + + expect(result.accepted).toEqual([]) + expect(result.rejections).toEqual([ + { + name: "photo.png", + reason: `exceeds the ${DEFAULT_ATTACHMENT_LIMITS.maxCount}-file limit`, + }, + ]) + }) +}) diff --git a/web/oss/src/components/AgentChatSlice/assets/attachments.ts b/web/oss/src/components/AgentChatSlice/assets/attachments.ts index ab4533acfa..75c5aaa696 100644 --- a/web/oss/src/components/AgentChatSlice/assets/attachments.ts +++ b/web/oss/src/components/AgentChatSlice/assets/attachments.ts @@ -1,25 +1,13 @@ -/** - * Attachment guardrails for the agent composer. Files are sent inline as base64 `data:` URLs - * (see `files.ts`), so an unbounded picker puts arbitrary bytes straight into the request body. - * These limits cap the count, per-kind size, and types. - * - * NOTE on ceilings: while attachments ride the request body, the real limit is the gateway's - * `client_max_body_size` (10 MB on the compose stack), and base64 inflates by ~33% on top of the - * whole resent history. These caps are deliberately per-kind rather than generous. They can rise - * once attachments travel as references instead of bytes. - * - * The limits are a single value object, not scattered constants, so they can be derived from the - * selected model / harness capabilities and passed down in place of `DEFAULT_ATTACHMENT_LIMITS` — - * narrowing `kinds` is the seam that capability gating plugs into. - */ +/** Attachment guardrails for files uploaded before an agent turn is sent by reference. */ -export type AttachmentKind = "image" | "audio" | "document" +export type AttachmentKind = "image" | "audio" | "document" | "other" /** Media types per kind: exact types (`application/pdf`) or `type/` prefixes (`image/`). */ const KIND_TYPES: Record = { image: ["image/"], audio: ["audio/"], document: ["application/pdf", "text/", "application/json"], + other: [], } /** `accept` hints for the native picker (a hint only — drag/paste is validated regardless). */ @@ -27,12 +15,14 @@ const KIND_ACCEPT_ATTR: Record = { image: "image/*", audio: "audio/*", document: "application/pdf,text/plain,text/markdown,text/csv,.md,.csv,application/json", + other: "", } const KIND_NOUN: Record = { image: "images", audio: "audio", document: "documents", + other: "other files", } export interface AttachmentLimits { @@ -40,44 +30,45 @@ export interface AttachmentLimits { maxCount: number /** Kinds the composer accepts. Narrowing this is how capability gating plugs in. */ kinds: AttachmentKind[] - /** Max bytes per file, per kind (before base64 inflation, which adds ~33% on the wire). */ + /** Max bytes per file, per kind. */ maxBytes: Record } const MB = 1024 * 1024 export const DEFAULT_ATTACHMENT_LIMITS: AttachmentLimits = { - maxCount: 5, - kinds: ["image", "audio", "document"], + maxCount: 100, + kinds: ["image", "audio", "document", "other"], maxBytes: { // A photo off a phone clears 5 MB routinely. image: 10 * MB, // Our own recordings cap near 2.4 MB; the headroom is for uploaded clips. audio: 15 * MB, document: 10 * MB, + other: 10 * MB, }, } -/** Which kind a media type belongs to, or null when it is not something we take at all. */ -export const kindForType = (mediaType: string): AttachmentKind | null => { - for (const kind of Object.keys(KIND_TYPES) as AttachmentKind[]) { +/** Which kind a media type belongs to. */ +export const kindForType = (mediaType: string): AttachmentKind => { + for (const kind of ["image", "audio", "document"] as const) { const matches = KIND_TYPES[kind].some((t) => t.endsWith("/") ? mediaType.startsWith(t) : mediaType === t, ) if (matches) return kind } - return null + return "other" } /** Whether a media type is allowed under the limits (right kind, and that kind is enabled). */ export const isAcceptedType = (mediaType: string, limits: AttachmentLimits): boolean => { const kind = kindForType(mediaType) - return !!kind && limits.kinds.includes(kind) + return limits.kinds.includes(kind) } /** `accept` attribute for the native file picker, built from the enabled kinds. */ export const acceptAttrFor = (limits: AttachmentLimits): string => - limits.kinds.map((k) => KIND_ACCEPT_ATTR[k]).join(",") + limits.kinds.includes("other") ? "" : limits.kinds.map((k) => KIND_ACCEPT_ATTR[k]).join(",") /** Human summary of what is accepted, e.g. "Images, audio, and documents". */ export const describeAccepted = (limits: AttachmentLimits): string => { @@ -127,7 +118,7 @@ export const validateIncoming = ( const type = file.type || "application/octet-stream" const kind = kindForType(type) - if (!kind || !limits.kinds.includes(kind)) { + if (!limits.kinds.includes(kind)) { rejections.push({name: file.name, reason: "isn't a supported file type"}) continue } diff --git a/web/oss/src/components/AgentChatSlice/assets/constants.ts b/web/oss/src/components/AgentChatSlice/assets/constants.ts index 068293bf47..f518db383c 100644 --- a/web/oss/src/components/AgentChatSlice/assets/constants.ts +++ b/web/oss/src/components/AgentChatSlice/assets/constants.ts @@ -36,8 +36,7 @@ export const isAgentVoiceInputEnabled = (): boolean => * File uploads and attachments — the composer attach button + attachment preview, and every drive * upload entry point (upload button, drop-to-upload in the Files drawer, drop-to-stage on a recents * peek). Off by default: the composer→model delivery contract is still open on the backend. Enable - * with `NEXT_PUBLIC_AGENT_FILE_UPLOADS=true`. Paste/drag-to-attach on the composer predates this and - * is not gated. + * with `NEXT_PUBLIC_AGENT_FILE_UPLOADS=true`. The composer gates button, paste, and drag paths. */ export const isAgentFileUploadsEnabled = (): boolean => (getEnv("NEXT_PUBLIC_AGENT_FILE_UPLOADS") || "").toLowerCase() === "true" diff --git a/web/oss/src/components/AgentChatSlice/assets/files.test.ts b/web/oss/src/components/AgentChatSlice/assets/files.test.ts new file mode 100644 index 0000000000..479e067642 --- /dev/null +++ b/web/oss/src/components/AgentChatSlice/assets/files.test.ts @@ -0,0 +1,48 @@ +import type {UploadFile} from "antd" +import {describe, expect, it} from "vitest" + +import type {SessionAttachmentResponse} from "./attachmentTransport" +import {filePartName, filesToParts} from "./files" + +const firstAttachmentId = "019c1e0a-f911-7000-8000-000000000001" + +const upload = (attachmentId: string): UploadFile => ({ + uid: attachmentId, + name: "notes.txt", + status: "done", + response: { + count: 1, + attachment: { + attachment_id: attachmentId, + filename: "notes.txt", + media_type: "text/plain", + size: 42, + created_at: "2026-07-31T12:00:00Z", + }, + }, +}) + +describe("attachment file parts", () => { + it("stores size under providerMetadata.agenta", () => { + const part = filesToParts([upload(firstAttachmentId)], "session-1")[0] + + expect(part).toMatchObject({ + type: "file", + filename: "notes.txt", + providerMetadata: { + agenta: {attachmentId: firstAttachmentId, size: 42}, + }, + }) + expect(part).not.toHaveProperty("size") + }) + + it("uses attachment when a replayed part has no filename", () => { + expect( + filePartName({ + type: "file", + mediaType: "text/plain", + url: "https://api.example.test/attachments/id/content", + }), + ).toBe("attachment") + }) +}) diff --git a/web/oss/src/components/AgentChatSlice/assets/files.ts b/web/oss/src/components/AgentChatSlice/assets/files.ts index 243664027b..b31f1e1db8 100644 --- a/web/oss/src/components/AgentChatSlice/assets/files.ts +++ b/web/oss/src/components/AgentChatSlice/assets/files.ts @@ -1,15 +1,14 @@ import type {FileUIPart, UIMessage} from "ai" +import type {UploadFile} from "antd" -/** - * Multi-modality helpers for the agent chat slice. Attachments are kept entirely on the - * client: there is no upload server, so a selected file is read into a `data:` URL and - * sent inline as an AI SDK v6 `file` part (`{type, mediaType, filename, url}`). The service - * receives the bytes in the request body — same channel as the text. - */ +import {attachmentContentUrl} from "./attachmentMedia" +import type {SessionAttachmentResponse} from "./attachmentTransport" + +/** Helpers for sending and rendering attachment references without embedding file bytes. */ export type FileKind = "image" | "audio" | "video" | "file" -/** Map an IANA media type to the `FileCard` `type` / a render branch. */ +/** Map an IANA media type to the `FileCard` `type` or a render branch. */ export const fileKind = (mediaType: string): FileKind => { if (mediaType.startsWith("image/")) return "image" if (mediaType.startsWith("audio/")) return "audio" @@ -17,6 +16,25 @@ export const fileKind = (mediaType: string): FileKind => { return "file" } +/** Convert uploaded tray entries into reference-carrying AI SDK file parts. */ +export const filesToParts = ( + files: UploadFile[], + sessionId: string, +): FileUIPart[] => + files.map((file) => { + const attachment = file.response?.attachment + if (!attachment) throw new Error(`Attachment upload is incomplete: ${file.name}`) + return { + type: "file", + mediaType: attachment.media_type, + filename: attachment.filename, + url: attachmentContentUrl(sessionId, attachment.attachment_id), + providerMetadata: { + agenta: {attachmentId: attachment.attachment_id, size: attachment.size}, + }, + } + }) + /** Read one `File` into a `data:` URL `file` part. */ const fileToPart = (file: File): Promise => new Promise((resolve, reject) => { @@ -39,13 +57,13 @@ export interface FileReadResult { } /** - * Convert picked `File`s into `file` parts for `sendMessage({text, files})`. + * Preserve the pre-upload voice path by reading recorder files into inline data URLs. * * Never rejects. Every send path drops this promise (composer submit, voice take, empty-state * Start, the first-run seed), so a `Promise.all` that threw on one unreadable file surfaced as an * unhandled rejection and a send that silently did nothing. Failures come back as data instead. */ -export const filesToParts = async (files: File[]): Promise => { +export const filesToInlineParts = async (files: File[]): Promise => { const settled = await Promise.allSettled(files.map(fileToPart)) const parts: FileUIPart[] = [] const unreadable: File[] = [] @@ -58,8 +76,15 @@ export const filesToParts = async (files: File[]): Promise => { /** The `file` parts of a message, in order. */ export const fileParts = (message: UIMessage): FileUIPart[] => - message.parts.filter((p) => p.type === "file") as FileUIPart[] + message.parts.filter((part) => part.type === "file") as FileUIPart[] + +/** The Agenta attachment id carried by a reference part, if present. */ +export const attachmentIdForPart = (part: FileUIPart): string | null => { + const agenta = part.providerMetadata?.agenta + if (!agenta || typeof agenta !== "object") return null + const attachmentId = (agenta as {attachmentId?: unknown}).attachmentId + return typeof attachmentId === "string" && attachmentId ? attachmentId : null +} -/** A readable label for a file part (filename, else the tail of its URL). */ -export const filePartName = (part: FileUIPart): string => - part.filename || part.url.split("/").pop()?.split("?")[0] || "file" +/** A readable label for a file part. */ +export const filePartName = (part: FileUIPart): string => part.filename || "attachment" diff --git a/web/oss/src/components/AgentChatSlice/assets/inFlightSubmit.test.ts b/web/oss/src/components/AgentChatSlice/assets/inFlightSubmit.test.ts new file mode 100644 index 0000000000..cff8bff809 --- /dev/null +++ b/web/oss/src/components/AgentChatSlice/assets/inFlightSubmit.test.ts @@ -0,0 +1,25 @@ +import {describe, expect, it, vi} from "vitest" + +import {runWithInFlightSubmit} from "./inFlightSubmit" + +describe("in-flight submit guard", () => { + it("drops a second submit while the first await is unresolved", async () => { + let release!: () => void + const pending = new Promise((resolve) => { + release = resolve + }) + const task = vi.fn(() => pending) + const inFlight = {current: false} + + const first = runWithInFlightSubmit(inFlight, task) + const second = runWithInFlightSubmit(inFlight, task) + + await expect(second).resolves.toBeUndefined() + expect(task).toHaveBeenCalledTimes(1) + + release() + await first + await runWithInFlightSubmit(inFlight, task) + expect(task).toHaveBeenCalledTimes(2) + }) +}) diff --git a/web/oss/src/components/AgentChatSlice/assets/inFlightSubmit.ts b/web/oss/src/components/AgentChatSlice/assets/inFlightSubmit.ts new file mode 100644 index 0000000000..f7ff034a90 --- /dev/null +++ b/web/oss/src/components/AgentChatSlice/assets/inFlightSubmit.ts @@ -0,0 +1,17 @@ +export interface InFlightSubmitRef { + current: boolean +} + +/** Admit one async submit at a time and always release the guard when it settles. */ +export async function runWithInFlightSubmit( + inFlight: InFlightSubmitRef, + task: () => Promise, +): Promise { + if (inFlight.current) return undefined + inFlight.current = true + try { + return await task() + } finally { + inFlight.current = false + } +} diff --git a/web/oss/src/components/AgentChatSlice/assets/messageParts.ts b/web/oss/src/components/AgentChatSlice/assets/messageParts.ts index 7e56c4c56a..3b50fb9923 100644 --- a/web/oss/src/components/AgentChatSlice/assets/messageParts.ts +++ b/web/oss/src/components/AgentChatSlice/assets/messageParts.ts @@ -1,6 +1,6 @@ import {type UIMessage} from "ai" -/** A part the transcript actually renders — non-empty text/reasoning, files, sources, tools. */ +/** A part the transcript renders: non-empty prose, files, sources, or tools. */ export const isVisiblePart = (p: UIMessage["parts"][number]): boolean => (p.type === "text" && Boolean((p as {text?: string}).text?.trim())) || (p.type === "reasoning" && Boolean((p as {text?: string}).text?.trim())) || diff --git a/web/oss/src/components/AgentChatSlice/assets/transcriptToMessages.test.ts b/web/oss/src/components/AgentChatSlice/assets/transcriptToMessages.test.ts index a07f630839..333fd9fdb6 100644 --- a/web/oss/src/components/AgentChatSlice/assets/transcriptToMessages.test.ts +++ b/web/oss/src/components/AgentChatSlice/assets/transcriptToMessages.test.ts @@ -351,3 +351,62 @@ describe("transcriptToMessages turn growth is invisible to a message count", () expect(full?.[0].parts.length).toBeGreaterThan(partial?.[0].parts.length ?? 0) }) }) + +describe("transcriptToMessages attachments", () => { + it("rebuilds user attachment references as file parts with filenames", () => { + const messages = transcriptToMessages([ + record( + "record-user", + { + type: "message", + text: "Inspect this", + attachments: [ + { + attachmentId: "019c1e0a-f911-7000-8000-000000000001", + filename: "diagram.png", + mediaType: "image/png", + size: 42, + }, + ], + }, + "user", + ), + ]) + + expect(messages?.[0]).toMatchObject({ + role: "user", + parts: [ + {type: "text", text: "Inspect this"}, + { + type: "file", + filename: "diagram.png", + mediaType: "image/png", + providerMetadata: { + agenta: {attachmentId: "019c1e0a-f911-7000-8000-000000000001", size: 42}, + }, + }, + ], + }) + expect((messages?.[0].parts[1] as {url: string}).url).toContain( + "/sessions/attachments/019c1e0a-f911-7000-8000-000000000001/content?session_id=session-1", + ) + }) + + it("ignores an attachment delivery record instead of rendering a part", () => { + const delivery = record("record-delivery", { + type: "attachment_delivery", + attachmentId: "019c1e0a-f911-7000-8000-000000000001", + outcome: "workspace_only", + reasonCode: "model_modality_unknown", + workingPath: "attachments/019c1e0a-f911-7000-8000-000000000001/archive.zip", + }) + + expect(transcriptToMessages([delivery])).toBeNull() + expect( + transcriptToMessages([ + record("record-text", {type: "message", text: "Done."}), + delivery, + ])?.[0].parts, + ).toEqual([{type: "text", text: "Done."}]) + }) +}) diff --git a/web/oss/src/components/AgentChatSlice/assets/transcriptToMessages.ts b/web/oss/src/components/AgentChatSlice/assets/transcriptToMessages.ts index bf12c4289a..c83c12c02d 100644 --- a/web/oss/src/components/AgentChatSlice/assets/transcriptToMessages.ts +++ b/web/oss/src/components/AgentChatSlice/assets/transcriptToMessages.ts @@ -1,6 +1,8 @@ import type {SessionRecord} from "@agenta/entities/session" import type {UIMessage} from "ai" +import {attachmentContentUrl} from "./attachmentMedia" + /** * Replay adapter — durable session-record `AgentEvent`s → v6 `UIMessage[]`. * @@ -105,6 +107,7 @@ function applyEvent( draft: DraftMessage, payload: Record, index: TranscriptIndex, + sessionId: string, ): void { const type = payload.type as string | undefined const str = (v: unknown): string => (typeof v === "string" ? v : v == null ? "" : String(v)) @@ -112,6 +115,25 @@ function applyEvent( switch (type) { case "message": { draft.parts.push({type: "text", text: str(payload.text)}) + const attachments = Array.isArray(payload.attachments) ? payload.attachments : [] + for (const raw of attachments) { + if (!raw || typeof raw !== "object") continue + const attachment = raw as Record + const attachmentId = str(attachment.attachmentId) + if (!attachmentId) continue + draft.parts.push({ + type: "file", + url: attachmentContentUrl(sessionId, attachmentId), + mediaType: str(attachment.mediaType) || "application/octet-stream", + filename: str(attachment.filename) || undefined, + providerMetadata: { + agenta: { + attachmentId, + size: typeof attachment.size === "number" ? attachment.size : undefined, + }, + }, + }) + } return } case "message_start": { @@ -236,6 +258,7 @@ function applyEvent( type: "file", url: str(payload.url), mediaType: str(payload.mediaType), + filename: str(payload.filename) || undefined, }) return } @@ -261,7 +284,7 @@ function applyEvent( draft.usage = next return } - // done / data / render-hints carry no renderable message part — drop. + // done / data / render-hints / attachment_delivery carry no renderable message part — drop. default: return } @@ -314,7 +337,7 @@ export function transcriptToMessages(records: SessionRecord[]): UIMessage[] | nu drafts.push(current) } if (traceId && !current.traceId) current.traceId = traceId - applyEvent(current, p, index) + applyEvent(current, p, index, row.session_id) } const messages = drafts diff --git a/web/oss/src/components/AgentChatSlice/components/AgentComposerDock.tsx b/web/oss/src/components/AgentChatSlice/components/AgentComposerDock.tsx index a558558908..cd68b54a64 100644 --- a/web/oss/src/components/AgentChatSlice/components/AgentComposerDock.tsx +++ b/web/oss/src/components/AgentChatSlice/components/AgentComposerDock.tsx @@ -139,6 +139,7 @@ const AgentComposerDock = ({ limits, atMax, attachmentsSettled, + uploadBlockReason, addFiles, removeFile, uploads, @@ -284,6 +285,8 @@ const AgentComposerDock = ({ if (!attachmentsBlocked()) addFiles(Array.from(pasted)) }} sendForceEnabled={files.length > 0 && attachmentsSettled} + sendDisabled={files.length > 0 && !attachmentsSettled} + sendDisabledReason={uploadBlockReason} streaming={busy} onStop={onStop} prefix={ @@ -354,12 +357,12 @@ const AgentComposerDock = ({ files={files} rejections={rejections} limits={limits} - audioPerceivable={audioPerceivable} onAdd={addFiles} onRemove={removeFile} onDismissRejections={() => setRejections([])} onView={uploadsEnabled ? setViewingUid : undefined} onRetry={uploads.retry} + canRetry={uploads.canRetry} /> } diff --git a/web/oss/src/components/AgentChatSlice/components/AgentMessage.tsx b/web/oss/src/components/AgentChatSlice/components/AgentMessage.tsx index 1420c6ff3b..7e4095de6d 100644 --- a/web/oss/src/components/AgentChatSlice/components/AgentMessage.tsx +++ b/web/oss/src/components/AgentChatSlice/components/AgentMessage.tsx @@ -1,4 +1,4 @@ -import {memo, useMemo, useRef, useState} from "react" +import {memo, useEffect, useMemo, useRef, useState} from "react" import {traceDataSummaryAtomFamily} from "@agenta/entities/loadable" import {buildRenderMap} from "@agenta/playground" @@ -23,7 +23,8 @@ import {useAtomValue, useSetAtom} from "jotai" import {openTraceDrawerAtom} from "@/oss/components/SharedDrawers/TraceDrawer/store/traceDrawerStore" -import {fileKind, filePartName} from "../assets/files" +import {useAttachmentMediaSrc} from "../assets/attachmentMedia" +import {attachmentIdForPart, fileKind, filePartName} from "../assets/files" import Markdown from "../assets/markdown" import { getMessageRunError, @@ -89,6 +90,7 @@ const TraceMetrics = ({traceId, usage}: {traceId: string; usage?: MessageUsageMe interface AgentMessageProps { message: UIMessage + sessionId: string /** This is the last message AND the conversation is streaming — i.e. the one being * generated right now. Only it shows the loading state; settled turns never do. */ isStreaming?: boolean @@ -228,6 +230,93 @@ const avatarFor = (isUser: boolean) => ( : } /> ) +const triggerDownload = (href: string, name: string) => { + const link = document.createElement("a") + link.href = href + link.download = name + link.hidden = true + document.body.append(link) + link.click() + link.remove() +} + +const AttachmentFilePart = ({file, sessionId}: {file: FileUIPart; sessionId: string}) => { + const attachmentId = attachmentIdForPart(file) + const kind = fileKind(file.mediaType) + const source = useAttachmentMediaSrc(attachmentId ? sessionId : null, attachmentId) + const src = attachmentId ? source.src : file.url + const name = filePartName(file) + const [fallbackDownloadPending, setFallbackDownloadPending] = useState(false) + + useEffect(() => { + if (!fallbackDownloadPending) return + if (source.src?.startsWith("blob:")) { + triggerDownload(source.src, name) + setFallbackDownloadPending(false) + } else if (source.failed) { + setFallbackDownloadPending(false) + } + }, [fallbackDownloadPending, name, source.failed, source.src]) + + const handleDownload = async (event: React.MouseEvent) => { + if (!attachmentId || !src || src.startsWith("blob:")) return + event.preventDefault() + try { + const response = await fetch(src, {credentials: "include"}) + if (!response.ok) throw new Error("Direct attachment download failed") + const objectUrl = URL.createObjectURL(await response.blob()) + triggerDownload(objectUrl, name) + window.setTimeout(() => URL.revokeObjectURL(objectUrl), 1_000) + } catch { + // The direct route can lack browser credentials; activate the lazy axios/blob fallback. + setFallbackDownloadPending(true) + source.onError() + } + } + + if (kind === "audio") { + return ( + + ) + } + + return ( + + {file.mediaType} + + ) : ( + + {source.failed ? "Download unavailable" : file.mediaType} + + ) + ) : undefined + } + /> + ) +} + /** * Read-only renderer for one agent conversation message, rendered inside an Ant Design X * `Bubble`. Walks `message.parts` in order (text → markdown, reasoning, tool calls + @@ -236,6 +325,7 @@ const avatarFor = (isUser: boolean) => ( */ const AgentMessage = ({ message, + sessionId, isStreaming = false, isLastMessage = false, onRewind, @@ -445,39 +535,8 @@ const AgentMessage = ({ // agent) as X `FileCard`s — images preview inline, other kinds show a typed // file chip with a download link. if (part.type === "file") { - const file = part as FileUIPart - const kind = fileKind(file.mediaType) - // A voice message is playable in the transcript, not an inert card. - if (kind === "audio") { - return ( - - ) - } return ( - - {file.mediaType} - - ) : undefined - } - /> + ) } return null diff --git a/web/oss/src/components/AgentChatSlice/components/AgentTurn.tsx b/web/oss/src/components/AgentChatSlice/components/AgentTurn.tsx index 026c12b350..f504d90eaa 100644 --- a/web/oss/src/components/AgentChatSlice/components/AgentTurn.tsx +++ b/web/oss/src/components/AgentChatSlice/components/AgentTurn.tsx @@ -11,6 +11,7 @@ import {WaitingForInput, WorkingDots} from "./TurnActivity" interface AgentTurnProps { message: UIMessage + sessionId: string /** Fade in once — this turn arrived after mount. */ enter: boolean isLast: boolean @@ -47,6 +48,7 @@ interface AgentTurnProps { */ const AgentTurn = ({ message, + sessionId, enter, isLast, isStreaming, @@ -77,6 +79,7 @@ const AgentTurn = ({ > { * in the composer tray before sending, and in the transcript afterwards — so both surfaces share * this rather than showing an inert file chip. */ -const AudioPlayer = ({src, name, className}: {src: string; name: string; className?: string}) => { +const AudioPlayer = ({ + src, + name, + className, + onError, +}: { + src: string + name: string + className?: string + onError?: () => void +}) => { const audioRef = useRef(null) // True while we nudge the element to resolve an unknown duration (below); the seek would // otherwise show up as a wild current-time reading. @@ -113,7 +123,13 @@ const AudioPlayer = ({src, name, className}: {src: string; name: string; classNa /> -