perf(desktop): keep the app smooth while a chat answer streams

[Git-on-my-level]

Summary

While an answer streams in the main chat, the whole app turns slow and stuttery; the moment the turn settles it is smooth again. I measured it first, then removed every per-flush cost that scaled with the answer instead of with the flush.

The streaming buffer flushes every 35 ms. Before this change each flush did all of the following on the main thread, over the whole accumulated answer:

1. Two Foundation Markdown parses of the full answer, one of them discarded. OmiMarkdownContent.textView ran inlineCopyContent/styledAttributedString before checking appKitProseSelection, so the live transcript (which always draws through ChatSelectableProse) paid a SwiftUI parse it never rendered.
2. A whole-answer NSTextStorage.setAttributedString on the live NSTextView, and a throwaway TextKit stack laying the whole answer out again in sizeThatFits (the render cache misses on every flush, since the key is the full text).
3. Hundreds of NSRegularExpression compiles per flush: applySentenceSpacing compiled two patterns per line, preprocessText two more per line, and the answer was normalized twice (message text and its text block), plus an Array(text) copy of the answer in strippingPendingTail and a quadratic tilde scan (first(where:) over every code span for every character).
4. The visible answer derived five to seven times per row per flush. ChatAssistantAnswerText.visible unconditionally built the block projection of every block — tool outputs included — and whitespace-normalized it plus the body, and ChatBubble re-derived it at each site that read it; the transcript also derived it for every mounted row on every body pass through ChatBubbleMetadataBand. Sampled on a real streaming turn this was the single largest main-thread cost (25% of samples).
5. A journal write of the whole row per flush, queued without bound, each echoing an older snapshot back into the transcript. ChatJournalWriteCoordinator chained one task per flush, so whenever a kernel round trip took longer than 35 ms the queue grew for the rest of the answer and drained only after it settled (exactly the "smooth once the response is done" signature). Each completed write then re-listed the journal and projectJournalTurns republished the entire transcript with the echo's older text, so the visible answer stepped back a few words and forward again on the next flush.

What changed

• ChatSelectableProseText edits the live text storage from the first character that differs (in characters or attributes), and answers sizeThatFits from the live view's own incremental layout when the container is already at the proposed width; the throwaway measurement remains the fallback for any other width. The view is created as TextKit 1 explicitly so the live and throwaway heights are the same number (asserted).
• OmiMarkdownContent.textView no longer runs the SwiftUI prose path for the AppKit branch. preprocessText applies its two rules by hand; the sentence-spacing and citation regexes are compiled once; the tilde scan walks code spans with one cursor; the table parser rejects lines with no | before splitting them into cells; strippingPendingTail inspects only the suffix; the streaming buffer projects a single text block once.
• ChatAssistantAnswerText.visible derives the block-projection fallback only on the paths that return it, and hasVisible answers !isEmpty without building the text. ChatBubble derives its row text once per body evaluation.
• The sentence-spacing normalizer moved to ChatProvider+AssistantTextNormalization.swift and carryingLocalOnlyFields to ChatProvider+JournalProjection.swift, so ChatProvider.swift shrinks under the agent-runtime convergence ceiling instead of growing.
• ChatJournalWriteCoordinator.schedule(coalescing:): a streaming write replaces the waiting one instead of queueing behind it (at most one in flight, one waiting, always the newest row); durable writes keep their place in line; cancelAll drops a waiting snapshot. carryingLocalOnlyFields keeps the live row's text and blocks when a streaming echo is behind it; projectJournalTurns publishes only when a row actually changed.

INV-6 is unchanged: one ChatProvider, kernel journal turns remain the durable source of truth (a terminal or kernel-owned row is never streaming and is taken whole; the coalesced streaming write still lands before terminalization), and no SwiftUI text selection enters the live transcript (check_chat_selection_boundary.py passes).

Measurements

All numbers below are from this branch on this Mac (Apple Silicon, macOS 26.6, debug builds). Details and raw profiles are in the session record.

Hermetic mounted transcript (ChatStreamingRenderBudgetTests, xcrun swift test --package-path Desktop --filter ChatStreamingRenderBudgetTests): 20-row history with tool-call blocks, one row streaming a 6,777-character Markdown answer in 142 flushes of 48 characters at the buffer's 35 ms cadence. Main-thread CPU per flush, and the same cadence with nothing arriving ("idle", the streaming mark still animating):

Build	per-flush CPU p50 / p95 / max (ms)	idle CPU per 35 ms (ms)	parses per flush	storage replacements	throwaway layouts
origin/main (plain-prose history)	30.9 / 37.0 / 49.1	—	2 (1 discarded)	140	143
text fixes only, mark still a TimelineView	40.4 / 43.7 / 44.5	39.6	1	0	3
this branch	13.9–16.8 / 18–22 / 30–41	1.0	1	0	3

The idle row is the headline: while a row streams, the mark's TimelineView alone kept the main thread fully busy (39.6 ms of every 35 ms) with no text arriving. The remaining ~15 ms per flush is SwiftUI/AppKit layout of the hosting view plus one incremental TextKit edit and one Markdown parse of the answer.

Live app (named bundle omi-chat-stream-perf, signed in, dev backend, real model; the same 1,500-word Markdown prompt through ask_main_chat_no_wait; main-actor round trips of the automation bridge's /state timed every ~50 ms while the answer streamed, plus sample of the process):

Build	answer	bridge hop while streaming p50 / p95 / max (ms)	hops > 50 ms	main thread busy during sample
origin/main	15,881 chars in 60 s	15.2 / 40.9 / 55.7	8 of 791	~72%
this branch	15,670 chars in 28 s	14.1 / 24.0 / 51.6	1 of 407	~44%

Idle hops are 0.5–0.9 ms on both. In the baseline profile the top main-thread costs were visibleAnswerText/bodyIsBlockProjection (25% of samples), SwiftUI graph updates from the per-flush transcript rebuild, normalizeAssistantSentenceSpacing with per-line NSRegularExpression compiles (8%), the discarded SwiftUI parse, and the throwaway TextKit layout. After: what remains is the Foundation Markdown parse of the whole answer per flush (~4.5%), sentence-spacing normalization over the whole answer (~2.5%), hosting-view layout, and Core Animation re-uploading the tall text view's backing store. Those are follow-ups, not regressions.

Note on the fast-lane bundle: a re-signed named bundle blocks its main thread inside SecItemCopyMatching on a keychain ACL prompt for the old signature; re-running omi-auth-seed.sh before relaunch avoids it. Unrelated to this change; recorded so the next measurement does not lose an hour to it.

Tests

• ChatStreamingRenderBudgetTests (new): mounts the production transcript through the gesture harness with tool-call-bearing history, streams a 6.8k-char Markdown answer in 142 buffer-sized flushes, and pins the per-flush work through ChatStreamingRenderProbe (DEBUG-only counters): no SwiftUI parse on the AppKit path, ≤1 whole-storage replacement, incremental edits for every later flush, ≤8 throwaway layouts, live-layout heights. Main-thread CPU per flush is printed, not asserted.
• ChatStreamingJournalCoalescingTests (new): 50 flushes behind one in-flight write execute as two writes (first, newest); durable writes keep ordering; cancelAll drops a waiting snapshot; a streaming echo behind the live row neither moves the text back nor republishes; a terminal replay still replaces the row.
• ChatSelectableProseIncrementalEditTests (new): append edits only the tail; a closing ** re-styles earlier words and the storage still equals the new prose; identical updates do not touch the storage; composed characters are never split; live height equals the throwaway height at three widths and declines other widths; preprocessText matches the original regular-expression rules on an edge-case oracle.

Verification

• xcrun swift test --package-path Desktop --filter 'ChatStreamingRenderBudgetTests|ChatStreamingJournalCoalescingTests|ChatSelectableProseIncrementalEditTests|ChatJournalWritePathTests|ChatTranscriptGestureHarnessTests|OmiMarkdown|ChatStreamingRevealTests|ChatStreamingTailProjectionTests|ChatProseRenderCacheTests|KernelTurnRecordedProjectionTests|ChatFollowUpChipTests|ChatWorkingIndicatorTests|MarkdownNestedScrollTests|ChatBubble|ChatTimelineContinuity|ChatMessage|ChatTranscript' — all green (the ChatTimelineContinuityTests source tripwire was updated for the renamed messageContentView(_:rowText:); the ordering contract it checks is unchanged).
• python3 .github/scripts/check_chat_selection_boundary.py — passes.
• Real user-facing path exercised on the named bundle as described above: the answer streams, settles, and the transcript follows the live edge; the working mark animates while streaming and rests when settled.

Product invariants affected

• INV-AUTH-1
• INV-CHAT-1
• INV-CHAT-2

Behaviour under each is unchanged; their guard tests are in the suites above (the chat-scroll harness drives the streamed rows this change touches).

Failure-Class: none

Line-Count-Exception: desktop/macos/Desktop/Sources/MainWindow/Components/ChatBubble.swift | 2119 -> 2148 | the row text is derived once per body evaluation; the memo lives beside the sites it replaces
Line-Count-Exception: desktop/macos/Desktop/Sources/MainWindow/Components/OmiMarkdown.swift | 1672 -> 1716 | hand-rolled header/bullet preprocessing replaces per-line regex compiles on the streaming path

🤖 Generated with Claude Code

[cubic-dev-ai]

6 issues found across 20 files

Confidence score: 3/5

• ChatSelectableProse.swift still rescans the accumulated answer during every append, leaving the streaming flush path O(answer length) and risking degraded performance on long responses — retain an incremental diff/state approach.
• ChatSelectableProse.swift can reuse liveHeight when formatting changes without altering rendered characters, potentially caching the wrong height for the new attributed render — compare full attributed storage or defer the reuse.
• ChatBubble.swift derives visibleAnswerText for every mounted user bubble on each transcript pass even though rendering uses message.text, adding avoidable per-row work to the streaming path — remove or narrow that derivation.
• The streaming budget test and chat-first-cohesive.yaml provide weaker validation than their assertions and coverage claims suggest: coalesced flushes can fail the budget precondition, perFlushSnapshots is unused, and the happy path does not verify streaming behavior — align the assertions and coverage metadata with behavior actually exercised.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="desktop/macos/Desktop/Sources/MainWindow/Components/ChatBubble.swift">

<violation number="1" location="desktop/macos/Desktop/Sources/MainWindow/Components/ChatBubble.swift:178">
P2: Every mounted user bubble now performs the full `visibleAnswerText` derivation on each transcript body pass, despite using only `message.text` for rendering. Avoid this extra per-row work on the streaming hot path while preserving the existing trimmed copy payload.</violation>
</file>

<file name="desktop/macos/e2e/flows/chat-first-cohesive.yaml">

<violation number="1" location="desktop/macos/e2e/flows/chat-first-cohesive.yaml:51">
P3: The three new `covers:` entries claim coverage for streaming-pipeline behavior that this happy-path flow only mounts, never verifies. The flow sends one short query ("Acknowledge this attached conversation.") and asserts only are-you-idle plus message-content — it never exercises the per-flush work counters, the journal-write coalescing semantics (newest-first, one-in-flight, terminalization gating, cancelAll), or the sentence-spacing/pending-tail normalization output. Each of those behaviors is asserted hermetically in `ChatStreamingRenderBudgetTests` / `ChatStreamingJournalCoalescingTests` / unit tests, not here. Per the covers discipline (list a source only when the flow exercises *and* verifies its behavior), these entries overstate what the flow covers. Either drop the entries or keep only files whose distinguishing behavior the flow actually asserts; the comment for `ChatStreamingRenderProbe` already concedes it is "asserted hermetically ... not here," which contradicts listing it as covered.</violation>
</file>

<file name="desktop/macos/Desktop/Tests/ChatStreamingRenderBudgetTests.swift">

<violation number="1" location="desktop/macos/Desktop/Tests/ChatStreamingRenderBudgetTests.swift:65">
P3: `perFlushSnapshots` is appended to on every flush but never read after the loop; only `totals` and the idle snapshot are asserted. The whole array (and the `delta` calls that fill it) is dead work in a hot loop that also skews the very CPU-budget measurement this test reports. Drop the array and the per-flush `previous`/`delta` bookkeeping, or assert on it.</violation>

<violation number="2" location="desktop/macos/Desktop/Tests/ChatStreamingRenderBudgetTests.swift:123">
P2: The precondition `appKitProseBuild >= flushes` requires zero coalescing across ~100 flushes, but the same test's own comment on the storage counters says "two flushes landing in one run-loop turn are one edit, which is coalescing" and therefore allows up to 10% coalescing for `storageIncrementalEdit`. If a single pair of flushes coalesces into one AppKit prose build on a loaded CI runner, this hard `>= flushes` precondition fails the whole budget test. Give the prose-build precondition the same slack as the incremental-edit assertion (e.g. `>= flushes * 9 / 10`) or settle the inconsistency.</violation>
</file>

<file name="desktop/macos/Desktop/Sources/MainWindow/Components/ChatSelectableProse.swift">

<violation number="1" location="desktop/macos/Desktop/Sources/MainWindow/Components/ChatSelectableProse.swift:241">
P2: Each append still scans the accumulated answer before editing it. `commonPrefix` and the attribute walk traverse the unchanged prefix on every flush, so this hot path remains O(answer length); retain incremental diff state or a validated append cursor, with a fallback when earlier Markdown can be restyled.</violation>

<violation number="2" location="desktop/macos/Desktop/Sources/MainWindow/Components/ChatSelectableProse.swift:304">
P2: When formatting changes without changing rendered characters, `liveHeight` can reuse the old layout and cache its height for the new render entry. Compare the full attributed storage before using liveHeight, or defer measurement until `updateNSView` applies the new attributes.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The three new covers: entries claim coverage for streaming-pipeline behavior that this happy-path flow only mounts, never verifies. The flow sends one short query ("Acknowledge this attached conversation.") and asserts only are-you-idle plus message-content — it never exercises the per-flush work counters, the journal-write coalescing semantics (newest-first, one-in-flight, terminalization gating, cancelAll), or the sentence-spacing/pending-tail normalization output. Each of those behaviors is asserted hermetically in ChatStreamingRenderBudgetTests / ChatStreamingJournalCoalescingTests / unit tests, not here. Per the covers discipline (list a source only when the flow exercises and verifies its behavior), these entries overstate what the flow covers. Either drop the entries or keep only files whose distinguishing behavior the flow actually asserts; the comment for ChatStreamingRenderProbe already concedes it is "asserted hermetically ... not here," which contradicts listing it as covered.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At desktop/macos/e2e/flows/chat-first-cohesive.yaml, line 51:

<comment>The three new `covers:` entries claim coverage for streaming-pipeline behavior that this happy-path flow only mounts, never verifies. The flow sends one short query ("Acknowledge this attached conversation.") and asserts only are-you-idle plus message-content — it never exercises the per-flush work counters, the journal-write coalescing semantics (newest-first, one-in-flight, terminalization gating, cancelAll), or the sentence-spacing/pending-tail normalization output. Each of those behaviors is asserted hermetically in `ChatStreamingRenderBudgetTests` / `ChatStreamingJournalCoalescingTests` / unit tests, not here. Per the covers discipline (list a source only when the flow exercises *and* verifies its behavior), these entries overstate what the flow covers. Either drop the entries or keep only files whose distinguishing behavior the flow actually asserts; the comment for `ChatStreamingRenderProbe` already concedes it is "asserted hermetically ... not here," which contradicts listing it as covered.</comment>

<file context>
@@ -45,6 +45,12 @@ covers:
+  # Streaming answers on this surface exercise the incremental prose edit and
+  # the coalesced journal write; the work-per-flush counters are DEBUG-only
+  # and asserted hermetically in ChatStreamingRenderBudgetTests.
+  - desktop/macos/Desktop/Sources/MainWindow/Components/ChatStreamingRenderProbe.swift
+  - desktop/macos/Desktop/Sources/Chat/ChatTurnLifecycle.swift
+  - desktop/macos/Desktop/Sources/Providers/ChatProvider+AssistantTextNormalization.swift
</file context>

[cubic-dev-ai]

P3: perFlushSnapshots is appended to on every flush but never read after the loop; only totals and the idle snapshot are asserted. The whole array (and the delta calls that fill it) is dead work in a hot loop that also skews the very CPU-budget measurement this test reports. Drop the array and the per-flush previous/delta bookkeeping, or assert on it.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At desktop/macos/Desktop/Tests/ChatStreamingRenderBudgetTests.swift, line 65:

<comment>`perFlushSnapshots` is appended to on every flush but never read after the loop; only `totals` and the idle snapshot are asserted. The whole array (and the `delta` calls that fill it) is dead work in a hot loop that also skews the very CPU-budget measurement this test reports. Drop the array and the per-flush `previous`/`delta` bookkeeping, or assert on it.</comment>

<file context>
@@ -0,0 +1,258 @@
+      let cpuAfter = Self.mainThreadCPUNanoseconds()
+      flushCPUMilliseconds.append(Double(cpuAfter - cpuBefore) / 1_000_000)
+      let now = ChatStreamingRenderProbe.snapshot()
+      perFlushSnapshots.append(Self.delta(from: previous, to: now))
+      previous = now
+    }
</file context>