You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After today's 300 ms silence trigger, run a small local model on the
audio. Wake the agent only if the model says the speaker finished. If it
says they are still going, keep listening.
Do not change the silence number. Do not touch push-to-talk. No cloud.
No new wire format.
Status: proposal. Flagged prototype and a measurement on a laptop
before anything becomes default. No code.
Today vs proposed
Left is what Buzz does now. Right is this RFC. Grey is unchanged.
Green is the only new step. Red is the premature wake we stop.
flowchart LR
subgraph T["Today"]
direction TB
t1[You pause mid-thought] --> t2[300 ms silence]
t2 --> t3[Flush and wake the agent]
end
subgraph P["Proposed"]
direction TB
p1[You pause mid-thought] --> p2[300 ms silence]
p2 --> p3{Local model}
p3 -->|HOLD still going| p4[Keep listening]
p3 -->|SHIFT finished| p5[Flush and wake]
end
classDef bad fill:#fdecea,stroke:#c0392b,color:#000
classDef good fill:#e8f8f5,stroke:#1e8449,color:#000
classDef same fill:#f4f6f7,stroke:#7f8c8d,color:#000
class t3 bad
class p3,p4,p5 good
class t1,t2,p1,p2 same
Loading
Why a stopwatch cannot be the turn boundary
A pause is not a finished sentence. The two cases can have identical
words and still mean opposite things.
LiveKit's turn-detection writeup uses the pizza example: the transcript
at the pause is the same for "I would like to order one large pizza…"
(about to continue) and "I would like to order one large pizza… and a
garlic bread" (already continued). No amount of reading the words
distinguishes them — the distinction is in how they are delivered
(intonation, pitch, rhythm).
Source: LiveKit, "Solving end-of-turn detection".
The academic name for that fork is SHIFT vs HOLD. Ekstedt & Skantze
(Voice Activity Projection, Interspeech 2022) state it directly: the
system "should be able to tell whether a mutual silence should be
identified as a SHIFT or a HOLD." A thinking pause is HOLD. A real turn
boundary is SHIFT. Those two are not distinguishable from silence
duration alone.
Paper: arXiv:2205.09812
(HTML).
That is the whole argument. Tuning the timer is asking a stopwatch a
question it cannot answer.
We already tried tuning, in both directions
Buzz's STT worker at 93114c9c6 records two failed attempts
(stt.rs#L160-L171):
/// How many 16 kHz samples of silence before we flush to STT./// 300 ms × 16 000 Hz / 256 samples-per-frame ≈ 19 frames./// Previous value (28 frames / 450 ms) felt sluggish in conversation.////// This window is a turn-taking quality knob, not a latency lever: an earlier/// env override (`BUZZ_STT_FLUSH_MS`) let it be lowered to 150 ms, which split/// natural mid-sentence pauses into separate messages and confused the/// listening agents. Reverted — the window is fixed at the production value.constSILENCE_FLUSH_FRAMES:usize = 19;constVAD_FRAME_SAMPLES:usize = 256;
Tried
Result
450 ms (28 frames)
Felt sluggish in conversation
150 ms (BUZZ_STT_FLUSH_MS)
Split mid-sentence pauses into separate messages; confused listening agents; reverted
300 ms (19 × 256 samples at 16 kHz ≈ 304 ms)
Value at 93114c9c6
The number is not the bug. The signal is the bug. This proposal does not retune SILENCE_FLUSH_FRAMES.
What was surveyed
This is not a two-product glance. The question we asked: do production
voice agents still use silence duration as the turn boundary, or do they
add a second signal — and is any of that shippable in a local Rust
desktop with no cloud STT?
In scope: named, currently shipping turn-end mechanisms with a public
doc or repo. Out of this table: unpublished lab models (except
Ekstedt & Skantze above, which names SHIFT/HOLD; it is not a product).
System
Signal beyond silence
Local / open?
Verdict for Buzz
Source
LiveKit Agents TurnDetector v1
Audio-native EOU probability (semantic + prosody)
v1-mini open-weight; full v1 is LiveKit Cloud
Architecture match; SDK is Python/Node, weights under LiveKit Model License — not the copy target
What the table is for: show the land was walked. Most surveyed systems
add a model-based signal beyond silence; most implementations are hosted
or proprietary. Pipecat Smart Turn v3 is the only open BSD-2 ONNX option
here designed for local, VAD-gated inference.
What we would copy
Pipecat Smart Turn v3. Evidence is the project's README at 4786657:
Claim
Evidence
Audio classifier on raw PCM after VAD, not a transcript
Why this one rather than LiveKit's open v1-mini:
Buzz already runs speech-to-text as Parakeet through sherpa-onnx /
ONNX Runtime
(stt.rs#L185-L194, stt.rs#L240-L250).
Smart Turn v3 is the same model format and inference class. It is not a drop-in Rust crate. Pipecat's local wrapper is Python
(README#L70-L80; LocalSmartTurnAnalyzerV3 API).
The prototype must verify a Rust/ONNX integration path; none was found as
a ready standalone reference.
Cost (Pipecat's hardware, not ours): ~10 ms on some CPUs and under
100 ms on most cloud instances
(README#L18);
~65 ms on a Pipecat Cloud standard 1x instance with LocalSmartTurnAnalyzerV3
(README#L80).
Do not quote a Buzz laptop number until it is measured.
Sequence we would add (nothing else)
Earshot VAD already waits for 19 silent frames (~300 ms). Unchanged.
On that silence, run Smart Turn v3 on the buffered PCM, truncated to
the last ~8 s if longer
(README#L88-L94). New.
HOLD → keep accumulating; do not wake agents.
SHIFT → flush to STT and wake agents, as today.
If speech resumes before SHIFT, re-run on the whole retained recording
(again within 8 s), not just the new segment
(README#L96-L98).
Out of scope — and why
Left out
Why
Retuning SILENCE_FLUSH_FRAMES
Already tried 450 ms (sluggish) and 150 ms (split sentences). The comment at stt.rs#L160-L171 is the experiment log. A stopwatch cannot tell SHIFT from HOLD.
Push-to-talk mention behaviour
Hands-free is the bug. At 93114c9c6, a held PTT shortcut never lets silence flush (stt.rs#L544-L552); the falling edge of transmit flushes the buffer (stt.rs#L310-L325). Mixing the two problems hides whether the model helped.
New Nostr / ACP wire types
The model sits after local VAD and before the existing STT flush. Nothing on the wire changes.
Those are steer-path races (a control signal aimed at no prompt). This RFC is when to flush audio. Different layer.
What we do not know yet
Unknown
Why it matters
Latency on a typical laptop
Pipecat's 10–65 ms is not a Buzz measurement
Non-English speech
README lists 23 languages (README#L15-L16); we have not tested huddle languages
30 s Buzz buffer vs Smart Turn's 8 s window
Buzz keeps up to 30 s of PCM (stt.rs#L40-L42, stt.rs#L458-L464). Smart Turn truncates to the last ~8 s (README#L88-L94). A long huddle turn can still sit in our buffer while the model only sees the tail. Unmeasured whether that truncation hides HOLD vs SHIFT.
Rust integration
Python ONNX wrapper exists; a Buzz-shaped Rust path does not, yet
Decision requested
Approve the direction: a flagged prototype plus a laptop
measurement, before any change to default hands-free behaviour.
Say go / wait / change the ask.
The ask
After today's 300 ms silence trigger, run a small local model on the
audio. Wake the agent only if the model says the speaker finished. If it
says they are still going, keep listening.
Do not change the silence number. Do not touch push-to-talk. No cloud.
No new wire format.
Status: proposal. Flagged prototype and a measurement on a laptop
before anything becomes default. No code.
Today vs proposed
Left is what Buzz does now. Right is this RFC. Grey is unchanged.
Green is the only new step. Red is the premature wake we stop.
flowchart LR subgraph T["Today"] direction TB t1[You pause mid-thought] --> t2[300 ms silence] t2 --> t3[Flush and wake the agent] end subgraph P["Proposed"] direction TB p1[You pause mid-thought] --> p2[300 ms silence] p2 --> p3{Local model} p3 -->|HOLD still going| p4[Keep listening] p3 -->|SHIFT finished| p5[Flush and wake] end classDef bad fill:#fdecea,stroke:#c0392b,color:#000 classDef good fill:#e8f8f5,stroke:#1e8449,color:#000 classDef same fill:#f4f6f7,stroke:#7f8c8d,color:#000 class t3 bad class p3,p4,p5 good class t1,t2,p1,p2 sameWhy a stopwatch cannot be the turn boundary
A pause is not a finished sentence. The two cases can have identical
words and still mean opposite things.
LiveKit's turn-detection writeup uses the pizza example: the transcript
at the pause is the same for "I would like to order one large pizza…"
(about to continue) and "I would like to order one large pizza… and a
garlic bread" (already continued). No amount of reading the words
distinguishes them — the distinction is in how they are delivered
(intonation, pitch, rhythm).
Source: LiveKit, "Solving end-of-turn detection".
The academic name for that fork is SHIFT vs HOLD. Ekstedt & Skantze
(Voice Activity Projection, Interspeech 2022) state it directly: the
system "should be able to tell whether a mutual silence should be
identified as a SHIFT or a HOLD." A thinking pause is HOLD. A real turn
boundary is SHIFT. Those two are not distinguishable from silence
duration alone.
Paper: arXiv:2205.09812
(HTML).
That is the whole argument. Tuning the timer is asking a stopwatch a
question it cannot answer.
We already tried tuning, in both directions
Buzz's STT worker at
93114c9c6records two failed attempts(stt.rs#L160-L171):
BUZZ_STT_FLUSH_MS)93114c9c6The number is not the bug. The signal is the bug. This proposal does
not retune
SILENCE_FLUSH_FRAMES.What was surveyed
This is not a two-product glance. The question we asked: do production
voice agents still use silence duration as the turn boundary, or do they
add a second signal — and is any of that shippable in a local Rust
desktop with no cloud STT?
In scope: named, currently shipping turn-end mechanisms with a public
doc or repo. Out of this table: unpublished lab models (except
Ekstedt & Skantze above, which names SHIFT/HOLD; it is not a product).
v1-miniopen-weight; full v1 is LiveKit CloudEagerEndOfTurn/TurnResumedUtteranceEnd)turn.resume) are a state-machine blueprint onlysemantic_vadserver_vadsilenceDurationMs; interruption cancellation;interruptedeventWhat the table is for: show the land was walked. Most surveyed systems
add a model-based signal beyond silence; most implementations are hosted
or proprietary. Pipecat Smart Turn v3 is the only open BSD-2 ONNX option
here designed for local, VAD-gated inference.
What we would copy
Pipecat Smart Turn v3. Evidence is the project's README at
4786657:LocalSmartTurnAnalyzerV3Why this one rather than LiveKit's open
v1-mini:Buzz already runs speech-to-text as Parakeet through sherpa-onnx /
ONNX Runtime
(stt.rs#L185-L194,
stt.rs#L240-L250).
Smart Turn v3 is the same model format and inference class. It is
not a drop-in Rust crate. Pipecat's local wrapper is Python
(README#L70-L80;
LocalSmartTurnAnalyzerV3API).The prototype must verify a Rust/ONNX integration path; none was found as
a ready standalone reference.
Cost (Pipecat's hardware, not ours): ~10 ms on some CPUs and under
100 ms on most cloud instances
(README#L18);
~65 ms on a Pipecat Cloud standard 1x instance with
LocalSmartTurnAnalyzerV3(README#L80).
Do not quote a Buzz laptop number until it is measured.
Sequence we would add (nothing else)
the last ~8 s if longer
(README#L88-L94).
New.
(again within 8 s), not just the new segment
(README#L96-L98).
Out of scope — and why
SILENCE_FLUSH_FRAMES93114c9c6, a held PTT shortcut never lets silence flush (stt.rs#L544-L552); the falling edge of transmit flushes the buffer (stt.rs#L310-L325). Mixing the two problems hides whether the model helped.What we do not know yet
Decision requested
Approve the direction: a flagged prototype plus a laptop
measurement, before any change to default hands-free behaviour.
Say go / wait / change the ask.