Feature request: let users add or swap local STT and TTS in conversation mode

[felixingenium]

Before filing

• I searched open and closed issues for duplicates.
• I am using a recent release and this doesn't already exist.
• This is one request, not several bundled together.

Closest existing issue

none found (searched voice, parakeet, pocket, stt, tts, conversation, nemotron, speech). Closest in spirit is #110 (closed ACP agent catalog). Different layer.

Is this new, or an improvement?

New capability — Berd can't do this at all today

The problem, in your terms

I use Berd conversation mode as the spoken loop into a live agent session (Grok 4.6 via Goose). That loop is the product I want: talk, the agent hears it in the current chat, the agent speaks back, I can barge in.

What I can adjust today:

• the agent and chat model (the brain)
• which Pocket speaker to use (Azelma, Mary, and the rest)
• playback speed
• Remove model for Pocket or Parakeet, then download that same pair again

What I cannot adjust:

• the speech recognizer
• the speech synthesizer family
• any second local STT or TTS pack

I wanted to try a streaming speech model (NVIDIA Nemotron speech streaming is the one I had in hand) inside the same conversation mode. Settings only offered Remove. There is no Add, no catalog, no path picker.

I then read the 0.6.2 source. Conversation STT is a pinned Offline CTC Parakeet 110M worker (native_voice.rs loads model.int8.onnx through sherpa-onnx nemo_ctc). Install and remove only know two kinds: Pocket and Parakeet (pocket_voice.rs, VoiceModelKind). That matches the UI. The missing piece is not a hidden button. The catalog is closed.

What you do today

I keep using the shipped Parakeet plus Pocket pair in Berd, because that is the only path that injects transcripts into the live Goose session and speaks deltas with barge-in and the speaker echo guard.

Separately I built a local sidecar that reuses the same Parakeet weights, then calls Grok headless, then macOS say. It works as a walkie-talkie. It is not conversation mode. It does not steer the open Berd session, it does not stream assistant audio from Pocket, and each turn pays a full new-process cost (about 20 seconds in my smoke).

Forking Berd to hardcode a third model would work once and then fight every Berd release. I am not treating a private fork as the daily app.

What you'd like to see

A supported way to register another local STT and/or TTS backend for conversation mode, without editing Berd source.

Minimum that would unblock me:

• Settings next to Remove: Add, or pick from a small list, or point at a local model directory plus a recognizer kind
• Conversation mode uses that backend through the same events it already has (user transcript, speaking or idle, error, clean shutdown)
• Install stays checksum-pinned, same as today's Parakeet and Pocket packs
• The shipped Parakeet plus Pocket pair remains the default

Nemotron speech streaming is one backend I would try first. I am not asking you to ship or maintain that model.

Why this belongs in Berd itself

Conversation mode is a Berd runtime, not a Goose skill. A skill, agent, extension, or automation cannot replace the Tauri STT worker, the Pocket playback path, the microphone claim, or the Settings voice install or remove surface.

berdctl can create and send sessions. It cannot swap the recognizer that feeds those sessions.

This is the same class of closed catalog as #110, on the ears and mouth instead of the agent process.

Non-goals

• No Nemotron-specific integration, and no request that Berd download or vendor NVIDIA speech models
• No cloud STT or TTS, no API keys, no Grok Voice SKU
• No change to the agent or chat-model picker (the brain already switches)
• No replacement of Parakeet or Pocket as the default
• No plugin marketplace, no unsigned model URLs, no request to accept an outside PR
• No change to telemetry defaults

Alternatives you considered

• Use the Remove button, then hope Add appears: Remove only retires the pinned pack. Reinstall puts the same two kinds back.
• Drop a different ONNX into ~/Library/Application Support/xyz.block.berd/pocket-tts/native-voice-v2/stt/: checksums reject it. The worker is Offline CTC, not a streaming RNNT graph.
• Skill or automation: runs after an agent is already selected. It cannot own the mic pipeline.
• Local sidecar (what I run now): useful for eval, missing conversation mode.
• Fork Berd and hardcode a third VoiceModelKind: works until the next release. CONTRIBUTING says not to send that patch.

A small generic local-backend registration point is cleaner than one-off model integrations.

Mockups, prior art, or other context

Verified on Berd 0.6.2, macOS, Settings → About. Native voice pack is installed:

• STT: sherpa-onnx-nemo-parakeet_tdt_ctc_110m-en-36000-int8 as native-voice-v2/stt/model.int8.onnx
• TTS: Pocket ONNX plus the Kyutai speaker wavs; selected voice Azelma; speed 1.5

Source read on block/berd main:

• src-tauri/src/commands/native_voice.rs
• src-tauri/src/commands/pocket_voice.rs
• src/features/voice-conversation/

Conversation mode today:

mic
  -> pinned Parakeet Offline CTC
    -> transcript events into the live Goose session
      -> agent tokens
        -> pinned Pocket TTS
          -> speakers, with capture suppressed while it talks

The ask is a supported third box on the STT and TTS steps, not a new conversation product.