Skip to content

Commit a43e421

Browse files
veltson-vapiveltsonbastienclaudeshubham-vapiai
authored
feat: expose the assistant audio element for playback volume control (#169)
* feat: expose the assistant audio element for playback volume control ## Problem The SDK creates an `<audio>` element per remote track in `buildAudioPlayer` and discards the reference, so there is no supported way to reach it. There is no way to set playback volume at all: `volume-level` only reports measured loudness, and `setMuted` affects the user's microphone. Consumers work around this by querying the DOM for `audio[data-participant-id]`, which relies on an internal detail and can match an orphaned element from a previous call (#161). Requested in #37. ## What this adds - `setVolume(volume)` for assistant playback volume, 0 to 1. The value is remembered, so it can be set before the track arrives and it survives across calls and mid-call track replacement. Out-of-range values are clamped and non-finite values ignored, since the DOM rejects both rather than saturating. - `getAudioPlayer()` returning the element, or null before its track arrives. - An `audio` event carrying the element the moment it exists, mirroring how `video` already emits one line above in the same handler. - An `error` of type `audio-start-failed` when the browser blocks playback, which is usually autoplay policy and is recoverable by prompting for a gesture. Previously this surfaced only as an unhandled rejection. Pairing a getter with an event matches `getLocalAudioLevel` alongside `local-volume-level`. ## Lifecycle correctness `buildAudioPlayer` awaits `play()`, which pends for the whole of media startup, so several players can be in flight, they can finish out of order, and the call or the participant can go away underneath them. Each build now takes a sequence number and records its participant while pending, so that: - a player built for a call that has since ended or been replaced is discarded - a teardown invalidates only the departing participant's builds, so an unrelated participant leaving cannot cost us the assistant's audio - a build that started before the one currently attached cannot overwrite it - a superseded element is removed when its replacement is adopted, which matters because renegotiation reuses the participant id and both elements would otherwise match the selector teardown uses - the reference is dropped in `stop()`, `cleanup()` and on `participant-left`, so `setVolume` can never write to a torn-down player Consumer emits are routed through `emitToConsumer`, because a listener that threw previously propagated out of the `track-started` handler and skipped `sendAppMessage('playable')`, stalling the call. This applied to the existing `video` emit too. `handleTrackStarted` is extracted so the initial and reconnect paths share it rather than being two byte-identical copies. ## Not addressed The orphaned elements themselves (#161). This ensures the new API never hands back an orphan, but does not stop them accumulating. Relatedly, `playable` is still not sent when playback genuinely fails, which is unchanged behavior and a question about what that signal means server-side. ## Testing 46 tests, `tsc`, `npm run build` and `npm run test:example` all pass. No dependency, lockfile, tsconfig, packaging or CI changes. The two functions that touch the DOM are reached through thin private wrappers so the lifecycle logic around them is covered in the existing node test environment without adding a DOM dependency. Every fix above was mutation tested: reverting it fails the suite. Still uncovered, as on main: the DOM calls themselves. Also verified by hand on two live calls, confirming a volume set before the call is applied to the real element and persists into the next call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * style: trim comments to the ones that carry non-obvious rationale The change raised comment density in vapi.ts from 8% to 12% and used multi-line paragraph blocks where the file's own style is short single lines. Removed comments that restate the code (teardown nulling fields, the guard condition in handleTrackStarted, method docs that repeat the method name) and comments duplicated between production and tests. Compressed the rest, keeping only what a reader cannot get from the code: why builds are tracked individually, why the emit is guarded and not routed to the error event, why volume is set before play(), why the DOMException fixture is not a real DOMException. vapi.ts is now 10%, and the test file 6%, against main's 8%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: veltsonbastien <bastienveltson@outlook.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: shubham-vapiai <shubham+github@vapi.ai>
1 parent 7f16793 commit a43e421

4 files changed

Lines changed: 779 additions & 39 deletions

File tree

‎README.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,47 @@ The `say(message: string, endCallAfterSpoken?: boolean)` can be used to invoke s
105105
vapi.say("Our time's up, goodbye!", true)
106106
```
107107

108+
The `setVolume(volume: number)` controls how loudly the assistant plays, from
109+
`0` to `1`. This is playback only: it does not change what the assistant hears,
110+
and it does not affect the levels reported by the `volume-level` event.
111+
112+
```javascript
113+
vapi.setVolume(0.5);
114+
```
115+
116+
The setting is remembered, so you can call it before a call has started and it
117+
will be applied as soon as the assistant's audio arrives. It also persists
118+
across calls, so a volume your user picked once is not reset by hanging up.
119+
Values outside `0` to `1` are clamped, and non-finite values are ignored.
120+
121+
The `getAudioPlayer()` returns the `<audio>` element the SDK uses to play the
122+
assistant, or `null` before its track has arrived. Use it for anything the
123+
setter doesn't cover, such as driving a waveform visualizer.
124+
125+
```javascript
126+
const player = vapi.getAudioPlayer();
127+
```
128+
129+
Pair it with the `audio` event when you need the element the moment it exists:
130+
131+
```javascript
132+
vapi.on('audio', (player) => {
133+
// e.g. new AudioContext().createMediaElementSource(player)
134+
});
135+
```
136+
137+
If the browser blocks playback, usually its autoplay policy when the call was
138+
not started from a user gesture, no player is attached and an `error` is
139+
emitted instead. Listen for it to prompt the user:
140+
141+
```javascript
142+
vapi.on('error', (e) => {
143+
if (e.type === 'audio-start-failed') {
144+
// show a "tap to enable audio" affordance
145+
}
146+
});
147+
```
148+
108149
## Events
109150

110151
You can listen to the following events:
@@ -130,6 +171,11 @@ vapi.on('volume-level', (volume) => {
130171
console.log(`Assistant volume level: ${volume}`);
131172
});
132173

174+
// The <audio> element playing the assistant, as soon as it exists
175+
vapi.on('audio', (player) => {
176+
console.log(`Assistant audio attached at volume ${player.volume}`);
177+
});
178+
133179
// Function calls and transcripts will be sent via messages
134180
vapi.on('message', (message) => {
135181
console.log(message);

0 commit comments

Comments
 (0)