Repository navigation
Description
Tip
How to use GitHub
- Please use the 👍 reaction to show that you are affected by the same issue.
- Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
- Subscribe to receive notifications on status change and new comments.
Feature request
Which Nextcloud Version are you currently using: 35 (Talk 25)
Is your feature request related to a problem? Please describe.
Live captions in Talk show only the last few lines, and only from the moment you turn them on. Somebody who joins a call late, reloads the page or looks away for a minute cannot read what was said before. "Active call transcription - to catch up, if you're late" is the top voted item in #19532.
The provider contract has no way to offer this today: a caption provider sends transcript messages over the HPB and nothing else, so there is nothing a client could ask for the earlier lines.
We maintain a caption provider (Cassini Live) that already keeps such a history for the running call and shows it in a panel in Talk's right sidebar. It does so with its own route and a bookmarklet that patches Talk's page, because Talk has no place for it. We would like this to be a native part of Talk that any provider can support.
Describe the solution you'd like
An optional extension of the live transcription provider contract, and a transcript panel in Talk that appears only when the provider supports it. A provider that does not announce the feature is treated exactly as today: nothing changes for live_transcription or for existing installs.
This depends on #20296 (choosing which ExApp provides live transcription; work in progress in #20369).
Contract
-
Capability. The provider adds
live_transcription_historyto thefeatureslist it already returns fromGET /capabilities:{"live_transcription": {"features": ["live_transcription", "live_transcription_history"]}}Talk passes this on as a capability of its own, so clients know whether to show the panel. Everything below applies only to a provider that announces it.
-
seqon final captions. The provider numbers the entries of a call's history from 1. Everytranscriptmessage withfinal: truecarries theseqof its entry. Partial captions (final: false) carry none. The rest of the message is unchanged, and clients that do not knowseqignore it.{"type": "transcript", "speakerSessionId": "…", "message": "let's start with the roadmap", "langId": "en", "final": true, "seq": 42} -
Joins. A participant joining the call is also an entry in the history and takes a
seq, but it is not sent over the HPB. A client notices it as a gap: the next final arrives with aseqthat is not the next one, and the client reads the history to fill the gap. -
History route on the provider. Required for a provider that announces the feature.
GET /api/v1/call/history?roomToken=<token>&since=<seq> {"entries": [ {"seq": 41, "type": "final", "speakerSessionId": "…", "message": "…", "langId": "en", "t": 1794568794031}, {"seq": 42, "type": "join", "speakerSessionId": "…", "t": 1794568801200} ], "head": 57}entriesare those with aseqgreater thansince, oldest first.headis the latestseqthe provider holds, so an empty answer withheadequal tosincemeans "up to date".- The history covers the running call only. It starts with the first request to transcribe the call and is discarded when the provider leaves the call. With no captions running the answer is
{"entries": [], "head": 0}. - The provider may bound the history. A first entry beyond
since + 1means the entries between were dropped and will not arrive. - The speaker of an entry should be identified with Talk's actor fields (
actorType,actorId, display name), so that a line can be attributed after the speaker has left, and so that live and recorded transcripts (feat(recording): speaker-attributed transcript and summary for call recordings #19708) name speakers the same way.
-
Access through Talk. Clients do not call the provider. Talk offers
GET /ocs/v2.php/apps/spreed/api/v1/live-transcription/{token}/history?since=and passes the request on to the provider server to server, as it does for the other live transcription requests. Talk applies the checks it already applies when captions are turned on: the caller is a participant and past the lobby, and the conversation is not classified (400 classified, as in 🛡️ Overview: Classified conversations #18679). The provider trusts Talk and needs no access checks of its own. Guests are served like any other participant.
Proposed Talk UI
Without the capability nothing changes: the caption button and the caption box work as today.
With the capability:
- A "Live transcription" tab is available in the right sidebar during a call.
- Captions are still turned on per participant, with the existing button. The provider starts keeping the history with the first request in the call, whoever sends it. A participant who has not turned captions on sees the empty panel with a button to start.
- When captions are turned on, the panel reads the history once from
since=0. This is the catch-up for late joiners and after a reload. - After that, finals from the HPB are appended in
seqorder. A final whoseseqis not the next one is held back, and the history is read from the lastseqthe panel has. The same read is done after a reconnect, and at a low frequency to pick up joins and lines after which nobody spoke. - Lines are grouped by speaker, the participant's own lines included. Joins are shown as markers ("Alice joined"). A partial caption is shown as a line in progress and replaced by its final.
Not all of this has to live in the browser. Part of it, such as merging the history with the live messages, could be done by Talk's backend.
Privacy
The history is what the participants of the call could already read as live captions. In our implementation it is kept in memory only, bounded, and discarded when the call ends, and it is served to participants of the conversation only. We would suggest making this the expectation for every provider that announces the feature. With Talk in front of the history, classified conversations are refused by the rule Talk already has.
Open questions
- Provider restart. The history is lost, which is acceptable, but the numbering starts again at 1 and a client still holds a higher
seq. Either the client treats aseqorheadlower than its own as "start over and read from 0", or the messages carry an identifier of the numbering that changes on restart. - Translation. Translated captions are sent as finals too, to the sessions that chose a target language, so the same line exists in several languages. One option: the provider keeps a history per language. The first request for a language that is not there yet translates the existing history and stores it; later requests get the stored one, so every line is translated at most once per language. The history request would then name the language.
- How often clients read the history without a gap to fill. Cassini Live polls every 3 seconds today, directly at the provider. Through Talk a lower frequency seems right.
Describe alternatives you've considered
- What we do today: the panel is injected into Talk's page by a bookmarklet and reads the provider's history through the AppAPI proxy, and the provider checks conversation membership itself by asking Talk as the user. This breaks with Talk releases, cannot serve guests, and puts access checks into the provider that belong to Talk.
- Keeping the history in the browser only, by lengthening the caption box. This needs no contract change, but gives nothing to late joiners or after a reload, and every participant ends up with a different record.
- Sending the earlier lines again over the HPB when captions are turned on. This has no way to recover a lost message later, and puts the catch-up of every late joiner on the signaling connection.
- Storing the transcript in the conversation, like chat messages or the transcript of a recording. That is a different feature with different privacy questions; this request is only about the running call.
Additional context
- Reference implementation: Cassini Live.
seq, the history and joins are implemented in the provider and in its panel. The panel's merge of history and live messages is a small, tested module we can point to or port. - The contract additions are optional and additive. We checked
live_transcription2.1.3 and Cassini Live side by side in one Nextcloud: both send the sametranscriptmessage, and Talk's caption box works unchanged with messages that carryseq. - We are happy to send pull requests for the Talk side, backend and frontend, once the shape is agreed.
Activity
ikusteu commentedon Nov 19, 2026
Note to maintainer: Sorry if this is a lengthy one, but we really thought this through and I wanted to propose a solution we already have, and I wanted to have the the whole structure in one place (contract + the UI).
Please let me know if I should split this into multiple related feature requests (contract updates and UI proposal)
ikusteu commentedon Nov 19, 2026
Open question: Provider restart
(I'm separating open questions to draw more attention to things which I didn't know how to answer and would open a discussion.)
Scenario:
How it manifests:
seq(e.g. 51)seq1Question: What should the UI do?
Option 1:
seqreturned from message feed (let's call itprovider_head) < latestseqof local history (let's call itlocal_head, query/historyendpoint, if the result is the same (provider_head<local_head) - recognise this as history was lost/historyendpoint returned)Option 2:
provider_head<local_head, andprovider_session_iddifferent from local history - provider restarted, history lostOption 3:
ikusteu commentedon Nov 19, 2026
Open question: Translations
(I'm separating open questions to draw more attention to things which I didn't know how to answer and would open a discussion.)
Live translations also send
final: truemessages. How are those handled? How often does translation happen? What does it look like in the UI?Some of those questions may be provider-specific, but how they're handled in the UI might be where the contract is defined.
My suggestion (from user's perspective -- UI):
/historyendpoint returns it)final: falseis used to render the current attempt -- original language message, same as native language behaviour)On the provider side (this may be provider specific, and we haven't tackled that in Cassini Live yet):
ikusteu commentedon Nov 19, 2026
Open question: How often does the client poll for history
(I'm separating open questions to draw more attention to things which I didn't know how to answer and would open a discussion.)
This belongs in the decision related to this feature as Talk is the client polling for history (regardless of the provider).
Our current implementation (Cassini Live) polls every 3sec (arbitrary interval). However, this should be reconsidered
Option 1:
Option 2:
message[t].seq = 51,message[t+1].seq = 55-- gap detected, poll/history/historyretrieved)My take: keep both, and consider the interval at which the pulse checks for history.
nickvergessen commentedon Nov 20, 2026
I think this should not be in the sidebar. It's an edge case and mostly temporary from my POV.