Skip to content

Support history for live transcription: extend the live transcription contract and the UI support #20400

Open

Description

@ikusteu

Tip

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Feature request

Which Nextcloud Version are you currently using: 35 (Talk 25)

Is your feature request related to a problem? Please describe.
Live captions in Talk show only the last few lines, and only from the moment you turn them on. Somebody who joins a call late, reloads the page or looks away for a minute cannot read what was said before. "Active call transcription - to catch up, if you're late" is the top voted item in #19532.

The provider contract has no way to offer this today: a caption provider sends transcript messages over the HPB and nothing else, so there is nothing a client could ask for the earlier lines.

We maintain a caption provider (Cassini Live) that already keeps such a history for the running call and shows it in a panel in Talk's right sidebar. It does so with its own route and a bookmarklet that patches Talk's page, because Talk has no place for it. We would like this to be a native part of Talk that any provider can support.

Describe the solution you'd like
An optional extension of the live transcription provider contract, and a transcript panel in Talk that appears only when the provider supports it. A provider that does not announce the feature is treated exactly as today: nothing changes for live_transcription or for existing installs.

This depends on #20296 (choosing which ExApp provides live transcription; work in progress in #20369).

Contract

  1. Capability. The provider adds live_transcription_history to the features list it already returns from GET /capabilities:

    {"live_transcription": {"features": ["live_transcription", "live_transcription_history"]}}

    Talk passes this on as a capability of its own, so clients know whether to show the panel. Everything below applies only to a provider that announces it.

  2. seq on final captions. The provider numbers the entries of a call's history from 1. Every transcript message with final: true carries the seq of its entry. Partial captions (final: false) carry none. The rest of the message is unchanged, and clients that do not know seq ignore it.

    {"type": "transcript", "speakerSessionId": "…", "message": "let's start with the roadmap", "langId": "en", "final": true, "seq": 42}
  3. Joins. A participant joining the call is also an entry in the history and takes a seq, but it is not sent over the HPB. A client notices it as a gap: the next final arrives with a seq that is not the next one, and the client reads the history to fill the gap.

  4. History route on the provider. Required for a provider that announces the feature.

    GET /api/v1/call/history?roomToken=<token>&since=<seq>
    
    {"entries": [
       {"seq": 41, "type": "final", "speakerSessionId": "…", "message": "…", "langId": "en", "t": 1794568794031},
       {"seq": 42, "type": "join", "speakerSessionId": "…", "t": 1794568801200}
     ],
     "head": 57}
    
    • entries are those with a seq greater than since, oldest first.
    • head is the latest seq the provider holds, so an empty answer with head equal to since means "up to date".
    • The history covers the running call only. It starts with the first request to transcribe the call and is discarded when the provider leaves the call. With no captions running the answer is {"entries": [], "head": 0}.
    • The provider may bound the history. A first entry beyond since + 1 means the entries between were dropped and will not arrive.
    • The speaker of an entry should be identified with Talk's actor fields (actorType, actorId, display name), so that a line can be attributed after the speaker has left, and so that live and recorded transcripts (feat(recording): speaker-attributed transcript and summary for call recordings #19708) name speakers the same way.
  5. Access through Talk. Clients do not call the provider. Talk offers GET /ocs/v2.php/apps/spreed/api/v1/live-transcription/{token}/history?since= and passes the request on to the provider server to server, as it does for the other live transcription requests. Talk applies the checks it already applies when captions are turned on: the caller is a participant and past the lobby, and the conversation is not classified (400 classified, as in 🛡️ Overview: Classified conversations #18679). The provider trusts Talk and needs no access checks of its own. Guests are served like any other participant.

Proposed Talk UI

panel-in-call

Without the capability nothing changes: the caption button and the caption box work as today.

With the capability:

  • A "Live transcription" tab is available in the right sidebar during a call.
  • Captions are still turned on per participant, with the existing button. The provider starts keeping the history with the first request in the call, whoever sends it. A participant who has not turned captions on sees the empty panel with a button to start.
  • When captions are turned on, the panel reads the history once from since=0. This is the catch-up for late joiners and after a reload.
  • After that, finals from the HPB are appended in seq order. A final whose seq is not the next one is held back, and the history is read from the last seq the panel has. The same read is done after a reconnect, and at a low frequency to pick up joins and lines after which nobody spoke.
  • Lines are grouped by speaker, the participant's own lines included. Joins are shown as markers ("Alice joined"). A partial caption is shown as a line in progress and replaced by its final.

Not all of this has to live in the browser. Part of it, such as merging the history with the live messages, could be done by Talk's backend.

Privacy

The history is what the participants of the call could already read as live captions. In our implementation it is kept in memory only, bounded, and discarded when the call ends, and it is served to participants of the conversation only. We would suggest making this the expectation for every provider that announces the feature. With Talk in front of the history, classified conversations are refused by the rule Talk already has.

Open questions

  • Provider restart. The history is lost, which is acceptable, but the numbering starts again at 1 and a client still holds a higher seq. Either the client treats a seq or head lower than its own as "start over and read from 0", or the messages carry an identifier of the numbering that changes on restart.
  • Translation. Translated captions are sent as finals too, to the sessions that chose a target language, so the same line exists in several languages. One option: the provider keeps a history per language. The first request for a language that is not there yet translates the existing history and stores it; later requests get the stored one, so every line is translated at most once per language. The history request would then name the language.
  • How often clients read the history without a gap to fill. Cassini Live polls every 3 seconds today, directly at the provider. Through Talk a lower frequency seems right.

Describe alternatives you've considered

  • What we do today: the panel is injected into Talk's page by a bookmarklet and reads the provider's history through the AppAPI proxy, and the provider checks conversation membership itself by asking Talk as the user. This breaks with Talk releases, cannot serve guests, and puts access checks into the provider that belong to Talk.
  • Keeping the history in the browser only, by lengthening the caption box. This needs no contract change, but gives nothing to late joiners or after a reload, and every participant ends up with a different record.
  • Sending the earlier lines again over the HPB when captions are turned on. This has no way to recover a lost message later, and puts the catch-up of every late joiner on the signaling connection.
  • Storing the transcript in the conversation, like chat messages or the transcript of a recording. That is a different feature with different privacy questions; this request is only about the running call.

Additional context

  • Reference implementation: Cassini Live. seq, the history and joins are implemented in the provider and in its panel. The panel's merge of history and live messages is a small, tested module we can point to or port.
  • The contract additions are optional and additive. We checked live_transcription 2.1.3 and Cassini Live side by side in one Nextcloud: both send the same transcript message, and Talk's caption box works unchanged with messages that carry seq.
  • We are happy to send pull requests for the Talk side, backend and frontend, once the shape is agreed.

Activity

  1. ikusteu commented on Nov 19, 2026

    @ikusteu
    Author

    Note to maintainer: Sorry if this is a lengthy one, but we really thought this through and I wanted to propose a solution we already have, and I wanted to have the the whole structure in one place (contract + the UI).
    Please let me know if I should split this into multiple related feature requests (contract updates and UI proposal)

  2. ikusteu commented on Nov 19, 2026

    @ikusteu
    Author

    Open question: Provider restart

    (I'm separating open questions to draw more attention to things which I didn't know how to answer and would open a discussion.)

    Scenario:

    • A (live transcription) provider restarts (in the background, for whatever reason)
    • the UI already contains some history for the conversation (retrieved before the restart)
    • since the provider restarted, and history is buffered in memory (our implementation doesn't necessarily have to be the case), the history might be lost

    How it manifests:

    • the UI has some latest seq (e.g. 51)
    • when receiving new feed (restarted provider), the message comes with seq 1

    Question: What should the UI do?

    Option 1:

    • when seq returned from message feed (let's call it provider_head) < latest seq of local history (let's call it local_head, query /history endpoint, if the result is the same (provider_head < local_head) - recognise this as history was lost
    • clear the rendered history (from the UI), and apply currently existing history (what /history endpoint returned)
    • tradeoff: users looking at previous conversation history might still find it useful (even if fleeting -- gone by next refresh)

    Option 2:

    • tag history session (provider side)
    • if provider_head < local_head, and provider_session_id different from local history - provider restarted, history lost
    • keep local history (still useful), and perhaps flag that in the UI somehow
    • tradeoff: consistency - older history might get lost on refresh

    Option 3:

    • enforce durable record of history - kept between restarts, and held until Talk call explicitly over
    • tradeoffs:
      • this would have to be part of the contract
      • perhaps too strict for an edge case
      • moves the failure handling to how long do we keep it (e.g. what happens if call over, but the signal hadn't made its way back to the provider)
  3. ikusteu commented on Nov 19, 2026

    @ikusteu
    Author

    Open question: Translations

    (I'm separating open questions to draw more attention to things which I didn't know how to answer and would open a discussion.)

    Live translations also send final: true messages. How are those handled? How often does translation happen? What does it look like in the UI?

    Some of those questions may be provider-specific, but how they're handled in the UI might be where the contract is defined.

    My suggestion (from user's perspective -- UI):

    • confirmed history shows only the translated history (/history endpoint returns it)
    • pending live caption (final: false is used to render the current attempt -- original language message, same as native language behaviour)

    On the provider side (this may be provider specific, and we haven't tackled that in Cassini Live yet):

    • keep history in native language, and feed to everyone listening (for native history)
    • once the first person starts listening for live translation, translate the whole history and store it (in memory -- same as original, native language history)
    • every additional person that requests the live-translation in the same language gets the buffered feed
  4. ikusteu commented on Nov 19, 2026

    @ikusteu
    Author

    Open question: How often does the client poll for history

    (I'm separating open questions to draw more attention to things which I didn't know how to answer and would open a discussion.)

    This belongs in the decision related to this feature as Talk is the client polling for history (regardless of the provider).

    Our current implementation (Cassini Live) polls every 3sec (arbitrary interval). However, this should be reconsidered

    Option 1:

    • poll every 3 sec (or similar tight interval)
    • tradeoff: perhaps unnecessary

    Option 2:

    • poll only when gap detected, e.g. message[t].seq = 51, message[t+1].seq = 55 -- gap detected, poll /history
    • tradeoff: something might have gone wrong and the client is just not receiving the feed over HPB (provider might have moved 10 messages, but no /history retrieved)

    My take: keep both, and consider the interval at which the pulse checks for history.

  5. added this to the 🍂 Backlog milestone on Nov 20, 2026
  6. nickvergessen commented on Nov 20, 2026

    @nickvergessen
    Member
    • General contract reads okay
    • Open question: Provider restart -> I'd vote for 1 or 2
    • Open question: How often does the client poll for history -> Option 2

    Proposed Talk UI

    I think this should not be in the sidebar. It's an edge case and mostly temporary from my POV.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions