Live captions: exactly where your meeting's audio goes

In a Nemi meeting, your camera and your voice go from your browser to the other people in the call. There is no media server of ours that can see or hear them. Live captions are the one feature that sends what is said to a company outside the call, and when a product makes an exception to its own rule the least it owes you is the exact route.

Here it is: when the host turns captions on, the host's browser mixes everybody's microphone into one stream and sends it to Soniox, a speech recognition company, in its European Union region. That audio never passes through a Nemi server. It does leave the host's browser, and it is heard by a company that is not in the meeting. Everything below is the detail of that one sentence.

How do live captions work in a video call?

Every live caption system does the same three things: it streams the audio to a speech recogniser, gets text back while people are still talking, and shows it. What differs is where the audio is sent from and who receives it. Start with the call itself. Each pair of people has its own encrypted connection, browser to browser. On networks that refuse direct connections, usually corporate ones, the encrypted stream goes through a relay we run, which forwards packets it cannot decrypt and keeps none of them. That is true with captions on or off. Then switch captions on and watch what is added.

Three new lines appear, and they carry different things. The thick one is audio: one stream, sampled at 16 kHz and compressed with Opus, sent in quarter second pieces so a word can be on screen while the sentence is still being spoken. It goes from Anna's browser to Soniox's EU endpoint and to nowhere else. What comes back is text. Anna's browser then posts each line to the room over the same channel the meeting chat uses, and our server passes it to everybody else without writing it down.

So the honest summary has two halves. The audio never touches our servers, and the words do pass through them, briefly, on their way to the other people in the call. We keep neither the audio nor those lines; the only lasting copy of the words is the transcript in the host's meeting notes. The privacy policy says both in section 3.9, and names Soniox in its list of the companies we use. Nemi AI makes a similar trade for files, and its post sets out where those go.

Why does the host decide for everybody?

Because one stream is the only shape in which this works well. An earlier design had each person caption their own microphone in their own browser. It sent nothing anywhere, and it could not do the job: many browsers have no recogniser of their own, and three people captioning produced three partial records of one conversation that could not be put together.

One mixed stream is one record, one bill and one switch. It is also the only way the service can tell voices apart, because two voices have to be in the same audio to be compared at all. The cost of that is real: when the host switches it on, everybody's words go to the service, including a guest's, including whatever was said before somebody noticed. That is why only the host has the switch, and why every participant sees a Captions on marker from the moment it is pressed, not from the first word. If you would rather not be transcribed, the moment to say so is in the meeting.

How does a transcript know who said what?

The speech service separates voices, but it only knows them as numbers: voice 1, voice 2. A transcript that says “Speaker 2 will send the list on Thursday” is not much use next week. The meeting, meanwhile, already knows who is talking at any moment, because the host's browser measures every participant's audio level to decide whose tile to highlight.

So the two are lined up on the same clock. Each finished line asks who the level meter said was speaking during it, and adds its length to a tally for that voice. The voice's name is whoever leads the tally. One line can be matched wrongly, when people talk over each other or the meter hands over a beat late. The total over a meeting is much harder to get wrong, and when it changes its mind, lines already written are corrected.

Two cases in that replay are worth stopping on. Ben starts talking while the meter still has Anna as the speaker, so his first line appears under her name. His next, longer line outweighs it, the tally flips, and the notes say Ben for both. Then there is the quiet question at 33 seconds, asked while the meter had nobody: there is no evidence for any name, so it stays Speaker 4. That is a poor label and an honest one. Guessing a participant's name would put words in somebody's mouth, in a document the host keeps.

An unmatched voice stays Speaker 4. A guess would put words in somebody's mouth.

None of this identifies a voice. There is no voiceprint, nothing is compared across meetings, and the tally is a sum of milliseconds that lives in the host's tab and is gone when it closes. What leaves that tab is the name it arrived at, on each line.

How many hours of captions do I get?

Captions cost money per minute the moment they run, so they are counted, in hours per calendar month, against the account that hosts the meeting. Joining somebody else's meeting never uses your hours, and a guest has no account to count anything against. The clock starts when the audio does, so the seconds spent asking for a key are not charged.

When the hours run out, captions stop and nothing else does: the meeting carries on, and the message names whose plan the hours belonged to. The allowance comes back at the start of the next month. See pricing for the plans, and the help page on captions for where the switch is.

What is kept, and by whom?

By us: a count of seconds of captioning per account for the current month, which is a number and nothing else, and the meeting notes, which are the host's own document. We do not hold the audio at any point, and we do not store the caption lines that pass through on their way to the room.

By Soniox: what its own terms allow for audio it is sent, inside its EU region, where both the processing and any storage stay. That part is governed by its terms, not by our retention table, and we say so rather than pretend otherwise.