Real-time speaker diarization: the build-or-buy guide

Real-time speaker diarization answers one question while the audio is still flowing: who spoke when. This guide walks the four-step pipeline and shows why clustering is the hard part, explains why offline diarization falls apart on live crosstalk, teaches you to read the DER and cpWER numbers vendors quote, states the session-length limits almost nobody publishes, and works the build-or-buy math, using measured streaming latency from this platform's transcription lane and an honest account of where speaker labels stand today.

Amounts below are US dollar numerals; electricity is in cents per kilowatt-hour. This platform's own rates are never typed by hand: they render from the live catalog in the table further down, and the worked math substitutes only external anchors.

What speaker diarization is, and what it is not

Diarization segments audio by speaker and labels the segments: speaker one said this, speaker two said that. The labels are generic (S1, SPEAKER_A, spk_0, Guest-1), not names. Who spoke is a separate question from what was said, and most products need both fused, which is where the trouble starts (NVIDIA's NeMo Framework documentation frames it exactly this way).

Two boundaries matter before any buying decision.

First, transcription is not diarization. Whisper, in any size, transcribes; it does not identify speakers. Pairing it with a diarizer is a common and workable stack, but it is a stack: two models to run and align.

Second, batch is not real-time. Batch diarization hears the end of the meeting before labeling the start; real-time diarization must label speech while more speech is coming. Every hard thing about this topic, the accuracy loss, the session limits, the crosstalk failures, comes from that one difference.

The four-step pipeline, and why clustering is the hard part

The classic cascaded pipeline has four stages.

1. Voice activity detection (VAD) finds where speech is.
2. Segmentation cuts the speech into short windows, often around a speaker change.
3. Embedding turns each window into a voice fingerprint, a vector close to same-voice vectors and far from others.
4. Clustering groups the vectors and assigns every segment a speaker label.

The first three stages are, in 2026, mostly solved engineering. The fourth is not, for a structural reason: clustering must discover the number of speakers as it goes, with no ground truth, while voices drift, room acoustics change, and two people talk at once. A community thread from July 2025 put it well: modern models distinguish speakers quite easily once the clusters are well formed, even with plain cosine distance, so the hard part is the grouping, not the distance math. A 2024 builder of the same stack by hand, centroid per speaker, similarity threshold for new clusters, found the same wall. That threshold is the whole problem, live.

Newer systems collapse the pipeline into one end-to-end model that emits speaker activity directly; NVIDIA's Sortformer family is the best-known open example. This removes clustering but trades it for fixed speaker capacity: the public streaming checkpoint handles up to four speakers, because the model has four sigmoid outputs, one per speaker slot (model card).

Why offline diarization breaks when you go live

The failures cluster the same way every time. A July 2025 speech engineering thread opened with the complaint that offline models, pyannote among them, "performance drops hard when used in real-time, especially when two people talk over each other", and asked for tools that survive "noisy or fast-turntaking environments"; the answers were hedged, one user reporting a named commercial API as "not perfect". A year earlier, a buyer asking for "affordable and accurate" real-time options never got a straight answer, and a commenter wiring NeMo, pyannote, and Whisper together "haven't gotten it work accurately". These are the median experience, and three mechanisms explain them.

Lost look-ahead. Offline clustering sees the entire meeting before it commits; a person who speaks once in hour two is clustered correctly because the algorithm already saw them. Live, you decide now, on partial evidence that may not yet contain the discriminating moment.

Irrevocable decisions. A commercial vendor's engineering blog states the constraint plainly: once a streaming system assigns a speaker label, the decision is permanent, and the capability "trades some accuracy for speed" (AssemblyAI, August 2026). Offline, a bad early grouping is silently fixed in the final pass. Live, it ships to the UI and stays.

Overlap and crosstalk. When two people talk at once, both voices occupy the same segment. The most honest open-source project in this space says it in its limitations list: two overlapping speakers are not separated, even after its second refinement pass, because overlap-aware transcription is a different and harder capability than labeling turns (hayamimi repository). Fast turn-taking starves it further: a speaker change every two seconds gives the clustering almost nothing to work with. One vendor admits the whole tradeoff: real-time diarization is valuable but expect lower accuracy due to low-latency constraints (Soniox docs).

Thick accents appear in these threads as a first-class requirement, and none of the vendor documentation read for this guide publishes an accent-conditioned accuracy figure. Until one does, the only honest accent test is the one you run on your own speakers before you commit.

How accuracy is measured: DER, cpWER, and when each lies

You will be quoted two numbers. Read them before you trust them.

DER (diarization error rate) is the fraction of scored time the system gets wrong, summed from three parts: missed speech (someone spoke, system heard nobody), false alarm speech (nobody spoke, system heard someone), and speaker confusion (speech attributed to the wrong speaker). Two systems can post the same DER while failing differently, and the confusion component is the one that wrecks a transcript (pyannoteAI's benchmark methodology post breaks DER into exactly these components; the open-source pyannote.metrics toolkit is the field's standard scorer).

DER lies when the benchmark is easy. The same model posts very different DERs on clean two-person calls versus DIHARD III, the community's hardest benchmark, recorded in meetings, restaurants, and courtrooms with no microphone discipline. Always ask: which dataset, how many speakers, and does the score include overlapped speech. NVIDIA's streaming Sortformer, for example, reports DER 13.24 on DIHARD III eval with one to four speakers, zero overlap collar, and a 1.04-second input buffer (model card). Drop the conditions and the number means nothing.

cpWER (concatenated minimum-permutation word error rate) measures what you actually ship: the attributed transcript. Reference and hypothesis are concatenated per speaker, and the score is the minimum WER over all speaker matchings. A diarization swap turns two clean speaker transcripts into garbage on both sides, which is why cpWER is the honest metric for speaker-attributed transcription; the definitions are collected in a 2023 ICASSP paper (arxiv 2211.16112).

For the transcription leg alone, you will see WER and CER (word and character error rate) and RTF (real-time factor, seconds of compute per second of audio; under 1.0 means faster than realtime). A vendor quoting DER for a transcription product, or CER for a diarization claim, is quoting the wrong number.

Our measured numbers, with their conditions

This platform's streaming transcription lane (model id hayamimi-stt, an OpenAI-compatible speech-to-text endpoint) carries dated measurements, stated here with their conditions, because a number without conditions is marketing.

Two honesty notes. Our 0.0000 round-trip is a small, clean, internal set: it proves the hosted pipeline exact on easy audio, not that it beats anyone's benchmark. The upstream project's own scorecard measures 3.8% CER on real Japanese TV broadcast audio (15 clips, remeasured 2026-09-01) and 2.3% WER on English audiobooks (hayamimi scorecard); that is the accuracy basis on difficult material, and ours are the operational numbers for a hosted session. And none of the numbers above are diarization numbers: the lane measured here carries no speaker labels.

The speaker-label gate we do not pretend to have passed

This platform has a designed streaming lane with speaker labels. It is not shipping; this section says why, in the language of the ruling that stopped it.

The design face is a WebSocket endpoint: a 101 upgrade, a JSON config frame (model, audio format, sample rate, channel count), binary PCM frames in, partial and final events with speaker labels out, a close frame flushing the last events. The row was dropped from the catalog on 2026-09-08 after live evaluation: the checkpoint emitted no per-speaker output fields, the event's speaker label stayed constant at "S1" regardless of who was talking, and in-text Speaker markers were observed misattributed, "Speaker 1" where ground truth said voice three. Shipping labels that are confidently wrong is worse than shipping none, so none ship.

Restore is mechanical and gated on three conditions, stated verbatim: (a) a checkpoint emits per-speaker output fields, not in-text markers; (b) it passes the 3-speaker case end to end with correct attribution; (c) the about-220-second edge WebSocket cap is resolved or chunked sessions exist. Until all three hold, no page on this platform will claim speaker labels. What you can buy today is the streaming transcription lane those measurements came from; its event schema includes a speaker field that arrives empty, and an empty field is not a promise.

The open-source path, and what you take on

If you need labeled live transcripts and want to own the stack, three routes exist, each with named costs.

hayamimi locally. A CPU-only, real-time, multilingual transcription project (MIT, no GPU, under 2 GB of RAM) that ships live speaker labels by design: an embedding model tags each utterance S1, S2, and so on by nearest-centroid matching, and a second pass re-diarizes each finalized group with pyannote segmentation and remaps the clusters. On five AMI meetings scored with a 0.25-second collar, it averaged 25.7% DER live and 13.9% after refinement; the project's limitations list says to read the labels as turn markers, not a speaker count, and that overlapping speakers are not separated (hayamimi repository).

WhisperLiveKit with streaming Sortformer. A self-hosted, real-time speech-to-text pipeline (Apache-2.0, about 11,000 stars) whose optional diarization module defaults to NVIDIA's streaming Sortformer. You declare an upper bound of one to four speakers; the model's DER is the 13.24 quoted above, and its license (CC-BY-4.0) permits commercial use (WhisperLiveKit, model card). The older Diart backend requires accepting pyannote's gated-model terms and works on CPU profiles only.

The pyannote or NeMo stack, assembled by hand. The components are excellent; the assembly is yours: voice activity detection, embeddings, and clustering wired into a live loop, plus a transcription model, plus alignment, plus ops. The 2024 thread above is what this route looks like when it half-works.

What you take on with any of them: the process and its restarts, model storage in gigabytes, a CUDA GPU for the Sortformer route, speaker-capacity ceilings you cannot raise without a new checkpoint, upgrades you apply yourself, and your own answer to the accent question. The deeper local-versus-hosted arithmetic for a heavier model family is our Ornith local vs hosted guide; the same logic transfers.

The API path: what to check before you commit

This list changes fast enough that forum answers go stale within a year (in July 2025 a staff member wrote that their company did not yet support real-time diarization; by August 2026 its release notes carry it). Verify at purchase time against the vendor's current documentation, and check five things:

1. Real-time labels, not batch. Ask for the streaming endpoint and the event schema, not the feature page.
2. Label granularity and permanence. Per-word labels (Speechmatics) let you cut turns yourself; per-segment labels decide for you; streaming labels are generally irrevocable.
3. The speaker ceiling, fixed or configurable.
4. Session limits, in minutes; the number almost nobody publishes (next section).
5. The meter: per audio second, minute, or stream-hour, and whether silence bills.

Who ships it, per their own documentation read September 2026? Speechmatics: per-word labels on realtime streams, configurable maximum speakers (docs). Deepgram: a v1 diarizer on streaming, its v2 batch-only (docs). Microsoft Azure speech: real-time diarization with Guest-1 identifiers and "Unknown" on early intermediates (quickstart). Google Cloud Speech-to-Text: diarization on all recognition methods including streaming, minimum and maximum speaker count fields (docs).

Amazon Transcribe: speaker partitioning in batch and streaming, up to 30 speakers (docs). Soniox: up to 15 speakers per session, with its own lower-accuracy-for-latency caveat (docs). pyannoteAI: a commercial streaming product claiming sub-300 ms latency and up to eight speakers (Live-1 announcement). AssemblyAI: real-time diarization in its Universal-3 Pro Streaming release, permanent labels in milliseconds (blog, August 2026).

This platform is not on that list: the transcription half runs, OpenAI-compatible, today, and the label half sits behind the gate described above. If your product can consume unlabeled live text now and add labels when they land, that sequencing works; if labeled turns are the day-one requirement, buy from the list above and keep an exit path. The endpoint conventions either way are in our OpenAI-compatible API guide.

Session length: the limit nobody publishes

Every streaming speech system has a session ceiling, and almost no vendor states it.

Google's quota documentation caps a streaming session at about five minutes of audio (quotas page). This platform's streaming law is that long-form streams are chunked around an approximately 100-second edge wall: a long session is carried by successive chunks, and client code should treat chunk boundaries as a fact of life, not an error. Batch requests are simpler: about 240 seconds of audio per request and a 25 MiB upload cap, with longer recordings split client-side and rejoined. Chunking interacts with diarization labels across boundaries; plan for it before the first long meeting, not during.

One metering detail: this platform bills streaming transcription by audio seconds summed from the final events' durations, not wall-clock. A session with long silences bills its speech, not its silence.

The build-or-buy math

The fence below renders this platform's live rates; the formula reads P_api straight off it. External anchors, each sourced: a rented RTX 3090 at 0.22 per hour (RunPod pricing page, read September 2026); residential electricity at 18.2 cents per kilowatt-hour for 2026 (EIA Short-Term Energy Outlook); board power 350 W, worked examples 450 W whole-system (NVIDIA specification page); used RTX 3090 street prices near 1,000 (September 2026 tracker reads).

api cost per month     =  H x P_api

rented diy per month   =  W x C_gpu        (W >= H for live audio: the pod
                                           runs while the event runs)

owned diy per month    =  G / A  +  H x E x R / 1000

crossover H*           =  ( G / A )  /  ( P_api  -  E x R / 1000 )

The named variables:

• H: audio hours processed per month; H* is the volume where owned hardware and the API cost the same.
• P_api: the per-audio-hour rate from the fence above, or any provider's rate substituted the same way.
• W: wall hours the compute runs. Live audio arrives at 1x, so a rented pod serving one stream runs W at least H, and idle hours still bill.
• C_gpu: rented GPU per hour; 0.22 is the community-cloud 3090 anchor.
• G: what the hardware cost you; 1,000 is the used-3090 anchor, zero if the box exists.
• A: amortization months; 24 by default.
• E: whole-system draw in watts while serving; 450 at the wall for the 3090 tier.
• R: electricity per kilowatt-hour in decimal form; 0.182 is the 2026 national average.

Read the shape before the numbers. Electricity is a rounding error: 450 W at the national average is about 0.08 per audio hour, so the denominator is nearly the whole rate and the crossover is mostly amortization over rate. On this guide's anchors, the used-3090 tier crosses over at about 80 audio hours per month at the fence rate; below that the API is cheaper, above it the card pays for itself inside its window. The rented lane is a margin, not a crossover: a pod serving one live stream costs roughly a third of the API per audio hour while audio flows, but bills through every silence, and the operator, the restarts, and the fourth-speaker ceiling are yours. One 3090 can serve several parallel streams in the self-hosted stacks, the DIY path's real lever at volume.

The sensitivity rule in one line: every 100 of hardware price shifts the owned crossover by about 8 audio hours per month, and every 10-cent drop in the hosted rate pushes it up by about 19 at this tier. The TTS sibling of this calculation, with its own worked tiers, is our TTS local vs API break-even guide; the spend-guard patterns for metered APIs generally are in how to avoid API bill shock.

Run it in five minutes

Two calls exercise the live lane. First, batch:

curl -sS $IRONSTRATUM_BASE_URL/v1/audio/transcriptions \
  -H "Authorization: Bearer $IRONSTRATUM_API_KEY" \
  -F file=@meeting.wav -F model=hayamimi-stt -F language=en

Contract facts that save an afternoon: language is required in practice on this leg; omitting it or sending a non-dedicated code reaches the card's default path and returns a terminal 502, and the fix is a different value, not a retry. The fields prompt, response_format, and temperature are accepted and then dropped; unknown fields are a 400. Containers are WAV, FLAC, or Ogg Opus, up to 25 MiB, effective ceiling about 240 seconds.

Second, the live stream, in Python:

import os
import requests

# feed 16 kHz 16-bit mono WAV (or raw PCM); anything else is a 400
with open("clip.wav", "rb") as audio, requests.post(
    os.environ["IRONSTRATUM_BASE_URL"] + "/v1/audio/transcriptions",
    headers={"Authorization":
             f"Bearer {os.environ['IRONSTRATUM_API_KEY']}"},
    files={"file": audio},
    data={"model": "hayamimi-stt", "stream": "true"},
    stream=True,
    timeout=300,
) as resp:
    for line in resp.iter_lines():
        if line.startswith(b":"):
            continue                  # keep-alive ping before the first event
        if line.startswith(b"data: "):
            print(line[6:].decode())  # JSON events, ending with [DONE]

The response is a text/event-stream of JSON events: transcription.partial while speech is in progress, transcription.final when a segment closes, and one transcription.refine per session, a re-emission of the first final's text carrying the auto-detected language code. The schema includes a speaker field that arrives empty on this lane, as described above. The stream ends with a terminal data: [DONE]; the upstream also closes a session by going silent after its events, a clean end, not an error. Language is auto-detected on the stream leg, so there is no language field to send.

Decision rules

Frequently asked questions

How accurate is speaker diarization?

It depends on the conditions, and any number without them is not an answer. On DIHARD III, the hardest public benchmark, a current open streaming model reports DER 13.24 with one to four speakers and a 1.04-second input buffer (NVIDIA model card); a commercial vendor's own benchmark table puts its Live-1 streaming model at 19.8% DER on the same corpus across languages, 18.7% on English-only, with the other commercial streaming systems it measured between 31 and 39% (pyannoteAI benchmark); and on five AMI meetings with a 0.25-second scoring collar, the honest open-source two-pass scheme averaged 25.7% live improving to 13.9% refined (hayamimi repository). Note what the middle row implies: on hard audio the open streaming checkpoint posts the lower number, and commercial streaming systems cluster in the twenties and thirties. On a clean two-person call all of these look far better, and overlap, noise, and accents move the numbers more than model choice does, so the only DER that governs your deployment is the one you measure on your own audio.

What is the best model for speaker diarization?

There is no single best; there is a best for your constraints. Offline work on hard audio: the cascaded pyannote lineage and NVIDIA's offline Sortformer. Streaming with four or fewer speakers: streaming Sortformer. CPU-only multilingual transcription with turn labels: hayamimi local, the unusual honest pick. cpWER-grade attributed transcripts at scale: the commercial APIs above. Anyone naming one best model without stating dataset, speaker count, and latency budget is selling, not measuring.

What is your target latency and maximum number of speakers?

Latency, measured: first streaming partial at 342 ms after audio start when audio is pushed ahead of realtime (1284 ms paced at 1x), partials arriving about every 229 ms (621 ms paced), and an effective real-time factor of about 0.39, all on a 20.32-second live speech clip on 2026-08-29. Maximum speakers: not claimed; a transcription endpoint without labels has no meaningful speaker maximum to advertise, and the label lane is gated as described above.

Which APIs do real-time (not batch) diarization?

Per their own documentation read September 2026: Speechmatics, Deepgram (v1 diarizer on streaming; its v2 is batch-only), Microsoft Azure speech, Google Cloud Speech-to-Text, Amazon Transcribe (batch and streaming, up to 30 speakers), Soniox (up to 15 speakers), pyannoteAI Live-1 (up to 8 speakers, sub-300 ms claim), and AssemblyAI (Universal-3 Pro Streaming). Verify before you build: this list changed inside the last thirteen months. This platform is not on the list; its streaming transcription lane carries live audio and its label lane is restore-gated.

Why does my offline diarization degrade when used live or on overlapping speech?

Three mechanisms. Offline clustering looks ahead over the whole recording; live, it must commit on partial evidence. Streaming labels are usually permanent once assigned, so an early mistake cannot be silently repaired offline-style. And overlapping speech defeats turn-based labeling outright: two voices in one segment are not separated by systems built to label turns, a limitation the most honest projects state in their own docs. Add fast turn-taking, which starves the embedding windows, and accents, which blur the fingerprints, and the degradation you observed is the expected physics, not a broken install.

How long can a live diarization session run before it breaks?

There is always a ceiling, rarely on the feature page. Google documents about five minutes per streaming session; this platform chunks long-form streams around an approximately 100-second wall, and batch requests top out near 240 seconds. Rejoin chunks at the client and keep labels consistent across boundaries; that boundary code is part of the real work. The chunking and limit math, with the per-provider session caps in one table, is worked end to end in the long audio and long text guide.

Is Whisper plus diarization the same thing?

No, and the distinction matters. Whisper transcribes speech to text; it has no concept of who is speaking. Diarization is separate: either a second model paired with the transcriber, word-level timestamps matched to speaker segments (the WhisperX or pyannote pairing), or one system trained to output both. Asking for "Whisper with speaker labels" usually means the second kind: one event stream carrying text and a speaker field together. On this platform the transcription half is that stream, and the speaker field arrives empty until the gated label lane passes.

The streaming transcription lane this guide measured is the same OpenAI-compatible endpoint the rest of this site documents: current rates sit on the pricing page, and a wallet plus an API key takes minutes on the signup page. Run the two samples on your own audio, your own accents, your own room, and let the numbers decide.

Last verified: 2026-09-11