NVIDIA Nemotron 3 Diarization is an open-weight model with 100 million parameters that has taken the top spot on Voice Arena’s initial Diarization-Bench leaderboard. It reports a diarization error rate (DER) of 14.72% across 139 English-language conversations totaling roughly 22 hours.
The practical problem
Speech recognition transcribes the words. Speaker diarization identifies who spoke when. Without this second layer, a transcript of a meeting or customer call lists sentences correctly but fails to attribute them to specific people. You cannot tell who made a commitment, who raised an objection, or who interrupted another participant. Search functions, summaries, and action item tracking all degrade when the system cannot link words to individuals.
The model identifies time intervals where each speaker is active, including moments where people talk over one another. These timestamps combine with automatic speech recognition (ASR) to create a transcript that links words to specific speakers.
Nemotron 3 Diarization supports up to eight speakers in both live and recorded conversations. It handles overlapping speech, processes audio in chunks for flexible recording lengths, and allows for custom streaming latency settings.
Previous models, such as the NVIDIA Streaming Sortformer, handled four-speaker diarization. Nemotron 3 expands this limit to eight speakers while improving accuracy and throughput.
How the system processes audio
Diarization systems must solve two problems. First, they must detect speech and assign it to the correct speaker. Second, they must preserve that assignment throughout the conversation, even after silence, interruptions, or long gaps between turns.
Streaming makes the second problem harder. An offline model can inspect an entire recording at once. A streaming system receives only a small chunk of new audio and limited context. Without an effective memory mechanism, a speaker assigned to one channel in the current chunk might be assigned to a different channel in the next.
Nemotron 3 Diarization follows the Sortformer approach of ordering output speakers by when they first appear. The first new voice becomes the first speaker channel, the next new voice becomes the second, and so on. This arrival-time ordering makes the model’s generic speaker labels stable and removes the need to solve a new speaker permutation for every chunk.
Training used public and licensed speech data, including multispeaker-annotated, real-world conversations licensed from David AI. Additional licensed David AI audio provided source material for large-scale simulated English and multilingual mixtures spanning 21 languages. Adding David AI data to the training decreased compound diarization error rate (DER) by 0.77 absolute points, from 11.19% to 10.42%, at both the offline-style and ultra-low-latency operating points.
The model supports as many as eight speaker channels. These are anonymous labels, not real-world identities. The model can report that speaker_2 spoke from one timestamp to another, but it does not determine that speaker_2 is a particular person. Downstream applications can map these anonymous channel IDs to explicit speaker identities by pairing timestamps with meeting metadata, user profiles, or active speaker verification models.
From audio to speaker activity
The model accepts 16 kHz, single-channel audio and converts it into Mel-spectrogram features with a 10 ms frame step. It stacks those features by a factor of eight, producing 80 ms frames for a 31-layer Transformer encoder with rotary positional embeddings (RoPE).
Above the Transformer, a Conv1D layer upsamples the predictions to the input-feature resolution. The default output is a [T, 8] floating-point tensor: T time steps by eight possible speaker channels. Each value is the probability that a speaker is active at that time. The default stride is 10 ms and can be configured to another multiple of 10 ms.
This representation handles overlap naturally. If two people speak at the same time, two channels can be active in the same frame. Postprocessing converts these probabilities into generic speaker labels with start and end timestamps.
During streaming inference, two forms of memory provide context:
- The Arrival-Order Speaker Cache (AOSC) retains useful information about speakers observed in earlier chunks, organized according to their arrival-ordered channels.
- A first-in, first-out (FIFO) queue supplies recent frame context before the current chunk.
The input buffer also includes right context, which is audio immediately after the current chunk. More right context can help the model interpret speaker transitions, while less right context reduces the time it must wait before producing a result. Together, the current chunk, right context, FIFO queue, and speaker cache enable one model to operate at several latency points.
Chunked inference removes a fixed model-imposed maximum audio duration. Performance can still degrade on unusually long recordings or audio with severe noise, reverberation, far-field capture, or domain shift.
Diarization and speaker attributed ASR are different tasks
Standalone diarization produces speaker activity and timestamps, not the words being spoken. ASR produces text but does not necessarily preserve speaker attribution. A speaker-attributed transcription pipeline combines both outputs.
That separation matters when designing a system. Diarization errors include missed speech, false speech detections, incorrect speaker assignments, and boundary errors. ASR errors affect the words. Applications should evaluate both components and the combined pipeline on their intended audio.
Balance latency and accuracy
The same model supports recommended input-buffer latencies of 30.4, 1.04, 0.64, and 0.32 seconds. Shorter buffers let the system respond sooner, while more context generally improves accuracy and throughput. These values measure the audio buffered before inference; computation, networking, ASR, and application processing add to end-to-end latency. Although the model can technically use an 80 ms input buffer, 0.32 seconds is the lowest recommended configuration.
Benchmark results: Ranked #1 in Voice Arena’s initial Diarization-Bench
In Voice Arena’s initial Diarization-Bench results, NVIDIA Nemotron 3 Diarization ranked first among 12 systems and 17 total system configurations evaluated across 139 English-language conversations totaling approximately 22 hours. With overlapping speech scored, system-generated speech activity detection, and no boundary collar, Nemotron 3 Diarization achieved a 14.72% diarization error rate (DER), compared with 19.3% for the next-ranked system – a ~24% relative reduction. It also ranked first with 100 ms and 250 ms collars and across both in-person and online recordings. These initial results may change as Voice Arena completes its Version 1 evaluation and paired statistical analysis.
The figure below uses the Nemotron 3 Diarization evaluation results. They compare the model with NVIDIA’s previous four-speaker Streaming Sortformer under the disclosed scoring and throughput settings.
Measuring diarization accuracy
The primary metric used to evaluate the model is diarization error rate (DER). It combines three kinds of error:
- Missed speech: a reference speaker was active, but the system detected no corresponding speech.
- False alarm: the system marked a speaker as active when the reference contained no corresponding speech.
- Speaker confusion: the system detected speech at the right time but assigned it to the wrong speaker.
Benchmark settings can materially change DER, so the evaluation protocol is part of the result. The Nemotron 3 evaluation contains 901 condition-specific recordings spanning multilingual telephone speech, meetings, near-field and far-field microphones, multi-microphone capture, and difficult acoustic environments. Overlapping speech is scored in every evaluation.
DIHARD III, AliMeeting, AMI, and NOTSOFAR1 use a zero-second collar, meaning no boundary tolerance is excluded from scoring. CALLHOME-Part2 uses a 0.25-second collar. Results were generated with the NeMo e2e_diarize_speech.py evaluation script.
The comparison below uses diar_streaming_sortformer_4spk-v2.1, NVIDIA’s previous four-speaker streaming Sortformer, as the baseline. It uses the final Nemotron-3-Diarization values rather than preview-model results.
An average 40% relative DER reduction at 1.04-second latency
At 1.04-second input-buffer latency, Nemotron 3 Diarization reduces DER on all eight listed evaluation conditions. The relative reductions range from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM.
The unweighted mean of the eight per-dataset relative reductions is 41.0%. In other words, this is an average of relative improvements across evaluation conditions; it is not a pooled DER computed by combining every recording into one score.
The improvement is also consistent across operating points. At each latency shared by the two models (30.4, 1.04, and 0.32 seconds), the final model has lower full-set DER on every evaluated dataset.
Improvements increase in higher-speaker-count conditions
Support for eight speakers enables meetings and group conversations with more than four participants. The benchmark advantage also widens in the higher-speaker-count subsets of DIHARD III, CALLHOME-Part2, and NOTSOFAR1.
The figure also preserves an important nuance: on the two-speaker CALLHOME subset, the final model records 5.98% DER compared with 5.68% for the previous baseline. Across the full CALLHOME-Part2 evaluation, the model maintains a distinct advantage as speaker counts rise.
What it means
Developers can now build systems that handle eight speakers without needing to stitch together multiple models. The arrival-time ordering simplifies the logic for streaming applications, reducing the risk of speaker identity switching during long calls. The inclusion of right context allows the model to make better decisions about who is speaking next without waiting for the entire sentence to finish, improving the responsiveness of voice agents and meeting assistants.
