NVIDIA has released Nemotron 3 Diarization, an open-weight model capable of identifying up to eight speakers within a conversation. The 100 million parameter system runs on Hugging Face and handles both pre-recorded files and live audio streams in a single checkpoint.
Deployment details
The weights are available under the OpenMDW License 1.1, allowing commercial application. Installation requires NVIDIA NeMo and support for Ampere, Ada Lovelace, Hopper, or Blackwell GPUs on Linux systems.
Why speaker diarization matters
Standard speech recognition transcribes text but fails to attribute it to a specific person. Without this attribution, a summariser cannot determine who made an offer or who raised an objection. Diarization solves this by outputting time intervals for each active voice.
Meeting tools, call analytics, and podcast pipelines combine these timestamps with ASR output to create speaker-attributed transcripts.
Improvements over Streaming Sortformer
NVIDIA’s previous model, the Streaming Sortformer checkpoint, managed four speakers. Nemotron 3 Diarization doubles this capacity to eight. The update targets messy, multi-party audio where voices frequently overlap.
Technical architecture
The system accepts 16 kHz, single-channel audio in .wav, .flac, .opus, or .mp3 formats. It converts input into Mel-spectrogram features using a 10 ms step, then stacks them by a factor of eight to create 80 ms encoder frames.
A 31-layer Transformer encoder with rotary positional embeddings processes these frames. A Conv1D layer upsamples predictions back to 10 ms resolution, outputting a tensor of per-speaker activity probabilities.
This design manages overlap directly. If two people speak simultaneously, two channels activate within the same frame.
The model follows the Sortformer approach of ordering speakers by arrival time. The first new voice occupies channel 1, the second channel 2, and so on. This stability prevents the need to re-match speakers for every streaming chunk.
Streaming relies on two memory mechanisms: the Arrival-Order Speaker Cache (AOSC) retains information from earlier chunks, while a FIFO queue supplies recent frame context. Labels remain anonymous until downstream applications map them to real identities.
Latency operating points
Input-buffer latency is calculated as (chunk + right context) multiplied by 80 ms. The following table lists performance based on DIHARD III full-set DER and batch-32 compiled throughput.
Performance metrics
Configuration
Buffer latency
DIHARD III DER
RTFx (batch 32, compiled)
- Offline style: 30.4 s latency, 12.73% DER, 15,113× RTFx
- Low latency: 1.04 s latency, 13.18% DER, 865× RTFx
- Very low latency: 0.64 s latency, 13.28% DER, 579× RTFx
- Ultra-low latency: 0.32 s latency, 13.55% DER, 292× RTFx
These figures exclude compute, networking, and ASR time. While the model technically runs with an 80 ms buffer, 0.32 s is the lowest recommended setting.
Benchmark results
In Voice Arena’s initial Diarization-Bench, the model ranked first among 12 systems and 17 configurations. The test covered 139 English conversations totaling approximately 22 hours. It scored 14.72% DER compared to 19.3% for the next-ranked system, representing a 24% relative reduction. NVIDIA notes results may change after Voice Arena completes its Version 1 evaluation.
Against the 4-speaker baseline at 1.04 s latency, DER dropped across all eight evaluation conditions. Relative reductions ranged from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM. The unweighted mean across the conditions was 41.0%.
There is one regression. On 2-speaker CALLHOME at 30.4 s, DER rose from 5.68% to 5.98%. Full-set CALLHOME-Part2 still improved from 10.32% to 9.10%.
Throughput also increased significantly. At 30.4 s, the model reached 15,113× RTFx versus 2,619× for the baseline. Tests used BF16 on an NVIDIA RTX PRO 5000 with torch.compile(). These are batched numbers, not single-stream application latency.
Training data
Training combined approximately 10,000 hours of real conversations with 82,611 hours of simulated multi-talker mixtures. The mix included real-world multi-speaker audio licensed from David AI. Adding this data cut compound DER from 11.19% to 10.42%. The licensed source audio for the simulated mixtures spans 21 languages.
Getting started
Install NVIDIA NeMo Speech with Python 3.12 or later:
uv pip install 'nemo-toolkit[asr]'
from nemo.collections.asr.models import SortformerEncLabelModel
diar_model = SortformerEncLabelModel.from_pretrained("nvidia/Nemotron-3-Diarization")
diar_model.eval()
segments = diar_model.diarize(audio=["conversation.wav"], batch_size=1)
Output segments take the form start end speaker_id. To retrieve words as well, pair the model with Parakeet TDT 0.6B v3 using the ASR integration guide.
The live demo Space offers synthetic conversations, a live mic, a multilingual live mic, and audio upload. For production, NVIDIA lists Baseten and DigitalOcean. On-device support is available through Argmax Pro SDK 3. The model is not yet available through Hugging Face Inference Providers.
The model has limits. Recordings with more than eight speakers may produce missed or misassigned speech. Heavy noise, reverberation, and far-field capture can also raise error rates.
What it means
Developers can now deploy a single model to handle complex, overlapping conversations without needing separate systems for different speaker counts. The trade-off is clear: pushing latency down reduces throughput, while higher latency improves accuracy. The 8-speaker limit means teams must pre-filter audio or split streams if they expect larger groups, but the commercial license removes legal barriers to adoption.
