NVIDIA Releases Nemotron 3 Diarization Open-Weight Speaker Model

NVIDIA on September 23, 2026 released Nemotron 3 Diarization, an open-weight speaker diarization model that identifies up to eight speakers in live or recorded audio, and Baseten announced same-day serving support with batch, streaming, and diarized-transcription presets that each run on one RTX PRO 6000 GPU.

Speaker diarization segments an audio stream by when each distinct speaker is talking, outputs timings for each voice, and assigns every span a consistent label; paired with transcription, it attributes transcribed words to the person who said them. Baseten’s announcement describes Nemotron 3 Diarization as an open, end-to-end model that replaces the usual segmentation-plus-embedding pipeline and its tuning with a single pass, labeling speakers at a configurable interval as low as every 320 milliseconds.

Architecture and Latency Profiles

According to NVIDIA’s model card, the model is designed to determine “who spoke when” in real-world audio, supports both streaming and offline inference, and is ready for commercial or non-commercial use. It is a 100-million-parameter, 31-layer Transformer encoder with Rotary Positional Embeddings. Incoming 16 kHz single-channel audio is converted into 10-millisecond mel-spectrogram frames, downsampled by a factor of eight to an 80-millisecond encoder frame rate, and a Conv1D layer above the encoder upsamples predictions back to the 10-millisecond input resolution. The model outputs a tensor of per-speaker activity probabilities with shape T, 8, and its eight speaker channels are ordered by each speaker’s first arrival in the audio.

For streaming, the model uses the Arrival-Order Speaker Cache and FIFO queue introduced in NVIDIA’s Streaming Sortformer work: the cache retains speaker information from earlier chunks so the first voice heard keeps its label, while the queue provides recent frame context for each processing step. The design requires no embeddings or clustering, and chunked inference places no fixed limit on audio duration.

A single checkpoint serves four recommended latency configurations: 0.32, 0.64, and 1.04 seconds, plus an offline-style 30.4-second input buffer. The card defines this latency as input buffer latency — chunk length plus right context, multiplied by 80 milliseconds — and notes that the figure excludes computational processing time. The checkpoint can run with a buffer as low as 80 milliseconds, though 0.32 seconds is the lowest recommended configuration, and output frame resolution is configurable in multiples of 10 milliseconds.

Reported Accuracy and Speed

NVIDIA’s model card reports diarization error rate, speaker counting accuracy, and speaker counting mean absolute error against the diarstreamingsortformer_4spk-v2.1 baseline, with overlapping speech included and a zero-second collar on every benchmark except CALLHOME-Part2, which uses a 0.25-second collar. On DIHARD III, the card reports a full-set error rate of 12.73 at the 30.4-second profile and 13.55 at 0.32 seconds, versus 19.09 and 19.85 for the baseline. On NOTSOFAR1 single-channel recordings it reports 11.00 versus 30.49 at the offline profile; on AMI Test SDM, 11.14 versus 21.42; on AliMeeting Test Near, 6.40 versus 11.57; and on CALLHOME-Part2, 9.10 versus 10.32. The card states that its AMI, AliMeeting, and NOTSOFAR1 scores were computed with forced-alignment reference labels and that results scored against different reference annotations are not directly comparable.

A speed table in the card reports real-time-factor speedup, defined as total audio duration divided by total processing time, of 1,340 in eager mode and 4,385 compiled at batch size one on the offline profile, measured on an RTX PRO 5000 at BF16 precision. At the 0.32-second profile, the corresponding figures are 12.5 and 54.

Baseten ran its own evaluation, also scoring error rate with overlapping speech included and no collar. It reports a 9.8 percent error rate on AISHELL-4 at the low latency profile against 27.2 percent for Streaming Sortformer v2.1, and says that moving from the 30-second offline profile to the 0.32-second ultra-low profile raises the error rate by only one to two points. Baseten reports the model outperformed Meta Muse Voice Transcribe on every dataset it tested, at even the ultra-low configuration, and lost to pyannote community-1 only on AMI.

Training Data and Licensing

The model card lists roughly 10,000 hours of real conversations (including Fisher English, the AMI and ICSI meeting corpora, VoxConverse, AISHELL-4, AliMeeting, the third DIHARD challenge, CALLHOME, NOTSOFAR1, the DISPLACE sets, licensed David AI recordings, and a pseudo-labeled YODAS-v2 subset) alongside 82,611 hours of multi-talker mixtures simulated with the FastMSS toolkit from about 28,000 hours of single-speaker recordings. Training initialized from a NEST self-supervised checkpoint and ran on eight nodes of eight A100-80GB GPUs in two stages: offline training on simulated data, followed by streaming fine-tuning on a combination of real and simulated audio. Use of the model is governed by the OpenMDW License Agreement, version 1.1.

Baseten Serving Presets and Capacity

Baseten exposes the model through three presets, each running on one RTX PRO 6000 behind a CUDA-graph engine built around NVIDIA’s NeMo streaming loop, according to its Model Library listing, which also describes the model as the successor to Streaming Sortformer v2.1. The Batch Diarization preset is an HTTP endpoint for recorded audio that returns speaker turns with a per-request latency profile. Streaming Diarization is a WebSocket endpoint for live audio that carries speaker identity across a conversation without re-clustering. Streaming Diarized Transcription pairs the diarizer with NVIDIA’s Parakeet V2 speech recognition model, a 600-million-parameter system, and returns speaker-tagged words with timings, per-speaker partials, and overlap flags. Baseten says this preset feeds per-speaker activity vectors directly into Parakeet’s initial encoder layers so overlapped speech is transcribed in a single pass, with speaker labels final on arrival and committed words never revised.

Baseten reports that one RTX PRO 6000 sustains more than 500 concurrent hour-long diarization streams at the 1.04-second profile, 200 streams at the 0.32-second profile, and 190 hour-long streams when transcription is added, while the batch preset processes 200 six-minute audio files per minute. Using identical files and hardware, Baseten reports that its optimized preset delivered roughly 1.35 times higher batch throughput than the NVIDIA reference setup it tested, under k6-generated load inside the cluster with a one-stream control on an idle replica.

Baseten also says the model’s per-speaker activity signal, tracked in 10-millisecond increments, can drive voice activity detection, speaker-specific end-of-turn detection, and turn-taking and interruption handling, responding in roughly 300 milliseconds on ultra-low latency settings.

Nemotron 3 Diarization is available on Hugging Face under the repository nvidia/Nemotron-3-Diarization and in Baseten’s Model Library, with NeMo Framework v3.0 integration and support for NVIDIA Ampere, Ada Lovelace, Hopper, and Blackwell GPUs.