NVIDIA has released NemotronLabs VoiceChat 11B, an open 11B speech-to-speech model designed for real-time, full-duplex conversation. It processes streaming speech understanding and generation within a single network, removing the need for separate ASR, LLM, and TTS components. This unified approach cuts end-to-end latency to 448 ms for smooth turn-taking on the Full-Duplex-Bench 1.0 metric. The system listens while speaking, allowing users to interrupt mid-turn with a take-over rate of 1.00 at 480 ms. It is also the first open full-duplex model to support tool calling while conversation continues, using a separate output channel for scripts and operator-defined “on-hold” lines to fill gaps during API execution.
Is it deployable?
Partial — deployable today for pilots, not for production. Weights and containers are public, and the license is permissive. However, the NVIDIA team states the checkpoint is ‘ready for research purposes only,’ and the repository documents real failure modes: a two-minute audio context ceiling, degradation into non-recoverable gibberish after several turns, runaway self-talk after a turn ends, and dropped words in user transcription.
- Which companies: any team that can allocate one GPU with at least 80 GB of VRAM — A100, H100, RTX 6000 Pro, or B200 on x86_64 Linux. That covers AI-native startups, funded scaleups, enterprise R&D and innovation labs, GPU cloud providers, and university speech groups. There is no hosted API and no inference provider currently serves the model, so teams without GPU access may not evaluate it.
- Industries: contact centers and CX platforms, automotive in-cabin assistants, retail and drive-thru ordering, telecom IVR modernization, games and NPC dialogue, and accessibility tooling.
- Applications: barge-in-capable voice agents, voice front-ends over internal APIs, live-lookup assistants (weather, pricing, order status), and duplex latency benchmarking harnesses.
Architecture
The model is a hybrid Mamba/Transformer, assembled from three existing NVIDIA components along with one new output path:
- A Fast Conformer speech encoder from Nemotron-Speech-Streaming-En-0.6b, which encodes the incoming 16 kHz stream continuously.
- The NVIDIA Nemotron Nano v2 LLM backbone, which consumes audio tokens and predicts text tokens.
- An NVIDIA TTS decoder and codec that predicts audio codes, rendered as 22.05 kHz agent speech.
- A separate output channel dedicated to tool-calling scripts.
Outputs include agent audio, agent text, and a running user transcription. Training used roughly 550k hours of audio across real and synthetic corpora, building on SALM-Duplex and Audio Flamingo 3.
Tool calling without dead air
Tool calls are emitted on the side channel as a <TOOLCALL> block; your code returns results in a <TOOL_RESPONSE> block. The notable piece is the on-hold message: per tool, an operator defines a line the agent speaks the moment the model generates the text triggering the call, so the conversation does not fall silent while an API runs.
Constraints are explicit. NVIDIA recommends a maximum of five tools per session, the model cannot reliably call multiple tools simultaneously, and the user cannot interrupt the agent during tool execution. System prompts and tool responses must be ASCII-only and TTS-friendly.
Performance
On Full-Duplex-Bench 1.0: smooth turn-taking TOR 0.82 at 448 ms, user-interruption TOR 1.00 at 480 ms, and pause-handling TOR of 0.153 (synthetic) and 0.255 (Candor), where lower is better.
On AU Harness BFCL-v3 spoken tool calling: 58.5% simple, 62.5% multiple, 42.5% parallel, 27.5% parallel-multiple, 89.6% irrelevance, 56.1% average. On Full-Duplex-Bench v3: 82.5% tool selection, 44.2% argument accuracy, 33% pass@1.
NVIDIA reports the model ranks #2 among open full-duplex models on VoiceBench and #2 among open models on Full-Duplex-Bench 1.0.
What it means
Developers building voice interfaces no longer need to stitch together separate recognition, reasoning, and synthesis engines. This single model handles the entire pipeline, including the ability to trigger external tools without pausing the conversation. The trade-off is hardware: you need a single 80 GB GPU to run it locally, and you must accept that it is currently a research prototype with known stability limits rather than a production-ready service.
