NVIDIA Releases Audex (Nemotron-Labs-Audex-30B-A3B): A Unified Audio-Text LLM That Preserves the Text Intelligence of Its Backbone

NVIDIA has released Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text large language model. The system processes and generates both audio and speech while maintaining the text performance of its underlying backbone. NVIDIA has made the checkpoints for Audex and a smaller Audex-2B version available under a noncommercial license.

Most multimodal models suffer a drop in text capabilities when they add audio or vision features. NVIDIA notes this issue persists even in models designed for speech-only output. Audex was built to prevent that regression.

TL;DR

• Audex is a single 30B-A3B Mixture-of-Experts model handling audio input and output.
• Audio inputs map to the text embedding space; audio outputs act as text tokens.
• Text scores match the backbone, showing minor gains or losses per benchmark.
• Multi-stage SFT combined with text-only Cascade RL avoids the usual multimodal text drop.
• It is one of the few open models generating general audio beyond speech.

What is Audex?

Audex is a single Mixture-of-Experts Transformer decoder with 30B total parameters and 3B activated per token. The backbone, Nemotron-Cascade-2-30B-A3B, is a text-only MoE LLM. It is a hybrid Mamba-Transformer with 52 layers, using 128 routable experts and 6 activated experts.

The design is deliberately simple. Audio inputs are encoded and projected into the text embedding space. Text tokens and quantized audio tokens are then treated uniformly during generation. There is no thinker-talker split and no stacked cascade of models.

Because the design stays simple, Audex runs on standard LLM stacks including Megatron-LM for training and vLLM for inference. It supports both an instruct mode and a thinking mode. Context length reaches 1M tokens.

How the Unified Design Works

Three components sit around the LLM backbone:

• An audio encoder reads sound. Audex uses AF-Whisper from Audio Flamingo 3. It shares the Whisper Large-v3 architecture and handles 16kHz input.
• Two-layer MLP adapters map audio features into the model dimension.
• An extended vocabulary holds discrete audio output tokens. The original 131,072 tokens grow to 205,312.

Audex uses two codecs for output. Speech uses X-Codec2 at 50 tokens per second. It applies single-layer finite scalar quantization with a 65,536 codebook.

Non-speech sound uses X-Codec at 200 tokens per second. It uses four flattened residual vector quantization layers. Complex sound gets a larger token budget than speech. The interactive demo below computes these token counts for any duration.

Training

Audex needs no audio pretraining. It starts from the text-only SFT checkpoint. Training then adds capabilities stage by stage.

The multi-stage SFT curriculum runs in order: text SFT, audio warmup, audio generation, then audio understanding. During audio warmup, text token embeddings stay frozen. Unfreezing them degraded text quality in ablations.

NVIDIA also tested a single-stage recipe that mixes all data at once. That recipe broke long-context retrieval on NIAH. Multi-stage training avoided this, so it became the default.

After SFT, the research team applies text-only Cascade RL and multi-domain on-policy distillation. Audio tasks show marginal or no regression after this text-only RL. Text scores improve at the same time.

The data mix is large. It combines 157.4B audio tokens and 320.5B text tokens. Tasks span ASR, AST, TTS, text-to-audio, and audio understanding.

Benchmark and Performance

On text, Audex tracks its backbone closely. It scores 86.4 on MMLU-Redux against the backbone’s 86.3. It even leads on IMO AnswerBench, 81.1 versus 79.3. Small drops appear on MMLU-Pro and GPQA-Diamond.

Audex also tops the text-only Qwen3.5-35B-A3B on several reasoning, alignment, and instruction-following benchmarks. The comparably sized Qwen3-Omni-30B-A3B-Thinking shows large reasoning drops versus its own backbone.

On speech recognition, Audex leads these open models. It records 6.82 average word error rate on the OpenASR leaderboard. That beats Step-Audio-R1.1-33B and Qwen3-Omni-30B-A3B-Thinking.

On audio understanding the picture is mixed. Audex leads open models on MMAU. It shows gaps on MMAR and MMSU versus the strongest audio LLMs. Audex also generates general audio, which the other leading open models cannot.

Audex leads on MMAU, Audio Entailment, and OpenASR word error rate. It trails these open baselines on MMAR, MMSU, and BigBenchAudio.

Use Cases with Examples

• Consider a multilingual call center. Audex can transcribe a German call and translate it to English. Its speech translation output lists the source language, transcript, then English translation.
• Consider accessibility tooling. A developer can add fixed-voice text-to-speech to a reading app. The Seed-TTS-Eval English word error rate is a low 1.70.
• Consider sound design or prototyping. A caption like “birds chirping in a forest” yields a 10-second clip. General audio generation uses an enhancement VAE for 48kHz output.
• Consider a voice assistant. Speech-to-speech runs as a cascade, but one checkpoint serves every step. Audex scores 90.0 on BigBenchAudio.

Quick Start Example

Audex follows the ChatML template. The reference container is vLLM 0.20.0. Audio input decoding needs the audio extras.

Audio understanding, ASR, and translation share one audio question-answering format. The <sound> placeholder marks where the audio goes.

[
  {
    "id": "sample_0",
    "sound": "/path/to/audio_0.wav",
    "conversations": [
      {"from": "human", "value": "<sound>\nDescribe the audio in detail."},
      {"from": "gpt", "value": "N/A"}
    ]
  }
]

The model card ships a vLLM audio-QA script for this input format.

# add audio codecs, then run audio QA offline
python3 -m pip install "vllm"

python inference_scripts_vllm/audioqa_scripts/run_audioqa_vllm.py \
  --model-path "$(pwd)/checkpoint_folder_full" \
  --input-json ./inputs.json \
  --output-jsonl ./results.jsonl \
  --tensor-parallel-size 8

For audio understanding, the research team recommends top_p=0.9 and Source Read original →