NVIDIA AI has released Molt, a reinforcement learning framework built natively on PyTorch. The system aims to reduce the maintenance burden researchers face when modifying algorithms. In standard setups, every change to an estimator or rollout scheme ripples through the trainer, distributed backend, and connectivity layers. That complexity falls entirely on the researcher.
Molt comes from NVIDIA’s NeMo team. It is designed to be compact enough for a human to hold in their head and for an AI coding assistant to read in full. The codebase contains roughly 8,600 lines of reinforcement learning code. By tracing the import graph from the entry point, this number is lower than the 62,000 lines found in verl, the 25,000 lines in slime, and the 7,200 lines in OpenRLHF.
Is it deployable?
Yes. Molt ships under the Apache 2.0 license and includes launch codes, Slurm scripts, and a prebuilt container. The research paper positions it as research infrastructure rather than a production training service. Hardware remains the main limitation. The provided recipes assume two nodes with eight H100 GPUs each, split between eight for training and eight for rollout.
This setup is accessible to frontier and frontier-adjacent labs, well-funded AI startups performing post-training, enterprise research groups in finance, healthcare, and robotics, and academic labs with multi-node H100 or H200 access. Applications include multi-turn tool-use agents, code-execution agents, vision-language environments, LLM-as-judge reward loops, and on-policy distillation onto a smaller student.
Three components, one loop
Molt combines Ray for placement and asynchronous queues, vLLM for rollout, and NVIDIA AutoModel with FSDP2 for training. None of the three is forked. Upstream improvements arrive via a container pin instead of a rebase.
The runtime consists of an agent pool, a set of vLLM engines behind a request router, and a single trainable policy actor. A streaming pool keeps prompt groups in flight so engines never drain while the actor trains. Partial rollout pauses the engines, broadcasts actor shards over NCCL directly to each engine, and resumes retained requests instead of discarding them.
The agent is an ordinary program
An RL run names one Python module that exports an AgentRunner. Everything else is ordinary code, including the reward function.
Two forms are supported. With Env, the framework owns the LLM loop in a Gymnasium-aligned step method. With ChatAgent, the user owns the loop through a stock OpenAI or Anthropic SDK. Molt launches a loopback server that speaks both wire protocols. Every request decodes server-side into one token-exact accumulation. When a long-horizon agent compacts its context and rewrites the prefix, the server seals the current segment and opens a fresh one automatically.
Never train on a token you did not generate
Three correctness invariants organise the design. Token identity: sampled token ids define the trajectory, not a retokenised transcript. Policy-version semantics: trainable tokens keep their behavior-policy log-probabilities, and asynchronous use is corrected per token behind a sequence-level gate. Forward consistency: rollout and actor must agree on model semantics.
For mixture-of-experts policies, the last invariant matters most. The rollout and training routers select experts independently, and small numerical differences can flip top-k choices. Molt applies rollout routing replay, where vLLM returns its per-token expert ids and the training forward replays them.
Interactive explainer
Users can test the framework with an interactive explainer tool.
Key Takeaways
- Molt is an Apache-2.0 agentic RL framework in about 8,600 lines of RL code, roughly seven times smaller than verl.
- Ray, vLLM, and NVIDIA AutoModel are composed, never forked, so upstream releases arrive as a container pin.
- Agents are plain Python; a stock OpenAI or Anthropic SDK trains as-is through a token-exact loopback server.
- Throughput is statistically comparable to a Megatron-based stack, with the MoE-mismatch caveat disclosed.
- Scale is a flag: the same loop runs a dense 4B model and a 700B MoE at –fsdp.ep_size 256.
The full paper is available at https://arxiv.org/pdf/2607.21653. The repository is at https://github.com/NVIDIA-NeMo/labs-molt.
