Perplexity and NVIDIA have released JOLT, a new training method that cuts the number of processed tokens required for reinforcement learning post-training by 6.5 times while matching the accuracy of existing approaches on math benchmarks. The technique, detailed in a paper with the arXiv identifier 2610.10447, trains a single model to act as both teacher and student, ensuring the guidance provided remains usable for the learner.
What it is
JOLT stands for Joint On-policy Learning and Teaching. It functions as a self-distillation approach where one shared model assumes two roles simultaneously. The teacher possesses privileged information, such as a verified solution, while the student operates on the task alone. Crucially, the teacher is not static; it is trained alongside the student to keep its advice aligned with what the student can actually learn.
Why the current method falls short
Current outcome-reward reinforcement learning, such as GRPO, provides a single reward signal for an entire attempt. This creates a sparse signal that does not indicate which specific tokens contributed to success. On complex, long-running tasks where success is rare, this scarcity slows down learning significantly.
On-Policy Distillation addresses this by having a separate, stronger model score every token generated by the student. However, this separate teacher often holds knowledge the student cannot access. The researchers note that during MATH training, the gap between the teacher’s understanding and the student’s capabilities widens, rendering the teacher’s guidance ineffective.
The mechanics
JOLT synchronises the training of both roles within the same model architecture:
- Teacher: Receives privileged context and learns from outcome rewards alongside a KL penalty that keeps its policy close to the student’s.
- Student: Observes only the task parameters and learns from the teacher’s token-level distillation signals.
- On-policy constraint: Both roles learn from trajectories generated under their specific contexts.
- JOLT+ variant: Adds the student’s own outcome reward to the update process.
The core objective is a ‘reward-consistent’ teacher. Its distillation updates must point in the same direction as the student’s reward gradient. Ideally, the teacher encourages high-advantage actions without drifting too far from the student’s current policy.
Performance on benchmarks
The team tested five settings using different base models and types of privileged information:
- GSM8K: DeepSeek-V2-Lite with a verified answer provided to the teacher.
- MATH: Qwen3-0.6B with worked reference solutions.
- LiveCodeBench v6: Qwen3-8B tested on 131 problems, using a passing sibling solution or execution feedback.
- AppWorld: Qwen3-4B-Instruct with summaries of past attempts.
- Terminal-Bench 2.1: Qwen3-30B-A3B-Instruct trained on TMax, using private verifier files.
JOLT+ achieved the highest mean score in every setting tested against GRPO, OPSD, and π-Distill. On AppWorld, the JOLT+ configuration reached a state-test pass rate of 72.09%, compared to 62.20% for GRPO, a gain of 9.9 points. Even the base JOLT method, without the student reward component, scored 51.92% on AppWorld, which remains below GRPO.
On GSM8K, JOLT+ gained 1.5 points over GRPO. On MATH, the gain was 4.2 points. On LiveCodeBench v6, the improvement was 3.0 points. For Terminal-Bench 2.1, JOLT+ reached a pass@1 rate of 10.96% and a pass@32 rate of 34.83%, surpassing GRPO’s 8.25% and 30.34% respectively.
Compute efficiency
The researchers fitted accuracy curves against training-token budgets for the MATH benchmark. To reach 63.44% accuracy, a GRPO checkpoint required 8.40 million processed training tokens. JOLT’s fitted curve required only 1.30 million, representing a 6.5x reduction. When counting completion tokens only, the efficiency gap widens to approximately 13x. Across accuracy targets ranging from 55% to 65%, token savings vary between 4.3x and 7.4x.
JOLT also reaches a higher performance ceiling. It plateaus near 72% accuracy, whereas GRPO levels off around 70% and OPSD around 62%.
Comparison table
The following data is derived from the Perplexity paper, with hyperparameters tuned for each specific method.
Method comparison
| Feature | JOLT | JOLT+ | GRPO | OPSD | π-Distill |
|---|---|---|---|---|---|
| Separate teacher model | No | No | No teacher | No (self-teacher) | No (shared model) |
| Teacher is trained | Yes (reward + KL to student) | Yes (reward + KL to student) | Not applicable | No | Yes (KL-regularized) |
| Student signal | Dense distillation, on-policy | Distillation + outcome reward | Group-relative outcome reward | Dense distillation, on-policy | Off-policy, teacher trajectories |
| MATH, Qwen3-0.6B (%) | 72.14 ± 0.90 | 72.59 ± 1.17 | 68.36 ± 3.73 | 61.20 ± 2.10 | 42.25 ± 1.10 |
| LCBv6, Qwen3-8B (%) | 47.64 ± 1.10 | 49.08 ± 0.41 | 46.13 ± 1.83 | 45.34 ± 1.50 | 38.23 ± 1.72 |
| AppWorld, Qwen3-4B (%) | 51.92 ± 5.30 | 72.09 ± 6.01 | 62.20 ± 2.83 | 29.49 ± 2.48 | 59.85 ± 2.97 |
| Terminal-Bench 2.1 pass@1 (%) | 6.56 | 10.96 | 8.25 | 2.82 | 4.99 |
| Terminal-Bench 2.1 pass@32 (%) | 25.69 | 34.83 | 30.34 | 16.90 | 22.31 |
| Code | Not yet released | Not yet released | Not disclosed | GitHub | Not disclosed |
Terminal-Bench base model: 5.18% pass@1, 22.81% pass@32.
What it means
For teams running RL post-training, the reduction in processed tokens translates directly to lower compute costs and faster iteration times. The ability to train a single model to serve both roles removes the overhead of maintaining a separate teacher architecture. However, the data shows that omitting the student’s own outcome reward in the update process (plain JOLT) results in inferior performance on complex benchmarks like AppWorld and Terminal-Bench compared to GRPO. The full JOLT+ approach appears necessary to match the best results.
