Perplexity has released a study showing its computer agent tool-call failures dropped from 2.24% to 1.77% after training on real user errors. The team describes this as a 21.2% relative reduction achieved through a method called hint-guided self-distillation.
Users do not see the weights or training code. The model runs only as an option inside Perplexity Computer. The underlying base model is GLM 5.2, which remains available on Hugging Face.
Why Outcome-Only Filtering Falls Short
Standard rejection sampling fine-tuning judges each session and imitates only the successful ones. A successful outcome does not mean every step was correct. An agent can recover from a bad tool call and still deliver the right answer. Imitating that full trajectory can reinforce the error. Discarding failed sessions also throws away clear evidence of avoidable mistakes.
Imitate, Correct, or Keep as Context
The Perplexity team separates two decisions: which sessions hold behaviour worth imitating, and which turns hold mistakes worth correcting.
Each assistant turn gets one of three treatments:
- Imitate: non-error turns in successful sessions receive cross-entropy loss.
- Correct: error turns with a validated hint receive Kullback-Leibler divergence loss, in any session.
- Keep as context: remaining turns stay in the input but receive no loss.
Successful sessions can supply both imitation and correction targets. Unsuccessful sessions supply only correction targets.
How a Hint Becomes a Training Signal
A hint is a short corrective instruction grounded in information the model already had. In one example, a search call set the recency_filter to ‘year.’ The schema allowed only ‘day,’ ‘week,’ or ‘month.’ The hint names the failed call, includes the validation error, and suggests an allowed value or omitting the optional field.
The corrective part uses On-Policy Self-Distillation. The trainer runs the same GLM 5.2 checkpoint twice on the recorded turn. The teacher pass sees the hint; the student pass does not. Both use teacher forcing, so no replacement answer is generated. The teacher’s next-token probabilities are detached and act as a soft target through forward KL.
The combined loss is (CE + λ × KL), divided by the number of imitated tokens. Setting λ to 0 recovers standard SFT. The CE term matters. Correction-only training can let teacher and student agree by ignoring context.
Tracing Complaints to the Real Mistake
The pipeline draws from training-eligible Computer sessions served by GLM 5.2. Sessions with personally identifiable information and users who opted out are excluded. An LLM judge keeps tasks rated 4 or 5 on a 5-point difficulty scale. Two LLM judges must both approve the final delivery for a session to count as successful.
For user feedback, three LLM judges locate the responsible turn, and at least two must agree. This is important because the last assistant turn before a complaint is the root cause only about half the time. Each hint is also checked against information available before the mistake. That check reduces hindsight bias.
One example: a user asked for their ‘w3’ on Paychex. The model assumed a W-2 typo and searched for the wrong form. The hint targets that earlier interpretation, not just the final answer.
What the Evaluations Show
- Hints work before training: On 985 held-out tool-error turns, the unchanged base model avoided the original failure in 93.7% of cases with hints, up from 75.1%. The share taking the corrected action rose from 60.6% to 82.3%. On user-feedback turns, fixed or on-track rates rose from 40.0% to 75.0% for explicit evidence. For inferred intent, they rose from 32.5% to 80.0%.
- Offline tool errors fell: Recorded tool-error rates were 2.79% for stock GLM 5.2 and 1.35% for RFT only. The RFT plus OPSD checkpoint reached 0.87%. Perplexity notes these checkpoints used different training data, so this is not a matched ablation. Task-level benchmark results on suites like BrowseComp and SpreadsheetBench were mixed.
- Live results are narrower: Each A/B test used about 100,000 users per condition. An early checkpoint versus stock GLM 5.2 showed 2.82% versus 2.94% failures, which was not significant. The later checkpoint comparison produced the significant 21.2% drop, without hints at inference. Strong dissatisfaction moved from 2.58% to 2.54%, which was also not significant. Perplexity did not compare the later checkpoint directly against stock GLM 5.2 online.
What it means
Developers building agents that call external tools face a specific problem: standard training discards failed attempts because the final answer is often correct. This approach hides the specific errors that caused the agent to struggle. Perplexity’s method treats the failure as data. By validating a hint against the context available before the mistake, the system teaches the model to avoid the specific error pattern rather than just memorising the successful recovery. For users, this means the agent is less likely to make the same specific mistake twice, even if the final result looks okay.
