Researchers at Princeton University and UC San Diego ran more than 8,000 test runs to study how “skills” improve AI agents, finding they act as compact instructions for specific tasks rather than simple fact repositories.
The study
Skills are seen as a practical way to make AI agents more capable without retraining them. A new study shows why they work and where they fall short.
At its core, a skill is a compact set of instructions. It spells out the steps an AI agent should follow for a task, what it needs to check, and which common mistakes to avoid. Instead of starting from scratch on every new task, the agent pulls from these stored experiences. Until now, according to a new study, their value was measured only by whether an agent with skills solved more tasks. Why that happened stayed unclear.
A team of researchers from Princeton University, UC San Diego, and other schools dug into that question through controlled experiments. The authors compared how agents behaved with and without a skill on identical tasks across 8,135 test runs.
Skills are a playbook, not a knowledge base
The main finding: skills help mostly because they give agents a reliable process to follow, not because they supply missing facts. This “procedural grounding” accounted for 65.7 percent of the cases where an agent with a skill did better than one without. Directly supplying knowledge helped in just 4.5 percent of the tested cases.
So skills mainly steady the agent’s actions. Which setup steps to run, which tools in which order, which intermediate checks are needed. That clearly cuts certain execution errors, like setting up the working environment or getting output formats wrong.
But skills also create a new source of errors: In 10 percent of cases, the study found, the agent applied an otherwise useful playbook mechanically or in ways that didn’t fit. And on tasks that call for a fundamentally different solution, the wrong skill obviously doesn’t help. An exact match, though, is neither enough nor necessary. Related skills often provide enough direction on their own.
A second bottleneck is finding the right skill in the first place. When the skill library grows from 5 to 100 entries, retrieval precision in actual use drops from 29.6 to 3.3 percent in the tests. Options that sound especially similar make the choice harder.
The researchers argue that skill use should be treated as a lifecycle. Better self-learning agents won’t come from storing more experiences, but from more reliable ways to create, retrieve, and apply them.
What it means
For people building these systems, the implication is clear. Adding a library of instructions does not automatically make an agent smarter. It only helps if the system can find the right instruction quickly. As libraries grow, the chance of picking the wrong tool drops sharply, meaning developers must focus on retrieval methods before they focus on adding more skills.
