Simon Willison released smevals, a new open-source tool for running and grading small evaluation suites against different model configurations.
Willison, who works with Jesse Vincent’s Prime Radiant applied AI research lab, built the framework to answer specific questions about model capabilities.
The software lives at github.com/prime-radiant-inc/smevals. Users instruct their coding agent to run uvx smevals docs to read the documentation, then ask it to construct an evaluation suite.
Once created, the suite exists as a directory containing YAML files. You can execute it against specific models like this:
uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6
The system separates the execution phase from the grading phase. You grade the output runs against your defined checks using:
uvx smevals grade path-to-eval/
Results can be viewed via a local web server:
uvx smevals serve path-to-eval/
Alternatively, the smevals build command generates a static HTML report suitable for hosting anywhere. Willison demonstrated this with a benchmark evaluating how well models write haikus.
The most difficult part of the project was defining the vocabulary. Willison settled on the following terms:
- An eval is a collection of challenges designed to answer a question about a model, for example, how good is that model at generating SVGs?
- Each eval is a collection of tasks. A task is a specific challenge, for example “Generate an SVG of a pelican riding a bicycle”.
- When you run the eval you do so against one or more configs. Each config specifies a model to be evaluated, but may also include other parameters to test, such as different system prompts, model parameters, or agent harnesses.
- A run records what happened when a specific config was used to execute a specific task. A runner is the script that executes a run.
- Once you have collected one or more runs, you need to evaluate the results to see how well the model (or config) did. This is done by a grader, which produces a grade.
- Each grader runs a sequence of checks. These can be simple operations, like checking for a specific string in the output, or confirming that the output is valid XML. They can also be more complicated custom operations (implemented as scripts called checkers), including using other models to answer questions about the run.
Willison has been searching for a usable approach to evaluation for several years. This marks his third iteration on the idea and he feels it is ready for use. He plans to expand the tool and apply it to his own projects in the future.
What it means
Researchers and developers now have a standardised way to test and compare AI outputs without building custom scripts from scratch.
For teams testing models, the tool handles the mechanics of running prompts and scoring results. You define the checks—whether verifying a specific string or running a custom checker script—and the software executes the runs and grades them automatically.
