Model benchmarking is the reputation stake for AI labs, a supposedly neutral system designed to decide if one product beats another.
Irene Solaiman, Hugging Face’s chief policy officer, told TechCrunch Disrupt last week that the methods behind these tests are still maturing. The industry cannot rely on them until testing becomes a consistent science.
Can we trust the current tests?
“Trust is such a weighted term,” Solaiman said. “In terms of whether we can trust results, one big part [of that] is can you reproduce the results of an evaluation? Do you have the level of access that you need to the model itself — this is where open weights come in in a big way,” she added.
Reproducibility has long been a sticky aspect of benchmarking and remains a topic of significant debate.
Outside of an individual test’s validity, there is the added question of how that test compares to other evaluations.
“They are an evolving science — where we don’t know how to meaningfully compare them to each other,” Solaiman said.
“They start to get ineffective relatively quickly, and we don’t have as much of a thriving ecosystem as we need,” she added. “There’s this hunger for an embedded evaluator ecosystem now, but the qualifications that determine who is an effective evaluator across what tasks need a lot more standardization.”
The idea of embedded evaluators
The idea of embedded evaluators, in which supposedly “neutral” third-party observers are placed inside AI organizations, has been popularised over the last several months. AI labs have grappled with a rash of safety incidents that have spurred concerns over the potential harms of their technology. Hugging Face was famously the target of one of those incidents.
Instead of active government regulation to identify and mitigate model safety concerns, labs have pushed for the third-party solution. However, doubts remain over how neutral or effective those evaluators will actually be. Anthropic’s decision to hire Accenture as its first evaluator spurred a good amount of backlash, with some critics mocking the idea that the consulting firm was qualified to do the task at hand.
From model to system
Evaluations are now moving from merely a test of the model itself to a test of the entire system around the model — including the harness and agent. The harness — or the tool the user is using to access the model — specifically, has a lot to do with a model’s performance. A weaker model with a better harness could easily out perform a frontier model, Solaiman shared.
“Agentic evals are really hard and really expensive,” Solaiman said. “We have just more evaluations for models. … We don’t have as many evaluations for harnesses, and we are developing better evaluations for the holistic agent — because that’s what a lot of folks are getting value out of,” she added.
In other words: the evaluation space is a bit of a free-for-all at the moment. If benchmarking is going to continue to be seen as a useful tool for industry leaders, a more uniform, industry-wide approach — where everyone is on the same page — may be necessary.
