Artificial Analysis has launched Optima, a platform allowing users to construct custom AI benchmarks using their own data or specific workflow descriptions.
Public benchmarks compare models on fixed tasks, yet they rarely reveal which model works best for a specific business need. Optima closes that gap by enabling comparisons tailored to individual workflows. Users can upload existing evaluation datasets from their files or Hugging Face, as well as AI agent traces from Arize, Braintrust, or Langfuse.
Three paths to a custom benchmark
Developers can install a skill to gather information from their coding environment and past sessions. Users without such data can describe their intended use case and provide sample inputs and outputs. Optima then generates suggested test inputs, evaluation criteria, and example tasks. Users review and refine these suggestions through feedback before running the actual benchmark.
Two scoring approaches are available. The first is rubric-based evaluation against objective criteria. The second is a pairwise comparison method. In this approach, users first evaluate a sample of response pairs and indicate which answer they prefer. Optima then derives the full ranking across the test dataset from those preferences.
Cost and speed become first-class comparison metrics
Beyond raw model quality, Optima tracks cost per task and time per task as standalone comparison dimensions. This allows users to check whether a performance gain actually justifies the higher cost or longer processing time of a given model.
For agentic applications, raw token price alone tells you very little. A cheaper model can end up costing more overall if it needs more attempts, fails more often, or requires extra cleanup work. Cost per completed task is often the more meaningful number.
Early testers built benchmarks for finance and accounting agents to find which model could cut costs by a factor of ten without major quality loss. Others tested which model best matched the writing style of lawyers or most accurately identified elements in a proprietary image dataset.
When building and running benchmarks, Optima charges only the actual token costs of the models used, with no markup. Rubric-based evaluations cost $0.125 per criterion per model, and pairwise evaluations cost $0.375 per comparison. At the start of benchmark creation, each benchmark run, and each evaluation round, the platform holds a balance based on a cost estimate. Billing is then based on actual usage and evaluation costs incurred.
Why general-purpose benchmarks fall short
Optima tackles a well-known problem with AI benchmarks. An analysis by Epoch AI showed that benchmark results depend on implementation details that are rarely disclosed. Different prompt wording and temperature settings caused the same model to score noticeably differently depending on configuration. For agentic benchmarks like SWE-bench, simply swapping the scaffold, meaning the agent’s control software and tool environment, accounted for up to 15 percentage points of difference.
An even broader study examining 445 benchmark papers from leading AI conferences found more systemic issues. Nearly all had methodological weaknesses in at least one area, including unclear definitions, unrepresentative samples, and missing statistical validation. Only about 10 percent of the benchmarks studied used complete real-world tasks that reflected actual application scenarios. Key concepts like reasoning or alignment were often poorly defined, limiting the reliability of any conclusions drawn from them.
Optima can address the problem of general benchmarks failing to capture a specific use case. But the deeper methodological challenges of benchmarking don’t go away. Even with a benchmark tailored to your own tasks, its usefulness depends on how precisely the target capabilities are defined, how representative the test cases are, and how the evaluation is implemented and documented.
There’s another limit worth keeping in mind. Even cost and time per task don’t tell you what the output is actually worth to the business. A cheap and fast AI workflow can still be inefficient if its results need heavy rework or add little value to the process they’re part of.
What it means
The tool shifts the burden of validation from the vendor to the user. It forces a move away from abstract scores toward concrete operational metrics like dollars per completed job. This helps teams stop buying the fastest model and start buying the model that fits their specific constraints.
