Supabase has released Evals, an open-source benchmark that measures how well AI coding agents perform on actual Supabase tasks. The tool runs agents like Claude Code, Codex, and OpenCode against real work, such as building a database schema, debugging a failed Edge Function, or fixing a broken RLS policy. Results appear on the public leaderboard at supabase.com/evals and feed an internal suite that runs daily checks.
Deployment requirements
The project is available today under the Apache-2.0 license. You can run the code locally using pnpm.
- Industries: Developer tools, cloud infrastructure, data platforms, and regulated backends in fintech or healthcare, where an agent writing a wrong RLS policy constitutes a security incident.
- Applications: Regression testing for documentation and skill edits, gating SDK releases, and comparing agent harnesses directly.
- Constraints: Running a local stack requires a Docker daemon, provider API keys, and ports 54321 through 54329 to remain free.
How the system works
Supabase organised the test suite across three dimensions: products, topics, and stages. Products include the database, authentication, storage, edge functions, real-time, cron, queues, vectors, and the data API. Topics cover RLS, security, migrations, SQL, SDKs, observability, self-hosting, tests, and declarative schemas. Stages represent the workflow of build, deploy, investigate, and resolve.
The team selected the smallest scenario set that touches each dimension once. Scenarios were grounded in support tickets, bug reports, and GitHub issues.
Scenarios split into two suites. Benchmark scenarios cover breadth and are published for public view. Regression scenarios cover known failure modes, refresh daily, and do not alter published scores.
Every scenario runs against a real environment. The framework boots a hosted-like stack and a local CLI project in containers. Agents call the actual MCP server and CLI. A platform-lite runtime exposes a Management API-compatible surface backed by @supabase/lite. Scoring combines deterministic checks with LLM-as-a-judge. Agents receive one retry before grading.
Each eval directory contains PROMPT.md, which holds the task and frontmatter, EVAL.ts for the scorer, and optional remote or local starting states. Shipping a local workspace or declaring interface: cli boots a Docker sandbox with the real CLI installed.
What it means
The shift to real environments changes the dynamic for developers. Previously, agents might pass tests on synthetic data but fail on live infrastructure. Now, the benchmark forces agents to interact with actual database rows, security policies, and deployment pipelines. This exposes weaknesses in how models handle context when managing complex, production-grade systems. For teams relying on AI to maintain infrastructure, this provides a concrete metric for reliability rather than just code completion speed.
