New benchmark DiG-bench shows that Fable 5 and Opus 5 are currently the only models capable of solving the hardest tier of discovery games.
The new frontier for analysing AI systems is understanding how good they are at inferring the unwritten rules of their environment. How well can AI systems figure out the rules of their environment through exploration and curiosity, versus being fed them? That is an important question for better understanding the intuitive and creative capabilities of AI systems and it is one being asked by DiG-bench (Discovery in Games). It is a new benchmark of 70 games designed to map the surface of discovery in well-controlled interactive systems.
Similar to the visual ‘ARC’ game, in DiG-bench each game is a self-contained miniature world with its own laws, but both the rules and the objective are hidden from the player and must be uncovered through interaction. You can play some of the games yourself online to get a feel for them at the official project website (digbench.ai).
The key thing this is measuring is the ability for players to spot the important mechanics that determine their success. Basically, by playing around with the games you get a sense for how your actions change the environment and through this you also uncover mechanics that you must understand to succeed at the game. The idea is that if you can solve these games you have a decent ability to spot important information in novel environments and update your priors.
Who did the research: The authors come from Thinking About Thinking, University of Oxford, Princeton University, King Abdullah University of Science and Technology, Swiss AI Lab, Inria, MIT. One of the authors is Juergen Schmidhuber, an extremely creative OG AI researcher.
Key facts:
- Purely text-based: The games are basically native to language models. They are also mostly “short enough that most traces fit entirely within the context window of current frontier models”.
- Handcrafted and novel and private: All of these games have been built by human experts. The majority of the games are kept private so that AI systems don’t train on them.
- Beatable but difficult: Every game has been beaten by at least one human “but players reported finding many games difficult”.
- Varied skills: Solving all these games requires different skills and strategies.
- Experimentation: The games come with an optional experimentation mode which lets people play around with them without having as intense a “step limit” on actions they can take.
- Reassuringly hard: The games are difficult enough that they are not beatable by today’s frontier models.
How well do AI systems do? The benchmark is split into seven tiers with tier 1 being the easiest and tier 7 the hardest. 21 games have been released publicly with the remaining held back. Most of the games have multiple levels and the number of available actions for players to take at each step ranges from 2 all the way up to 34.
- Opus 5 and Fable 5 with Claude Code are the best overall models, followed by GPT-5.5
- Only Opus 5 and Fable 5 were able to beat any tasks (0.2) in (Tier 7). Opus 5, GPT-5.5, and Kimi K3 were able to beat some tasks in Tier 6 when given access to a harness (e.g., Claude Code).
- GLM-5.2 and Gemini 3.1 Pro were able to beat some levels in Tier 4.
- Overall, this seems really hard!
Why this matters – proxies for creativity and discovery Tests like this are attempts to isolate a prerequisite for creativity, which is being able to autonomously discover useful undocumented things about novel situations you find yourself in. As this test shows, some frontier models are already capable of some fairly impressive feats of discovery, but still struggle compared to humans (for instance, a 20% success rate on Tier 7 is pretty poor compared to the fact individual humans were able to get 100% on the tests). My guess is we will reach human parity on DiG-bench by middle of 2027, at which point we should expect things like recursive self-improvement to seriously kick off.
Read more: DiG-bench: Discovery in Games (GitHub, PDF).
Play the games and view the leaderboard at the official site (digbench.ai).
***
Get a feel for recursive self-improvement by playing this browser-based game:
…Cookie Clicker, but for the singularity…
Here is a fun game from the folks at Paradigm Research which aims to simulate what it is like to run a company building AI systems which become capable of recursive self-improvement. If you play the game you can get a good feel for how different components of AI research interact, ranging from how you balance investing in researchers versus compute, how and when to license data, and more. Be warned, it is hard – but then again, so is frontier AI development.
Why this matters: Developing better intuitions about recursive self-improvement is of existential importance to us all; games like this help make it easier for us to reason about this technology and the labs building it.
Play the game here: RSI Simulator (Paradigm Research).
***
AI systems are showing early signs of scientific research taste:
…Inherent post-trains an open weight model into an AI scientist that supervises a frontier model…
Taste is a hard thing to quantify but an intuitive thing to sense, as any of us know who have sat in a well-designed room, looked at someone wearing a particularly good fit, or read a research paper that asks just the right questions. Now, researchers with AI startup Inherent have published a paper showing how they are building Faraday, an AI scientist model that they hope can develop some taste in terms of research.
What they did: The company built a supervisory harness and relatively small LLM which sits on top of large, proprietary frontier models, and controls them in a way that improves their effectiveness at science. (In some ways, this is a capabilities-centric version of the scalable oversight problem).
To help them train and evaluate the system they assemble a dataset (“Replica”) consisting of research papers that have key graphs or results missing from them, then they see how well AI systems can autonomously do experiments that fill in the blanks, and they continuously train a small supervisory model (“Faraday”) via GRPO on well-designed fill-ins to achieve better and better results.
Faraday is a 27B model that uses a coding agent (OpenAI Codex) as an underlying tool and is post-trained on top of Qwen-3.6-27B.
What Replica consists of: Replica is a set of 100 ML and AI-for-science papers published between 1990 and 2026. The authors convert this dataset into a set of 310 replication tasks by knocking out individual results. “For each task, we use Claude Opus 4.7 prompted with a meta-rubric to generate a task-specific grading rubric,” they write. They then use a Codex-based Judge model to provide “an overall reward and per-turn credit assignment weights, which are used to train the Faraday agent using a modified version of GRPO.”
Results: Faraday using Codex is able to beat standard Opus 4.8 and GPT-5.5 on some replication tasks, exceeding their performance “on 73% of in-distribution ML tasks, and on 60% of held-out AI-for-science tasks, according to our rubric-based judge.”
“We achieve a comprehensive uplift in performance compared to the base Qwen model, on both train and test tasks,” they write.
Why this matters – the better systems like Faraday get, the higher the chance AI systems will become capable of recursive self-improvement: These days, most high-signal AI evaluations are trying to capture some property of creativity and intuition and Faraday/Replica is the same. The better AI systems get at this, the more likelihood we can assign to the idea that AI systems will imminently become capable of building themselves.
“The skills that allow Faraday to fill in vaguely-specified details may be the very same skills that would allow it to advance the state of the art by designing its own experiment,” the company writes. “The skills Faraday acquires – deciding what to investigate, scoping experiments to a budget, and judging a replication – compound with advances in frontier coding models. One might hope that a single post-trained outer agent can track the frontier as better models are released, at least over some time period.”
Read more: Training AI Scientists to Replicate Research (arXiv).
***
Mark Zuckerberg seems to be a technological pessimist:
…Zuck’s big essay on AI seems to ignore or elide or not confront what AI systems capable of invention mean…
Mark Zuckerberg has written an essay called “The Future is for Everyone” that serves as something of a manifesto for how he and Meta are approaching the development of AI systems. The core idea inherent to Zuck’s strategy is to massively proliferate AI capabilities to everyone on the planet in a bid to avoid concentrating power and creating tyranny in a small number of players. It is a broadly sensible idea except for the fact that superintelligences capable of inventing new ideas might want to do different things to what Mark Zuckerberg proposes and on this crucial area his essay is silent.
Mark’s view: “The defining questions of our age are who will have access to superintelligence and what will we direct it towards,” Zuckerberg writes. “We propose a philosophy based on individual empowerment as the source of prosperity, invention as the primary purpose of superintelligence, and balance of power as the foundation of safety.”
What it means
These developments highlight a narrowing gap between human and machine intuition in closed systems. The ability to discover hidden rules without explicit instruction is a prerequisite for autonomous scientific progress. If models can replicate missing data points in papers, they are likely on the path to designing their own experiments. The difficulty of DiG-bench suggests we are not there yet, but the trajectory points toward human parity within the next two years.
