A British startup is betting that the thumbstick twirls and trigger squeezes of casual gamers hold the key to training the next generation of artificial intelligence.
Many in the industry believe large language models will hit a wall when they try to operate in the physical world. These text-based systems struggle with the finesse required for autonomous driving or robotic manipulation. To fix this, researchers like Fei-Fei Li and Yann LeCun are shifting focus toward world models.
These models require visual and action data to understand real-world physics. Before a system can guide a robotic arm, it needs footage of a factory floor paired with precise details on grip strength and torque. Unlike the oceans of text available for language models, there is currently no comparable dataset for these physics-based systems.
“For world models, you need cause and consequence,” says Xiatian Zhu, an associate professor at the University of Surrey. “On the internet, we have very little of this type of data.”
Worldmodeldata, advised by LeCun, aims to solve this by packaging controller inputs and other data collected by video game studios into training sets. While companies like General Intuition and Niantic already gather data from their own platforms, Worldmodeldata positions itself as a broker. It curates and organises this material so labs do not have to negotiate individual deals with every game studio.
“There are millions of great games, and they are more and more similar to the real world,” says Rhea Loucas, CEO of Worldmodeldata. “Why don’t we take the vast, abundant, diverse experiences from video games, and teach AI?”
Researchers generally assume that world model performance will scale with the size of their training data, mirroring the success of large language models. This shortage of suitable material is currently a major bottleneck for progress.
Some labs have tried to generate their own data by attaching sensors to humans and robots in testing environments. This approach produces only small amounts of data and fails to account for the fringe scenarios a model might encounter in the real world.
“You can pay people to demonstrate pick-and-place tasks. But repetition alone won’t capture the disorder of the world you’re asking a machine to operate in,” says Nicole Fraenkel, a partner at Khosla Ventures, which has invested in General Intuition.
Worldmodeldata argues that data collected from video game environments is both available in the necessary quantities and varied enough to capture all-important corner cases.
“The corner cases are the ones to actually get right,” says Fraenkel. “The cost of error with a car, plane, drone, factory forklift, or autonomous quadruped is very high.”
The startup says it has licensed almost 1 million hours’ worth of data from studios behind various popular video games, though Loucas declined to name them. In the future, the company aims to create avenues for individual players to be compensated.
Loucas believes video game data will eventually make up the majority of training material for world models. These models will later be optimised using data specific to a given real-world environment or task. “This could well lead to the GPT moment for world models—making them really useful,” Loucas claims.
Not everybody shares the same optimism about video game data, though. Nvidia, which publishes a family of world models optimised to run on its chips, prefers to use a custom engine it devised specifically to replicate real-world physics as the backbone for its AI.
Ming-Yu Liu, who leads world model development at Nvidia, says that models trained on video game inputs are unlikely to fare well with tasks that require fine-grained motor control, like the careful manipulation of objects. That is because video game physics is often eccentric, and developers take shortcuts to create the illusion of realism. A character might dip a hand to collect an apple from the table without coding in the details of the pressure applied by each finger to prevent the apple slipping.
“I would be more conservative on using video game data for manipulation,” Liu says. “The physics for manipulation is much more involved.” Training on video game data is better reserved, he says, for world models meant for generating hyperrealistic video or 3D environments.
Zhu, the academic from the University of Surrey, has similar concerns. “Video games are, in essence, simulators. They do have some degree of physical grounding,” he says. “But they are very coarse, approximate.”
But until world models reach their ChatGPT moment, all kinds of ideas for getting them there remain on the table. “There are many paths to the promised land,” says Fraenkel. “The truth is, the jury is still out on which one is going to work best.”
What it means
For people making things, this suggests a future where AI learns from the messy, imperfect physics of games rather than perfect, theoretical simulations. However, the gap between game logic and real-world mechanics means these systems will likely still need significant fine-tuning before they can safely control heavy machinery or delicate tools.
