A swarm of artificial intelligence agents failed to beat a human at fantasy Premier League football, yet the experiment showed models improving rapidly at processing real-world data to make predictions.
I recently deployed a small group of bots to build and manage an experimental squad. The goal was simple: determine if a collection of software agents could assemble a winning team.
I explored this idea a few years ago using a tool from the Turing Institute. The results were chaotic. The system ignored real-world factors like player injuries or managerial changes, relying solely on historical league data.
This time, I asked Codex to build a framework. I fed the model research papers and instructed it to write code that would gather recent news, run predictive models, and guide different agents through specific tasks.
The resulting setup collated far more information than I ever would. However, the initial team suggestions were poor. Leaving Erling Haaland on the bench was a notable error.
I then asked Codex to create a team of agents with distinct roles. Some read news and blog posts; others analysed recent results; more ran prediction models; and a final group critiqued the suggestions. This structure improved the output.
I increased the number of agents to over 100, hoping for better results. The system did not become smarter, though it did increase my API costs.
The improvement from previous years was striking. Haifeng Xu, a computer scientist at the University of Chicago, notes that newer models perform better because they can consume and analyse huge amounts of real-world information.
Xu explains that the fundamental reason for this improvement is that AI is simply getting smarter at understanding the world.
Late last year, Xu tested how commercial models would fare on prediction markets like Kalshi and Polymarket. He and colleagues asked models to guess who would win the NBA Championship and who President Trump would meet with.
Models did very well, sometimes outperforming the average person without special instructions. In their testing, GPT-5 was the best at making predictions, followed by Grok 4, Claude Sonnet, Gemini, and Llama.
The team developed Prophet Arena, a benchmark that measures a model’s ability to make predictions. Another test, Prediction Arena, from researchers at Harvard University and Arcada Labs, gave frontier models $10,000 to wager on prediction markets over 57 days.
The models consistently lost money in that experiment. Researchers saw evidence of them improving consistently over time, however.
Xu states that AI works best with a custom harness. Having agents with unique tasks, such as reading the news or analysing the whole system, seems to be the most effective approach.
AI still often lags behind very skilled experts known as superforecasters. Xu explains that humans cannot consume as much information, but their understanding of the real world is deeper.
There is growing evidence that the human superforecaster’s edge is eroding. A contest called the Metaculus Cup recently showed that AI can sometimes beat even the best human predictors.
A benchmark called ForecastBench could provide further evidence that AI can match human skills when the results of an ongoing contest conclude soon. I expect the big labs to turn their attention to future-gazing as a powerful new way to measure progress in machine intelligence.
All of these predictive experiments suggest that industries predicated on trading may soon be upended by AI. Perhaps the best human traders will take on a new kind of role: supervising and orchestrating AI agents capable of divining the future far better than they can.
For the time being, my AI’s suggestions still do not seem impressive. Its team has not outscored the average over the past weeks. I certainly will not use any of its suggestions in my own selections.
AI will no doubt get better, perhaps making it all too easy to dominate fantasy league pretty soon. I am kind of hoping it won’t, though, as it would spoil the fun of trying to earn bragging rights by winning all by yourself.
