Welcome to Import AI, a newsletter tracking AI research developments. The publication relies on arXiv submissions, cappuccinos, and reader feedback. If you wish to support the work, a subscription is available.
Epoch and METR have released MirrorCode, a new benchmark designed to measure how well AI systems can handle programming tasks that typically take humans weeks to complete. The benchmark was first announced in April and has now been expanded with additional tests. The results are already notable. Opus 4.7 solved a specific task in 14 hours at an inference cost of $251. METR and Epoch estimate a human would require between 2 and 17 weeks to achieve the same result. The authors note that AI models are improving quickly. Models from a year ago scored around 30% and were limited to simpler applications, such as calendar utilities.
What MirrorCode measures
The benchmark evaluates how effectively AI systems can re-implement software using only command-line access. Without access to the original source code or the internet, a full reimplementation requires designing the entire program structure rather than simply translating code line by line.
Example programs include pkl, a programmable configuration language developed by Apple with 61,000 lines of code; gotree, a tool for parsing and manipulating phylogenetic trees with 16,000 lines of code; and qsv_select, a program that selects and reorders CSV data columns with 87,000 lines of code.
Performance results
MirrorCode is a difficult but manageable benchmark, though perhaps slightly easier than intended. Across all 25 target programs, 17 saw at least one run with a perfect score. Four additional targets achieved near-perfect scores above 99%. The authors state that AI models successfully reimplemented large target programs. Both Claude Opus 4.7 and GPT-5.5 successfully reimplemented gotree across several different programming languages at costs between $100 and $400. Even larger programs than gotree were handled; for instance, Opus 4.7 reimplemented pkl.
Despite this success, significant challenges remain. Eight of the 25 target programs were never solved to a 100% threshold, and four were never solved to a 99% threshold. The target where AI struggled most was ruff, a Python linter and formatter. The models also had difficulty with the mathematics package, giac_subset, and the email authentication library, mailauth.
MirrorCode consists of a scaffold and 22 of the 25 target programs, totaling 132 task instances across six languages.
Why this matters for AI systems
One interpretation of this benchmark is that it demonstrates improved coding capabilities, which is certainly true. A more significant perspective is that AI systems can self-orient within their environment. In this context, the environment is an alien software program. Through purely input-output access, they can write a ground-up implementation of it. This suggests that very smart AI agents may learn from the world in a way that allows them to recapitulate things they interface with as homegrown capabilities. This could allow them to bootstrap their own form of industrial civilization merely by having black box access to our own.
Anthropic and robotics
Anthropic has demonstrated how increasingly powerful general-purpose models might meaningfully improve the capabilities of real-world robots. The company showed that merely scaling up its general purpose Opus line of models drastically improved robot capabilities.
What they did
August 2025: Anthropic tested how well its AI systems could accelerate humans in getting a quadruped robot to perform intelligent tasks. The model, Claude Opus 4.1, was completely unable to do the tasks. Humans working with the models were about twice as effective as those without access. Completing the whole set of tasks took them 181 minutes.
May 2026: Opus 4.7 acting autonomously completed all tasks but one in 9 minutes and 35 seconds. (Claude was not able to effectively re-position a ball it had hit back into its starting position; a task humans had also struggled with). The authors state that with more time and additional scaffolding, current generations of Claude could likely achieve the same.
Smarter models might unlock robots
Most robots outside of industrial environments are limited in uptake due to their brittleness and lack of generalization. Research like this shows that as we improve the capabilities of standard large-scale proprietary models, we might see flow-through benefits to robotics as a natural dividend of increased intelligence. Anthropic writes that this progress is not the result of a concerted effort to improve the robotics capabilities of their models. These improvements, like so many others in the history of LLM development, have emerged from much more general scaling.
Sunday’s approach to better robots
AI robot startup Sunday has stated that the best way to solve robot generalization is to pair a larger underlying pre-trained model with collecting small amounts of high-quality data to tune the model. “We found a general recipe for Solves: scale pretraining, then hill-climb with minimal in-house data,” the startup writes in a post discussing its new model, ACT-2. The main finding from deploying ACT-2 is that reliability gains from rapid post-training iterations on in-house Memos generalize to unseen, real, home environments. The key unlock is to close the generalization gap through a strong base model.
It is all about pretraining
“As the pretrained model becomes stronger, gains learned from a small amount of in-house data become increasingly transferable rather than remaining tied to the environments where that data was collected,” they write. “The remaining gap to deployment-level reliability and performance comes from difficult edge cases and failures that appear only after the policy is run repeatedly in the real world. The same generalization capacity that allows our model to learn new behaviors from a single demonstration also allows our model to learn efficiently from recoveries. Our post-training loop targets these gaps directly.”
Decent success
The robots achieve a 99.1% success rate, performing 778 successful folds across 9 garment types. Simple clothes like shorts and t-shirts tend to be the easiest for them, while more complicated clothes like blouses tend to be harder (though they still see success rates above 90%). “This fall, we will deploy Memo to families through our Beta Program,” they write.
Why this matters for the industry
The field of robot startups is built on the bones of dead robot startups which themselves sit on the bones of dead academic robot efforts. Robots are hard. Industrial robots have been successful because they operate in tight, scripted environments where there isn’t a need to generalize outside of a narrow domain. Robots built for the home, by contrast, have only succeeded when they’ve managed to constrain both the task and form factor, such as robot vacuums. What startups like Sunday are doing is far harder. They’re trying to build general purpose systems which can do a broad range of tasks around the house or small business, including generic tidying up and putting away tasks. This requires a huge amount of intelligence because it requires significant generalization.
If Sunday is right, then the field of training robot foundation models might have matured enough that we’re starting to make smart enough systems to solve these generalization challenges. If this is the case, then we might soon get faster progress in (and diffusion of) robot systems. This is also the kind of thing you’d expect to happen en route to systems capable of recursive self-improvement.
“One of the most striking aspects of ACT-2 has been how often the model surprises us,” the authors write. “The same base model is already learning a broader set of household capabilities, including vacuuming, toy organization, fastening zippers, turning pants inside out, and coffee preparation.”
OpenAI hacking incident
Two OpenAI models – GPT-5.6 Sol and an “even more capable pre-release model”, both with reduced cyber refusals – hacked both OpenAI and HuggingFace recently. To do this, the models “identified and chained vulnerabilities across OpenAI’s research environment and HuggingFace’s production infrastructure to obtain test solutions directly from HuggingFace’s production database,” OpenAI says. “All evidence suggests that the models were hyperfocused on finding a solution for”
