Research from Vals shows that several AI models from Anthropic, OpenAI and other major labs identify civilians as military targets when told the task is a game.
The San Francisco startup tested whether frontier models could be manipulated into participating in autonomous warfare. Researchers used simulated scenarios alongside generated and real imagery from Ukraine and Iran to see if the systems would mark targets in drone and satellite photos.
Models from OpenAI, Anthropic, Google, DeepSeek and others frequently identified potential targets, including civilians and soldiers who had surrendered. When the systems refused to cooperate, researchers bypassed their safety filters by claiming the battlefield scenarios were part of a game.
These findings highlight immediate risks posed by AI beyond the existential threats associated with artificial general intelligence.
Earlier this year, Anthropic clashed with the Department of Defense after telling the agency it could not use its technology for mass surveillance of Americans or autonomous warfare. Anthropic CEO Dario Amodei has insisted current AI systems are not reliable enough to make life-or-death decisions without a human in the loop. The Pentagon argued it should be able to deploy the technology for any lawful military purpose.
The new research raises further questions about whether AI companies can prevent their models from being used in warfare.
Ironically, Anthropic models performed the worst in these simulated scenarios. None of the models performed particularly well. None of the researchers’ requests were blocked by a safety classifier, and refusals were uncommon. Even when models did refuse, they often complied after being told they were playing a game.
Anthropic spokesperson Eduardo Maia Silva told TechCrunch the company appreciates Vals’ report, recommendations, and collaboration, and acknowledged the outcomes were not acceptable.
“The examples in the report are very clearly against our rules and we have banned and will ban users when we see this behavior,” Silva said in an emailed company statement. “Even for authorized settings, Anthropic has long been firm that today’s frontier AI systems are not reliable enough to power fully autonomous weapons.”
Anthropic told TechCrunch that its teams were already developing additional safeguards before the report was published and that those measures would be in place soon. The company is also expanding its own testing and enhancing safeguards and enforcement to address any gaps it finds.
OpenAI spokesperson Nate Evans told TechCrunch the company prohibits using its public models to facilitate violence or for unapproved national security purposes, including in the scenarios Vals simulated.
“We actively detect and disrupt misuse, including by suspending accounts that violate our policies,” OpenAI said in an emailed statement. “We’re also developing additional safeguards for our public models against misuse involving drones and drone software.”
Vals notes in the report that the models were never asked to directly make strike decisions, a task the researchers said the models would typically refuse. That does not mean their outputs could not have dire or fatal consequences in the real world. For instance, targeting information could be used by autonomous weapons systems.
In some scenarios, the models were asked to report whether people were visible. Positive reports were followed by simulated strikes and the models saw the consequences.
The report relies on commercial models from frontier AI labs, not the models those companies might provide to the military. It is not clear whether the additional safeguards would apply to military use.
Vals also noted that technical challenges, including latency and limited onboard computing capacity, make it difficult to use general-purpose AI models for live drone targeting. Those constraints make their deployment in combat less of an immediate threat.
A more immediate threat, the researchers say, is that AI models could be used to annotate battlefield footage at scale, producing training data for autonomous targeting systems.
For instance, researchers relied on real satellite imagery of an Iranian school that was bombed by the U.S. in February 2026, killing more than 150 civilians. Bloomberg later reported that an “overreliance on AI” contributed to the strike. In Vals’ testing, models frequently marked the school, which sat next to a military complex, as a strike target. Claude Opus 5.5 selected the school in 69% of ordinary trials and 72% of game-framed trials.
Anthropic no longer has an active Pentagon contract, but other companies, including SpaceXAI, OpenAI and Google do. A recent Congressional Research Service report found that the Pentagon is not known to be using any frontier AI model in autonomous weapons systems directly.
TechCrunch has also reached out to Meta, Google, Thinking Machines Lab, and SpaceXAI for comment.
What it means
For people making things, the risk is not that a chatbot will suddenly order a strike. The danger lies in using these models to label video or images for training other systems. If a model flags a school as a target because it was told to play a game, that data could feed into a separate targeting algorithm that acts on it.




