Anthropic has filed its initial public offering prospectus, a document that places the company’s formerly private data on public display. While the filing details eye-popping revenue and even larger expenditures, the largest section is dedicated to “risk factors.” These are the specific threats that could derail Anthropic’s plans, or as the document implies, threaten humanity itself.
Although the idea that artificial intelligence poses an existential threat to mankind is widely discussed, the exact mechanisms for such destruction remain unclear. The 94 pages in Anthropic’s filing devoted to risk factors clarify where the danger originates and how the company’s researchers are accelerating development to win the AI arms race.
“Sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight and security measures, pose catastrophic or existential risks to humanity,” the prospectus states. This warning cites concerns from critics, regulators, and within Anthropic itself. The phrase “existential risks” appears three times in the document.
Anthropic links this outcome to a specific process: recursive self-improvement.
The concept behind recursive self-improvement is simple. It describes a state where humans hand over the research and development of AI models to other AI models. Artificial intelligence already plays a significant role here, and at least one lab has used a similar approach to speed up development on a model that performs exceptionally well. The scenario that triggers doomsday talk occurs when humans are no longer involved in the loop.
As the Anthropic filing puts it: “Risks will increase as AI systems substantially automate AI research and development if, for example, they accelerate AI progress beyond our ability to evaluate and safeguard new models.”
The prospectus repeatedly states that recursive self-improvement could lead to AI models developing faster than the company’s safeguards. The company may not realise it has lost that race until it is too late. Anthropic did not immediately respond to a request for comment.
The document also details what might follow. “As AI systems become more autonomous and capable of taking consequential actions with less human oversight,” the prospectus says, “there is a risk that such systems could, whether due to misalignment with human intentions, errors in objective specification, or self-preserving behaviors, take harmful actions independently and at scale without human intervention.”
Anthropic also flags concerns regarding robots. “When our models direct physical equipment,” the filing says, “errors, misaligned behavior, adversarial manipulation, or misuse could result in bodily injury, property damage, damage to costly equipment or experimental materials, or the release of hazardous substances, and such consequences may be immediate and irreversible.”
Given the apparently existential stakes, one might expect serious consideration of whether recursive self-improvement should be pursued at all. In the competitive frontier AI industry, however, it has become a fait accompli. Here is how Anthropic describes that same process in a section about the startup’s “core advantages”:
“We refer to this recursive dynamic as ‘closing the loop,’ the prospectus says, “and we believe it is one of the most significant structural advantages in frontier AI.”
Failure to “close the loop,” Anthropic later says, could result in “falling progressively further behind as competitors continuously use AI to further enhance their own development capabilities.” In other words, the systems causing existential concerns are the ones Anthropic sees as critical to its success.
Every AI company faces this same tension. When former Anthropic researcher Jacob Coxon went viral for loudly resigning, he placed curbing recursive self-improvement at the top of his wish list for the industry. Employees at Google DeepMind and elsewhere have had similar revelations. There are multiple bills in Congress, including one cosponsored by Bernie Sanders, that propose a ban on recursive self-improvement. At the same time, these are for-profit companies, and there are trillions of dollars at stake.
It is a tricky dynamic. Recursive self-improvement could become a runaway train that destroys humanity. Until then, just look at how fast it can go.
What it means
For people making things, the filing confirms that the race to automate research is now an open admission. Companies are prioritising speed over human oversight because the fear of falling behind is considered a greater risk than the potential for uncontrolled AI growth.
