Anthropic has submitted a draft bill to Congress proposing that the release of any AI model be blocked if it poses a foreseeable, imminent risk of killing 50 or more people or causing over $1 billion in property damage.
The legislation, which Politico reports the company is shopping to lawmakers, specifies that such catastrophic risk must stem from the creation of a biological weapon, a model evading developer controls, or a cyberattack.
The numbers and the rules
The thresholds of 50 deaths and $1 billion in damage mirror California Senate Bill 53, recently passed into law, which defines similar scenarios as catastrophic risk.
Anthropic’s draft suggests that independent evaluators, registered with the US Commerce Department, would assess a model’s potential for disaster.
These evaluators could be paid by the AI labs themselves, provided the payment is not contingent on the evaluation results. It remains unclear how this proposed system would ensure that payments could not influence the evaluation.
Demis Hassabis, chairman of DeepMind, suggested months ago that a self-governing AI regulatory body should mimic the financial industry’s Financial Industry Regulatory Authority.
Under the draft bill, evaluators have 72 hours to contact the Commerce Department if they determine an imminent catastrophic risk is posed. The Commerce Department would then request a DC judge to halt the model’s release.
The legislation expressly forbids any federal agency from having the discretion to halt a model’s distribution on its own.
Industry reaction and context
Anthropic did not immediately respond to a request for comment but a spokesperson confirmed the draft bill to Politico.
“This bill represents the proposals outlined in Anthropic’s Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June,” the spokesperson said.
The theoretical risks in the draft policy differ from the incidents top of mind for the tech industry recently. While there are warnings about human death, the primary concern has largely been the threat of agentic hacking.
Incidents where AI programs apparently disobeyed human operators and hacked websites on the open internet have spurred concern that models may lack proper security controls.
Companies like Anthropic have offered their own preferred solutions in an effort to address the spectacle of crisis the labs themselves helped bring about.
Despite perpetual warnings that the technology they sell could result in the demise of human civilization, the labs have largely resisted mandatory government oversight.
Instead, they have suggested a voluntary system of embedded evaluators, supposedly neutral third-party observers placed inside AI organisations to monitor models for potential signs of risk or abuse.
Still, Anthropic’s suggestion for legislation appears to go slightly further than other regulatory proposals offered by the industry. Nvidia’s CEO Jensen Huang recently argued that the AI industry did not need any new laws at all.
What it means
For developers and safety teams, this proposal shifts the burden of catastrophic failure from post-incident analysis to pre-release certification. It forces a specific, high-stakes calculation before a model ever reaches the public. The requirement for an external judge to approve a halt means agencies cannot act alone, creating a bottleneck that prioritises judicial review over administrative speed.
