Recent reports show foreign scientists attempting to bypass AI safety filters to access information on enhancing pathogens.
The danger exists without artificial intelligence
Public fear often focuses on machines running amok or chatbots helping bad actors unleash a plague. While those scenarios are not impossible, the immediate threat is already here. Anthropic released a report earlier this month detailing five instances where researchers tried to evade guardrails on “gain of function” inquiries. The term describes tinkering with a pathogen’s genes to make it more infectious, deadlier, or better at spreading among mammals.
The company denied those users the answers they sought and closed their accounts. Anthropic noted the users’ motives might not have been sinister. Such information is also needed to develop pandemic countermeasures like vaccines and drugs. Like genetic engineering, AI is a dual-use technology. It can be used for good or evil.
Researchers want to publish risky data
I studied to be a molecular biologist and have covered this debate long before AI entered the mainstream. My experience shows researchers working on high-risk pathogens default to the view that it is better to publish than not. They argue we are better off understanding which genetic traits make organisms a threat to humankind. They often ignore the potentially catastrophic consequences of an accidental release or a deliberate one.
I have doubts about whether humanity is better off doing this work. My concerns grew after reading the latest book from Annie Jacobsen. I last interviewed her about her best-selling book Nuclear War: A Scenario. Her new book, Biological War: A Scenario, came out in late July. It follows the same template. I know you were about to ask, yes, I am sure she is great fun at cocktail parties.
Access to the truth
Jacobsen has decades of high-level national security reporting under her belt. She has unrivaled access to current and former military and government officials and scientists. These people spend their days thinking about threats to humanity and gaming out how the government might respond if such-and-such horrific thing were to happen.
Beyond the notion of human extinction, the most terrifying aspect of her books relates to the limitations in government preparation and response. America’s general plan for nuclear war was, and no doubt remains, complete madness. As for biological war, it turns out we really cannot defend against it.
Michael Mechanic: So, Annie, now that you’ve done both the nuclear scenario and the biological scenario, which is more likely to keep you up at night?
Annie Jacobsen: The Defense Department document that I reprinted in the book, on the weapons of mass destruction threat spectrum, lists nuclear weapons at the far end of the curve because they are the most destructive. But I was really surprised to see how biological weapons are a close second. And I believe the reason they’re the greatest overall threat very much ties into the discussion the entire world is having today, which is that the threshold for entry is extremely low, and therefore it is more likely to happen.
State actors versus everyone else
To have a nuclear weapon, you have to have weapons-grade fissile material. You must think about containing those weapons within the arsenal. You have to have an ICBM, a B-2 bomber, a nuclear-armed submarine.
You’ve got to be a state!
Annie Jacobsen: Yes. A terrorist with a dirty bomb, that’s just a completely different threat. The component that makes biological war so incredibly dangerous and the greatest overall threat is that there are nine nuclear-armed states tasked with this containment component, but then you look at the rest of the world and who has access to [potential pandemic agents] … Most everyone.
Yes. And that becomes terrifying.
The airborne threat
If you were a bioweapons designer, what would you look for in a pathogen?
Annie Jacobsen: My mind does not think that way because I am a mother and I’m a member of society. But If I were a pandemic planner, what would scare me the most is the concept of an airborne agent and the public’s naivete to that very concept. I would think about how important it is to educate the public about the reality of what is an extraordinary threat and what isn’t.
Accidents versus intent
Your scenario involves the accidental release of a weaponized pathogen from a high security lab in Russia. Based on the experts you spoke with, is an accidental release more or less likely than someone deploying a bioweapon?
Annie Jacobsen: Accidental release is positioned as a much greater threat for a couple of reasons. One, I learned there are very serious counterproliferation efforts at the CIA and other intelligence agencies to disrupt bioweapons programs. Whether you want to sleep better at night knowing the CIA is out there countering these things is up to you, but there are examples of biological weapons programs in the hands of terrorists being shut down.
What I came to understand is that the only people crazy enough to deploy or release a biological weapon into the community, knowing full well that the blowback would be extraordinary and would create a pandemic around the world, would be someone who had a death wish. Many terrorists fit that that category, so that is deeply unsettling. But that’s a smaller group.
And then you think about all the people—scientists, nation states, academic institutions—that are manipulating pathogens for greater virulence, for greater transmissibility in these BSL-3 and BSL-4 labs. BSL-4 is the highest security designation. I mean, there are 3,600 of those labs around the world. There are not 3,600 terrorist organizations that I am aware of. So the chance for an accident is statistically far greater. That’s why I begin my scenario with a lab leak, and also because it’s happened before.
One can imagine a researcher sticking themself with a syringe at a BSL-4 lab and freaking out, humans being humans, and kind of going into denial and heading home.
Annie Jacobsen: One hundred percent—and as has happened. Then you have other examples, like [Nikolai] Ustinov, whom I wrote about in a little history lesson in the book, sticking himself with a deadly Marburg virus, from which there is no cure, and doing the good Soviet scientist thing, which is tell on himself, and then be left in a isolation ward behind sort of submarine-lock type doors to die a horrific death by hemorrhagic fever. Accidents happen.
The arrogance of publishing
You and I have both written on the risks of gain-of-function research, like reconstituting pandemic flu strains that can spread among ferrets—god help us if that ever got out. You write in the book that it was after 9/11, when someone was mailing around anthrax, when the riskiest research accelerated.
Annie Jacobsen: Yes, and I think it’s interesting. With anthrax, you’re a “dead-end host,” meaning if you breathe it in, it will kill you but it’s not transmissible. The anthrax wasn’t ever going to become a pandemic. It was meant to make people experience terror. But it’s very different when you’re talking about an airborne pathogen.
And so, flash forward to where we are now, with [advanced] genetic engineering, with gain of function research, with AI-enabled bio, and you have a situation whereby everything is on the table in terms of pandemic potential. You can tweak things. Scientists in a lab can work to create a pathogen that’s able to evade existing immunity, like the mouse pox experiment, or make something airborne, like the H5N1 ferret flu experiments. We are in a completely different world now.
I was taken aback when I called up that flu researcher, how arrogant he was: Of course you have to publish the sequence of this incredibly deadly thing. He didn’t think twice about it. There’s a hubris with some of these guys.
A hubris, or an elitism, where you talk down to the laymen because they are not scientists. As if they can’t make decisions about national security or public health because they are not scientists. I find that incredibly alarming. One of the things that’s been most impactful about my scenario books, both nuclear and bio, is that I have had subject-matter experts around the world, people who write treaties, the cardinals at the Vatican, diplomats at the United Nations, say to me: “Thank you for making this very simple to understand.”
AI is a distraction
Simplicity is powerful! I’m glad you mentioned AI, as we’re being deluged lately with existential concerns, including the Anthropic report about people using Claude to potentially try and facilitate bioweapons work. Your book doesn’t address AI at all. Did your sources have serious concerns about this technology further lowering the bar to creating deadly pathogens?
Annie Jacobsen: Not even a question! I chose to not include AI-enabled bio in the book because I wanted to show readers how easy it is [even without AI] for a scientist in a lab just to tweak Mother Nature’s pathogens and have a pandemic that is a near extinction-level event. But in the acknowledgements, I write about the very frightening concept of mirror life, which David Relman, the biosecurity expert and presidential advisor, explained to me, where you can extinguish all humans on the planet. I wrote Biological War: A Scenario, not Biological War: The Worst Scenario, which is pretty crazy to think about.
What’s been interesting is the subject matter experts I’ve spoken to since the book published. I was recently invited to the BSL-4 lab in Fort Detrick [in Frederick, Maryland] by Dr. Nicholas Bergman, the lab director, to do a Q&A with the staff. He explained that when the book published, the phone started ringing off the hook. Everyone [who works there] was being asked, Is this really true? Could this happen? And people’s next question was, Why don’t we know about this?
Annie Jacobsen: Dr. Bergman said to me, “Annie, you said the quiet part out loud.” And so, I think the fact that AI-enabled bio-threats have exploded into our consciousness is a good thing, because it is making people pay attention to the quiet part.
The threat has always been there, but you are not going to have people who work at these labs raising the flag. That’s not their job, as it was explained to me. Their job is to work on the countermeasures. Many of their questions to me were about AI-enabled bio—they were looking to me for the zeitgeist. This conversation is now happening literally in the grocery store, and also at the highest levels of subject matter expertise.
Official denials
Your book talks about how President Richard Nixon unilaterally scrapped the US bioweapons
