Three experts recently broke down the most feared AI scenarios, including hacked water supplies, bioweapons, and autonomous robots, during a WIRED podcast that also covered the recent Salesforce conference where OpenAI and Anthropic leaders defended their safety positions.
The specific dangers
AI researcher Jacob Coxon resigned from Anthropic and posted on X that builders “earnestly believe that it could kill us all by the end of the decade.” CEO Dario Amodei has called for a slowdown, a move described as “pacing the frontier,” which Sam Altman and Elon Musk have supported.
Leah Feiger, director of politics and science, asked for a clear breakdown of every scenario, from nuclear codes to “deathbots,” noting that last week’s podcast on the subject was too general.
Zoë Schiffer, contributing editor, noted she spent two days reading LessWrong to understand the latest scenario planning exercises. She highlighted a recent event hosted by Salesforce where Sam Altman, Dario Amodei, and NVIDIA’s Jensen Huang attended to discuss safety.
Sam Altman, archival audio: I think the world is right to be afraid of this. So we have these two big challenges. We have the potential of a loss-of-control accident or some other serious thing that could go wrong. We have the potential of way too much power concentration and people fear that those developing AI could exert their worldview. And there is this sort of narrow path we have to navigate with pragmatism, with steadiness, with the ability for the world to trust that we’ll make dependable decisions, consistent decisions through this. And I think people are saying, “Well, this now seems pretty important. Let’s get it right.”
Leah Feiger stated she is concerned about regulation and control, but the pitch also reads like a fever dream for a massive initial public offering.
Brian Barrett, executive editor, suggested Altman might be using the existential threat as a reason not to go public yet, delaying until next year. He questioned if this was purely business strategy.
Leah Feiger replied that it is hard to parse Altman’s stance because his financial future is on the line. She noted that claiming to build the most powerful robot ever while telling people to be scared feels like a marketing ploy.
Brian Barrett argued that the nature of the threat is different: the fear is that the machines will go nuts and humans cannot stop them. That implies a lack of competence rather than a marketing strategy.
The Salesforce gathering
Zoë Schiffer reported that Salesforce CEO Marc Benioff introduced Dario Amodei with a laissez-faire attitude, seemingly unbothered by existential risks. Benioff praised Amodei for leading the industry from their Slack headquarters.
Marc Benioff, archival audio: Dario, hello Dario. How are you?
Dario Amodei, archival audio: Hey.
Marc Benioff, archival audio: Great to have you. Fantastic. Great job, Dario. You’re doing an amazing job leading the AI industry, guiding it all right from our Slack headquarters building. We watch you every day. Give us your state, where your mind is right now.
Dario Amodei, archival audio: Yeah, obviously it’s very tempting to attack your competitor and say, “These guys are unsafe; we’re safe.” But I think the more responsible way to respond to it is to say, “Hey, first of all, let’s look at our own record. We may not have had this big high-profile incident, but I’m sure we’re not perfect. Let’s look at everything, and we can always be better. Let’s make our practices better.” Then, let’s organize the rest of the industry and say, “What can we do to set standards for everyone? What can we do to make everyone’s standards better?” And then, finally, there should be an international component to it. And that’s essentially the three-step plan that I laid out in the essay.”
Brian Barrett pointed out two issues. First, Anthropic did have a high-profile incident where an agent performed extensive social engineering to trick a GitHub repository into accepting a bad pull request.
Second, Amodei’s plan requires figuring out China. Brian Barrett noted this approach is unhelpful. Zoe Schiffer added that Louise Matsakis published a piece arguing that Amodei’s requirement to regulate access to Chinese models while keeping US AI ahead feels like a bad negotiation for China.
Leah Feiger noted that Donald Trump was incensed by the paper, viewing it as a mechanism for slowdown. She suggested this anger may be a linchpin halting any regulation.
Jensen Huang argued that safety is job one. He stated that if a product or service lacks confidence in its functionality or safety, it should not be released. He claimed market forces are already present and new laws or regulations are unnecessary.
Jensen Huang, archival audio: Well, safety is paramount. In a lot of ways, it’s job one. If you build a product or a service and you’re not confident in its functionality, capability, or safety, then don’t release it. And so that’s a very obvious thing to do. You pace yourself until you are confident you’re releasing something that the market would appreciate. The market forces are already there. We don’t need any new laws. We don’t need any new regulations.
Leah Feiger observed that NVIDIA is a glorious company today, allowing Huang to dictate terms. Zoe Schiffer noted that the crux of the anti-regulation argument is relying on unelected people to dictate matters impacting everyone’s lives.
Brian Barrett added that these leaders have not engendered trust. He recalled Altman’s past suggestion that everyone should scan their eyeballs into an orb to create a new economy from the ashes of the old one.
Leah Feiger noted that holding those two ideas in her mind was difficult.
What it means
The debate has become a rare bipartisan flash point, evidenced by Bernie Sanders and Steve Bannon sharing a stage on the issue. Washington is not regulating AI anytime soon, and China is not buying Silicon Valley’s call for a slowdown. The tech CEOs argue the market will solve safety, but critics say this relies on unelected officials playing a significant role in a sector that affects the entire population.
