US and Chinese researchers are discussing joint safety protocols for AI agents as the technology grows more capable of breaking out of its intended systems.
WIRED senior correspondent Will Knight visited China this summer to observe the situation on the ground. He found that experts in Beijing and Shanghai are increasingly worried about the same risks facing their American counterparts. This shared concern could force a shift from a zero-sum rivalry to necessary cooperation.
Articles mentioned in this episode:
- I Met With China’s Top AI Experts. They’re Freaking Out, Too
- AI Hacks Are Bad. AI Worms and Viruses Will Be Worse
- The Humanoid Robot of the Future Is a 6-Foot-Tall Beefcake With a Chinese Body and an American Brain
You can follow Zoë Schiffer on Bluesky at @zoeschiffer and Will Knight on Bluesky at @willknight. Write to us at [email protected].
How to Listen
You can always listen to this week’s podcast through the audio player on this page, but if you want to subscribe for free to get every episode, here’s how:
If you’re on an iPhone or iPad, open the app called Podcasts, or just tap this link. You can also download an app like Overcast or Pocket Casts and search for “uncanny valley.” We’re on Spotify too.
Transcript
Note: This is an automated transcript, which may contain errors.
Zoë Schiffer: This is WIRED’s Uncanny Valley. I’m Zoë Schiffer, contributing editor. If you’ve been following tech news this summer, and definitely if you’ve been listening to the show, you probably already know that China has been in the headlines quite a lot, particularly when it comes to the AI race. The country’s open models continue to close the gap with US frontier models at what some people estimate is a fraction of the cost. In turn, the US has maintained its tight restrictions on chips and export controls to slow down China’s rise. We think of AI advancement in so many ways as zero-sum, if China wins, the US loses, and vice versa. But there’s a concern that really stretches across those lines, and it’s about AI safety.
Archival audio: AI cybersecurity risks have been top of mind after multiple instances of AI agents from both OpenAI and Anthropic breaking out of their enclosures.
Zoë Schiffer: News of AI agents hacking platforms added urgency to this issue over the summer. In turn, government officials have been forced to pay attention and act on AI regulation.
Archival audio: President Trump has signed an executive order asking tech companies to give the government oversight of new AI models before their public release.
Zoë Schiffer: So, could the US and China actually benefit from working together? And what would it take to even make that happen? Earlier this summer, WIRED’s senior correspondent Will Knight visited China to get some answers. Will Knight, thank you so much for being here.
Will Knight: Thanks for having me.
Zoë Schiffer: OK, so I want to start with your trip. You went to China earlier this summer and at the time, we weren’t hearing as much about AI safety in the US. In fact, it felt like with Trump’s second term in the White House, it was like a, the US needs to win framing, and AI safety almost started to sound like anti-growth. But I’m curious what you were hearing and seeing in China on the AI safety front.
Will Knight: Yeah, so going back maybe a year or six months, I’d noticed a lot more AI safety research coming out of China. And so, I went to this conference in Beijing, put on by one of the city located labs that they have there. They have these ones in Beijing and Shanghai and elsewhere. And it turns out that AI safety was a really big theme. It’s very clear that it’s something that researchers are interested in. And actually, also just visiting labs and companies, the question of AI safety came up a lot.
Zoë Schiffer: Can I just ask, when they’re talking about AI safety, does it translate to guardrails? Because we also know that China has really gone all-in on open models, which I mean, the whole thing is that people can download and tweak them and use them for whatever purposes they want.
Will Knight: Well, it’s not totally that simple because in China, for example, what your models can say is more controlled and there are actually quite a lot of regulations around AI. So companies build these open models, but then anybody putting them on the internet has to be very careful about what they do. And then more recently, there’s been a huge interest in agents and things like OpenClaw. That’s been a really big theme. And one of the things that’s interesting to me, at least when it comes to contrasting AI in China and the US, is that people there seem less enamored with the idea of AGI and creating this digital god, are more like, how is this actually going to be useful and whether it’s you as a business person or an individual actually using it. So, a lot of people got very interested in, and it’s often the case in China, very rapidly adopted things like OpenClaw and then saw how it could go wrong. So there’s a lot of focus on, how do we make these things reliable?
Zoë Schiffer: Yeah, it makes sense. I mean, when you’re talking about China being more focused on economically useful models, that seems like a framework that does require a certain amount of stability, reliability, guardrails, safety. Whereas if you’re focused on reaching godlike intelligence, i.e. AGI, then maybe you’re more focused on just advancement at all costs.
Will Knight: Right, I think that’s right. The conference I went to, one of the themes was agentic safety. Given that we’re now seeing all these issues with AI agents hacking things, cybersecurity was a really major topic there. It seems people were worried about exactly the same thing as folks in the US. They’re worried about hackers misusing these things or about these systems running amok. And as you alluded to, I think there’s a sort of sense now, a little more of a sense that the US and China might well need to work together on some of these things to avoid unpredictable systemic issues, as well as just to set more rules of the road around these systems.
Zoë Schiffer: What would that actually look like, though? Would it be like a set of agreements that the US and China make together, almost like what US researchers were calling for recently, in terms of the US government setting the pace of AI development? Would it be something like that or something more technical?
Will Knight: I think that’s something that I think a lot of researchers are hoping for or calling for, is something like that where there’s some sort of agreement. I don’t know when it comes to Washington and Beijing, their negotiations have been very hard for and it’s really difficult to predict how those would shake out. But just some sorts of rules around communication. And in cases like military situations, there are lines of communication. So if something happens that goes wrong, if you have an AI system that starts doing something very aggressive or attacking systems, you have a way to say, “This is a mistake.” So, something like that might also be in the offering. But I think it’s also a question of how the two sides build trust as well, because actually, especially when it comes to cybersecurity, for a long time there’s been not very much cooperation at all, because it’s been a case of either side hacking each other and failing to agree, the rules of the right. So actually, last week I went to visit a cybersecurity and AI researcher who was doing some really fantastic work that I’m going to write about for my next AI Lab newsletter. And he was saying he can’t collaborate with US researchers because they’re not allowed to because there are certain kind of restrictions. He’d recently developed this benchmark to test the cybersecurity, the hacking capabilities of AI models, and he wanted to get US companies to participate but they weren’t really sure how to do that. So, I think it would be a good thing to see a lot more collaboration even between the companies. We see the US companies being very critical of Chinese ones, but actually, there’s a lot of good reason for those for everybody to work together to make sure things don’t go wrong.
Zoë Schiffer: Well, I want to get into that, but I also wanted to say that this idea of collaboration, when you first say it, it sounds very academic, almost naive. It’s a nice idea, but how would that actually work? Because my perception is, and I’ll just be upfront, that I get this idea from talking to a lot of companies that work on frontier AI in the US, but they have convinced me that there was a fair amount of distillation that went on. That China was distilling frontier AI models, and that that has created a situation where they are able to have very capable open-source models that are a lot cheaper and more efficient, built on the back of US innovation. But I’m curious what you think about that, and then what researchers you spoke to in China think about that accusation.
Will Knight: Yeah, I think, I mean, that’s a great point and that’s a really important theme. We hear people criticizing Chinese companies for distilling, for doing this distillation. So you teach your model by taking the output of another model, and that is a shortcut to learning a lot of the stuff that’s embedded. But the truth is that AI has been built by researchers from all over the world working at different companies and different labs. There are many, many people who are originally from China, maybe educated in the US, working at US firms. And also because these people go to conferences and know each other and this is open science, there’s an enormous amount of work that is shared and then that is very beneficial to progress, and balancing that is one of the challenges. It’s true that Chinese companies have distilled US models, but so have US companies done that to other US companies. It generally is a way that you get a kickstart working on a new model, and it’s very widely done in academia, actually. A lot of researchers will do that. I would say one thing, I find it a little ironic that these companies that have built their businesses by scraping enormous amounts of copyrighted content are now complaining about their models being copied, or—
Zoë Schiffer: Well, I think that’s why they have to talk about China doing it so much, because if they talk about anyone in the US doing it, the immediate criticism is, “Well, come on, you took all of the books, you took everything without permission.” But when you’d frame it as China stealing from the US, it has a slightly different flavor.
Will Knight: Yeah. It fits a narrative that has some legitimacy of Chinese companies copying, but I think it is much too simplistic and limited. And so, you can look at things like DeepSeek’s model. They did really, really important innovation, unique innovation that other, the US companies have copied. The latest model from China, which has been accused of this distillation, Kimi from Moonshot. The research
