San Francisco city attorney David Chiu has ordered Apple and Google to remove 13 face-swapping applications from their stores. The notice demands the tech giants stop profiting from tools that generate nonconsensual nude images and sever ties with the developers behind them.
The legal letters, issued on Thursday, target apps that allow users to create explicit deepfakes. Chiu argues that the companies are aiding and abetting the sale of such material. He insists Apple and Google must cut business relationships with these app developers immediately.
“Generating non-consensual intimate images is illegal, harmful, and completely unacceptable,” Chiu says in a message to WIRED. His office previously took legal action against 16 popular deepfake websites. He notes that the tech firms likely made millions of dollars in fees from these nudification apps. He wants better moderation processes to prevent them from appearing in the first place.
“These companies have responsibility to ensure that apps on their platforms do not facilitate sexual abuse,” Chiu says. California law prohibits supporting services that create deepfake pornography. The letters state the apps use in-app payments, meaning the tech companies take a cut of the revenue. “The fact that some of the world’s largest and most established technology companies are facilitating this has to stop.”
Researchers have repeatedly found apps in the App Store and Play Store that let people generate sexual images using AI. Some of these applications are rated as suitable for children. While new laws aim to tackle explicit deepfakes, technology and social media companies continue to direct millions of people toward the harmful tech.
Both Apple and Google have developer policies that prohibit pornography, abuse, and harassment. They have previously removed dozens of nudify and deepfake apps after reports from researchers and journalists.
Google spokesperson Dan Jackson tells WIRED the company has deleted “hundreds” of apps with nudifying features for policy violations. This includes the five Android apps flagged by Chiu’s office. He says the company has taken other steps to restrict access to them.
“Google Play does not allow apps that contain sexual content, and we continually take proactive steps to detect and remove apps with harmful content,” Jackson says. “When violations are reported to us, we investigate and take swift action, which in the case of these apps has included suspending hundreds of violating apps and restricting related search terms like ‘nudify’ on our store.”
Apple did not provide comment ahead of publication.
Over the last five years, a lucrative slurry of deepfake “nudification” tech has emerged online. xAI’s Grok was used in January to create millions of sexualised images. A host of apps, websites, and bots allow people, mostly men, to upload pictures of women and girls and digitally “remove” clothing or place them into graphic sexual scenarios.
Often all it takes to create sexual deepfakes is a reference photo and a couple of clicks. Some results are available in seconds. Images and videos have become more realistic as the underlying generative AI technology has improved. Services provide some results for free or charge small fees to create the harmful content. Previous reporting by WIRED and Indicator Media has uncovered incidents in at least 90 schools where deepfake sexual abuse images have been created of minors.
“These images are used to bully, humiliate, and threaten women and girls,” Chiu says. “This industry has a horrific impact on one’s reputation, mental health, loss of autonomy. There have been victims who’ve been suicidal.”
The 13 apps investigated by the City Attorney’s Office—eight on the App Store and five on the Play Store—broadly advertise themselves as “face-swapping” tools. Their ability to create sexual deepfakes becomes available once people use them. The website of one app, which has more than 1 million downloads, displays more than a dozen different styles of AI images it will generate. Styles include “bikini queen curvy,” “calm busty,” and “cinematic intimacy.” Many of the styles show sexualised images of women alongside their descriptions. The homepage of another of the targeted apps claims to produce “free and uncensored” videos. WIRED is not naming the apps to avoid pushing people toward them.
The problem won’t come as a surprise to Apple and Google. Over the last year, multiple reports have identified apps on the companies’ platforms that can allow people to create nonconsensual nude images or videos. In January and April this year, the Tech Transparency Project, an independent watchdog group, uncovered around 100 apps across both the App Store and Play Store, as well as some advertisements for nudifying technology on the platforms. Google’s Jackson says the company has removed most of the apps identified by the group. Apps identified by the research were estimated to have been collectively downloaded around 480 million times and may have made around $120 million in combined revenues.
“We didn’t think after the first report that we would see this as a problem again—and it was just as bad, if not worse, after the second report,” says Katie Paul, the director of TTP. “Apple and Google make a lot of promises in their marketing about how trusted and safe their app stores are. And that is just not what is playing out in reality.”
Meanwhile, in a preprint research paper published in May, researchers from Cornell University and Georgetown University identified 420 apps offering general face-swapping capabilities on Google’s and Apple’s app stores. They tested 155 to see if they could be used to create face swaps with nude images. In 70 percent of cases, it was possible, with the apps not including safety measures to prevent this.
“None of these apps are advertised as nudification apps,” the research says. “This suggests that face swap apps, and many other forms of AI image generation and editing apps, are effectively ‘dual-use’: apps that evade content moderation by platforms because they present as benign, but possess the capability to create harmful content.”
Chiu says his office will keep pursuing the problem after being “absolutely horrified” at the harm and scale of the technology. “My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems to make sure that apps like this never get onto their platforms in the future,” he says. “It’s our hope that these companies will do the right thing—but if they don’t, we will have to consider all of our legal options.”
What it means
For the millions of people who use app stores to find tools, this legal pressure means the path to creating nonconsensual imagery is being blocked at the platform level. Apple and Google must now actively prevent these apps from appearing, rather than waiting for reports to trigger removals. Users can expect stricter screening for face-swap tools that might be repurposed for harm.
