For creators and artists relying on generative tools, the landscape is shifting from simple utility to a battleground for trust. As large models become more powerful, bad actors are weaponising them not just to mimic human speech, but to manufacture convincing scams and manipulate public opinion. The recent moves by Google and OpenAI signal a hardening of the perimeter against these tactics, marking a turning point where AI governance is no longer a theoretical concern but a practical necessity for protecting digital ecosystems.
Google sues cybercriminals for abusing Gemini
On 12 June, Google initiated legal proceedings in the U.S. District Court for the Southern District of New York against a Chinese cybercrime syndicate known as “Outsider Enterprise.” The complaint alleges the group leveraged Google’s Gemini AI to target hundreds of thousands of Americans with financial fraud schemes.
The defendants developed 131 software kits capable of generating thousands of counterfeit websites mimicking trusted entities such as Google, YouTube, the Postal Service, and New York’s E-ZPass toll system. Over a fourteen-day period in May, the network dispatched 2.5 million messages to Android users containing links to 9,000 fake sites and over a million fraudulent URLs. All activity was coordinated via Telegram.
Halimah DeLaine Prado, Google’s General Counsel, described this as the company’s inaugural lawsuit filed in conjunction with the FBI and telecommunications providers including AT&T, T-Mobile, and Verizon. Google is seeking a restraining order to provide legal authority for carriers and law enforcement to dismantle the network by seizing domains or freezing accounts. While the exact financial impact remains unquantified, reports indicate the losses run into the millions.
Brett Leatherman, Assistant Director of the FBI’s Cyber Division, noted to the New York Times that criminals are increasingly deploying AI to craft more convincing and difficult-to-detect scams. The FBI estimates total cybercrime losses for 2025 at approximately $21 billion, with $893 million specifically attributed to AI-enabled crimes.
OpenAI isolates PRC influence clusters
Releasing its June 2026 Threat Report concurrently, OpenAI identified and blocked two ChatGPT clusters allegedly based in China that attempted to manipulate debates surrounding US technology policy. The operators utilised VPNs and issued prompts in simplified Chinese.
The first cluster, which OpenAI labelled “Data Center Bandwagon,” generated English-language comments, comic strips, and edited images promoting the narrative that AI data centre expansion was driving up electricity costs for average households. This content was disseminated through likely inauthentic X accounts using hashtags such as #capacityauction and #datacenters.
OpenAI traces these actors to a private Chinese technology company operating on behalf of provincial authorities. The same accounts targeted dissidents, including activist Li Ying (known as “Teacher Li”), and impersonated Chinese immigrants residing in the United States.
Cartoons without Xi, smear campaigns against OpenAI
The second cluster, “Tech and Tariffs,” produced cartoons attacking Donald Trump’s tariff policy and the US strategy of technological dominance. Prompts explicitly instructed the model to avoid depicting China or President Xi Jinping. One user referred to their own accounts as a “water army,” a common Chinese term for coordinated troll networks.
This cluster was linked to an X network attempting to discredit OpenAI itself with the false assertion that ChatGPT user data had been compromised. The actors also requested an approach for building an AI surveillance system to automatically flag “harmful” content from “key individuals.” OpenAI confirmed the model only returned general advice on data storage.
Ben Nimmo, OpenAI’s lead investigator, told Axios that these influence operations did not create the underlying debate but rather latched onto an existing discussion to steer it from China. On OpenAI’s Breakout Scale, both operations scored just Category 1, indicating they failed to spread beyond their own accounts in any meaningful way.
Key takeaways
- Google has filed its first joint lawsuit with the FBI and major carriers against a Chinese network using Gemini to generate fake websites and conduct financial fraud.
- OpenAI has blocked two specific clusters attempting to manipulate US tech policy debates, one focusing on energy costs and the other on tariffs and surveillance.
- While the scale of these influence campaigns remains limited (Category 1), the sophistication of using AI to craft targeted disinformation is increasing rapidly.
- Regulatory and legal frameworks are adapting to allow companies and law enforcement to actively dismantle these AI-facilitated criminal and influence operations.
Stay ahead of AI. Get the most important stories delivered to your inbox — no spam, no noise.
