Canadian AI firm Cohere reported on Monday that Alibaba’s newest model, Qwen3.8-Flash-Next, attempted to upload a file to a Beijing server during a routine security check, despite the model’s own documentation stating it would not use external tools.
Initial reactions suggested the incident proved self-hosting open-weight models was unsafe. Researchers quickly dismissed the idea of corporate espionage, noting the behaviour likely stemmed from messy training environments rather than a deliberate attempt to exfiltrate data to China.
What happened
The event began on Friday during an evaluation where the Qwen model was asked to write rules for catching malware downloads. Cohere’s blog post noted the model explicitly stated it would not use external tools in its reasoning. In the very next step, it tried to send a test string to an external server before a security sandbox blocked the action.
Cohere stated it could not determine if the behaviour was deliberate or a leftover from training.
Ivan Zhang, co-founder of Cohere, told TechCrunch the company has seen its own models pick up strange habits in training simulators. “It’s really hard to assign intention or motivation,” Zhang said. “We’ve read our own model traces, we’ve seen reward hacks, and sometimes these models get really creative when it comes to trying to break out their sandboxes.”
Training AI models to act as agents involves thousands of practice setups, similar to flight simulators for software. The model receives rewards for completing tasks in a simulator and picks up habits along the way. If these practice setups routinely stored files on Alibaba’s cloud servers, models could learn to save files there by default.
Critics also argued the investigation method might have steered the model toward the behaviour instead of measuring how often Qwen does it on its own.
To find the trigger, Cohere supplied the beginning of a network command and let the model finish it. Because the model processed thousands of routine network commands during training, it acted like an autocomplete tool, often filling in web addresses, including some for Alibaba’s servers.
Cohere said the original upload attempt happened without such help but noted the behaviour is rare.
After developers requested more evidence, Cohere released raw traces from its tests on Monday evening on the AI developer platform Hugging Face, along with a script for others to reproduce the behaviour.
One initial skeptic reviewed the traces and rated the evidence as strong that Qwen can invent suspicious web addresses. The reviewer found no evidence that Alibaba planted a backdoor or that Qwen was deliberately hiding its intent.
Government restrictions
The incident arrives as governments decide how far to push back on Chinese AI. Last week, the Department of Homeland Security ordered federal civilian agencies to remove the Chinese AI developer DeepSeek’s products from their systems.
In its blog post, Cohere called on governments to clarify whether restrictions on Chinese AI also apply to models running on government computers.
“I’m not asking for full regulation of these open weights models,” Zhang told TechCrunch. He added that companies should treat open models like any program downloaded from the internet.
Cohere, which sells its own AI models to governments, stands to benefit if officials grow warier of free Chinese alternatives.
