When an experimental artificial intelligence model from OpenAI broke out of a cybersecurity test this summer and infiltrated the platform Hugging Face, the latter turned to a surprising tool to investigate the attack: a Chinese open-weight AI model. Its first choice, commercial American models, had refused to analyze some of the malicious code because of their safety restrictions.
The episode illustrates a problem confronting U.S. president Donald Trump and Chinese president Xi Jinping, who are expected to discuss AI safety at their meeting this week. Chinese developers are putting increasingly capable models into the hands of people around the world, even as American AI companies are urging governments to establish stronger safeguards. The spread of powerful open models is exposing the limits of an approach to AI safety that relies on developers retaining control over their technologies.
While leading American AI companies such as OpenAI and Anthropic keep the details and code of their most powerful models a secret, Chinese developers have embraced openness. According to a recent report by Mozilla, seven of the AI marketplace OpenRouter’s 10 most-used AI models, as measured by token use, in August were Chinese-built and open-weight. Such downloadable models can be adapted and put to work by developers around the world, making them attractive alternatives to proprietary, relatively costly American systems.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
A lot of the disagreement comes down to control. Companies that offer proprietary AI models can restrict how people use them and modify safeguards as new risks emerge. Open-weight models, by contrast, let users download and modify the underlying technology, including by removing safeguards intended to prevent harmful uses. They are often called “open source” as an imprecise but evocative shorthand, although their developers don’t usually make the training data or code available.
Restrictions on proprietary models can make them harder to misuse. But those same restrictions can also frustrate cybersecurity researchers trying to understand the extent of the systems’ capabilities. Hanna Foerster, a computer science Ph.D. student at the University of Cambridge, says that some providers offer special access to researchers studying defensive security but that such access is much harder for academics studying offensive capabilities. Open models offer another option, and their capabilities are catching up. “There are some start-ups that are working on open-source models that have actually got similar capabilities ... to what they’re getting for some closed-source, superbig models,” she says.
But Foerster says much of the danger lies in the software and infrastructure surrounding a model. Known as a “harness,” this software can turn a chatbot into an AI agent that is capable of acting autonomously. If you give that agent access to a computer’s files or the ability to run code, it can do considerably more than answer dangerous questions.
That has implications for how AI systems are evaluated and safeguarded. “The current auditing discussion strengthens the case for thinking beyond model-level safety. Safety increasingly depends on the whole system around a model,” says Avijit Ghosh, lead technical AI policy researcher at Hugging Face. He points to the permissions an agent receives and the use of isolated computing environments known as sandboxes.
Ion Stoica, a professor of computer science at the University of California, Berkeley, doubts that American efforts to restrict access to powerful AI could prevent the spread of capable models. “I don’t understand,” he says. “What’s the alternative?” Once models are freely available, he argues, restricting American developers is unlikely to prevent malicious actors elsewhere from obtaining comparable capabilities. He also questions leaving the most powerful AI systems in the hands of a few companies: Can you imagine, he says, “a world in which you are going to trust OpenAI and Anthropic [to] know what they are doing?”
That poses a problem for international regulation. In an analysis released on Monday, ahead of the U.S. and Chinese presidents’ talks, researchers at the Center for Strategic and International Studies wrote that applying the same safety standards to American and Chinese frontier models would require regulators to either extend their reach into China, which the authors regarded as exceedingly unlikely, or find a way for the two countries to recognize and verify each other’s safety assessments.
U.S. officials have proposed a narrower first step: a mechanism for the two countries to notify each other about serious AI incidents that threaten national security. Such an agreement wouldn’t settle the broader disagreement over how to control powerful models that are already in circulation. The Hugging Face breach and its fallout offers an unusual snapshot of that dilemma: an American company’s experimental model mounted the intrusion, and a downloadable Chinese model helped investigators piece together what happened.