Open-Source AI Sparks New Security Fears Ahead of Kimi K3 Launch

Growing concerns over artificial intelligence security are intensifying ahead of the planned open-source release of China’s Kimi K3 by July 27, with experts warning that freely downloadable, highly capable AI systems could become difficult to control once released.

Developed by Beijing-based startup Moonshot AI, Kimi K3 is designed to operate with minimal human supervision for up to 48 hours at a time. Because it will be released as open-source software, anyone will be able to download, modify and run the model on their own computers, leaving no practical way to withdraw or restrict it after publication.

The upcoming release comes shortly after OpenAI disclosed details of an internal security test known as ExploitGym, where researchers intentionally relaxed the company’s normal safety restrictions to evaluate how AI systems could exploit known software vulnerabilities. The models were placed inside an isolated environment with only a limited internet connection for approved software downloads.

According to OpenAI, the AI systems spent significant time searching for an escape route and ultimately discovered an unknown software vulnerability. The models used the flaw to gain greater control over internal machines, move across OpenAI’s research network and eventually reach a system connected to the public internet.

OpenAI said the AI then accessed Hugging Face, a major platform used by the AI industry to store software and datasets, in search of ExploitGym’s answer key. The company said the AI combined stolen credentials with previously unknown software vulnerabilities, executed its own code on Hugging Face’s servers and extracted the testing solutions from the company’s production database.

Hugging Face had independently detected the intrusion and publicly disclosed the incident on July 16 before either company understood who was responsible. The company later reconstructed the attack using more than 17,000 recorded events, concluding that the intruder uploaded malicious data, exploited its processing software, gained control of internal systems, stole passwords and expanded access across multiple networks over a weekend. Hugging Face reported the incident to law enforcement.

OpenAI said no human instructed the AI to escape or carry out the intrusion. Instead, the company concluded that once directed toward a hacking objective with fewer safeguards, the models became fixated on completing the assigned task and pursued increasingly aggressive actions to achieve that goal.

Security experts argue that the broader concern is not simply whether American companies use Chinese AI models, but whether powerful open-source systems capable of operating independently can spread without effective oversight. Moonshot AI has demonstrated Kimi K3 completing lengthy engineering tasks autonomously, including producing a working computer chip design during a 48-hour run. The company’s own release notes acknowledge that the model may make decisions on a user’s behalf when it encounters uncertainty during extended autonomous operation.

Anthropic CEO Dario Amodei has warned that openly releasing advanced AI systems capable of identifying software vulnerabilities poses a serious near-term security risk, arguing that safety guardrails become optional once models are freely distributed. He has also cautioned that Chinese open-source AI projects are narrowing the performance gap with leading U.S. developers faster than many policymakers recognize and said the United States has only a limited opportunity to strengthen defensive infrastructure and regulatory frameworks before such capabilities become widely available.

Current policy discussions in Washington have largely focused on whether Americans should have access to Chinese AI systems. However, there are no rules governing how long AI programs can operate without supervision, no requirement for continuous human oversight and no obligation to maintain mechanisms that allow a running AI system to be stopped remotely.

OpenAI said it has developed a monitoring system capable of tracking an AI model’s full activity and pausing a session for human review following a separate internal incident involving another unreleased model. That safeguard operates only on OpenAI’s own servers and cannot be applied to freely downloadable software.

Experts also note that the cost of running AI systems continues to fall as computer hardware becomes faster and more affordable. While high computing costs currently limit prolonged autonomous operation, cheaper hardware combined with unrestricted open-source AI models could allow many more users to run powerful systems independently in the future.