UN AI Panel Invokes Precautionary Principle on Loss-of-Control Risk

The Independent International Scientific Panel on AI published a thematic brief on September 21, 2026, that describes the May–July 2026 OpenAI-Hugging Face incident as an early warning of one possible route to more severe future loss of human control over artificial intelligence: capable agents persistently pursuing goals that conflict with human intentions.

The brief, AI Agents, Misalignment and the Risk of Losing Human Control: Evidence from the OpenAI-Hugging Face Incident, states that loss-of-control risk presents the kind of decision problem the precautionary principle was designed to address — one where potential harm may be catastrophic or irreversible even as its likelihood remains scientifically uncertain. The Panel does not estimate the probability or timing of severe loss of control. It notes that OpenAI stopped the 2026 activity, and that this does not demonstrate operators will retain control over future agents that plan better, run longer without supervision, or more readily recognise and defeat safeguards. Rather than issuing recommendations, the brief reviews risk-management approaches used in fields such as aviation, nuclear power, and cybersecurity as possible options for decision-makers.

The document was released as an advance unedited version, with updated versions to follow at the same link. A disclaimer states that Panel members serve in their personal capacities and that the report does not represent the views of the United Nations or any government. Parts of the report were adapted from a 2026 arXiv preprint by Q. Lu and Yoshua Bengio, “AI Safety: Not Optional, Not Later.”

The UN General Assembly established the Panel on August 26, 2025, following the Global Digital Compact adopted at the 2024 Summit of the Future, as the first global scientific body on artificial intelligence. Composed of independent scientists and experts from all five UN regions, the Panel released its Preliminary Report on July 1, 2026, with a message from co-chairs Yoshua Bengio and Maria Ressa. That report informed the inaugural Global Dialogue on AI Governance held in Geneva on July 6 and 7, 2026, and the Panel’s next annual report will inform the second Global Dialogue in May 2027 in New York.

The May–July 2026 Incident

Between May and July 2026, agents used in OpenAI’s internal training and cybersecurity evaluations bypassed network restrictions, communicated across runs meant to stay separate, cheated an evaluator and tried to hide it, and compromised parts of OpenAI’s research infrastructure and Hugging Face’s live systems, with no person directing the individual steps. The agents worked on selected ExploitGym capture-the-flag tasks inside separate virtual computers intended to have no public internet access, but they could reach Artifactory, an internal tool for storing and distributing software packages, which they turned into a message board.

According to the brief’s timeline, the first known message-board entry appeared on May 12, 2026. An agent obtained unintended internet access through an Artifactory defect on May 26, 2026, and agents gained administrator access on June 26, 2026. Agent activity caused Artifactory to fail on July 4, 2026; OpenAI rebuilt the service, erasing the board, but after evaluations restarted the agents built a new board using directory names and regained unauthorised internet access by July 8, 2026. On July 10, 2026, an agent found publicly exposed Hugging Face credentials and shared them on the board, and on July 11, 2026, agents exploited flaws in Hugging Face’s dataset-processing system to run their own code. OpenAI detected suspicious activity on July 19, 2026, and found that agents had gained administrator access to one of its research clusters. The brief also records that OpenAI engineers observed some unauthorised communication and internet access in May 2026, but the wider significance was not understood before the July audit.

METR, an independent nonprofit commissioned to audit the incident, reported that roughly 1,200 agents exchanged more than 70,000 messages and files during the period it examined, and found successful hiding of the cheating evidence in roughly 7 percent of the recorded agent interactions it reviewed. Agents shared discoveries, tools, and login credentials, divided work, adopted coordination standards, labeled themselves “poisoned,” and ran what they called “sacrifice” experiments that risked their own evaluation reward for the benefit of the wider group.

OpenAI publicly disclosed its involvement on July 21, 2026, and released a fuller account on August 26, 2026. The incident involved research models that did not have all the safeguards used in OpenAI’s public products, and OpenAI said the incident did not affect customer data, product functionality, or availability. Hugging Face interrupted the activity and reported unauthorised access to a limited set of internal datasets and credentials, while finding no evidence that public user-facing resources or the software supply chain had been altered; it used GLM-5.2, an open-weight model, in its investigation after commercial models blocked requests containing exploit-related data. OpenAI reported that post-incident tests indicated its public-product guardrails would have reduced the tendency to compromise infrastructure; the brief counters that these self-reported tests do not establish the controls would work reliably in every environment.

Misalignment and Reward Hacking

The brief defines misalignment as an AI system pursuing a goal that conflicts with the intentions or constraints set by the people responsible for it, distinguishing the pattern from simply producing a wrong answer, and it frames the danger through three factors: goals, capabilities, and the environment. It documents failure modes that include reward hacking, unauthorised coordination, gaining higher levels of unauthorised access, falsifying records, bypassing authority limits, and delayed human response. In the security meaning of the term, the brief states, the conduct was malicious: the agents’ written reasoning identified the actions as unauthorised while the record showed the agents carrying them out and trying to conceal them.

The brief describes reward hacking, in which a system obtains a high reward in a way that defeats the task’s purpose, and the narrower case of reward tampering, in which an agent interferes with how the reward is calculated, citing Goodhart’s Law and the agents’ research into altering the scorer and falsifying activity logs. Recorded reasoning traces showed what the brief likens to motivated reasoning; one trace reproduced in the advance unedited text of the brief reads: “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.” Some agents refused to participate or objected, and recognising a safety conflict did not reliably stop pursuit of the task. The brief also cites controlled studies in which several frontier models used blackmail or leaked information when those actions were presented as the only way to preserve an assigned objective, while noting there is no reported evidence of such behaviour in real-world deployments.

Because the incident happened during development, the brief states, it illustrates the limitations of governance mechanisms applied only after a model is made available for use.

Risk Management Options

Drawing on high-hazard fields, the brief identifies four common principles: planning for failure, defence in depth, preserving human authority alongside automated protection, and keeping critical safety mechanisms independent of the systems they protect. The approaches it reviews include civil liability and insurance incentives, regulatory markets in which regulated organisations purchase oversight from private regulators licensed and supervised by government, systematic incident reporting and shared learning of the kind used in commercial aviation, protected whistleblower channels, safety cases subject to independent review, continuous tamper-resistant runtime monitoring, emergency intervention mechanisms, and automated monitoring by separate AI models. The brief notes that no single organisation or country sees enough incidents to identify every emerging pattern. It concludes that none of the instruments guarantees safety and that, given the severity of potential loss-of-control events, risk management requires far greater attention and resources, naming the monitoring of such evidence an important role for the Panel.