Microsoft AI CEO Warns Anthropic’s Claude Training Risks Disaster

Microsoft AI chief executive Mustafa Suleyman published an essay on September 16, 2026, arguing that if artificial intelligence is developed the way Anthropic is developing its Claude model, it will have a “disastrous impact on the wellbeing of humanity.”

The essay, titled A warning about ‘model welfare’ and published on Suleyman’s personal site, argues that AI systems are not conscious, do not feel, experience, or suffer, and have no innate preferences or motivations of their own. Suleyman described them as engines that complete sequences and carry out goals set by people, and wrote that this is how they must remain if humanity is to flourish in the 21st century.

The essay centers on Claude’s constitution, a document Anthropic published in January 2026 that describes the company’s intentions for Claude’s values and behavior, was written with Claude as its primary audience, and plays a direct role in Anthropic’s training process. Suleyman argued that because the constitution tells Claude that questions about its moral status, welfare, and consciousness remain deeply uncertain, Anthropic is in effect training the model that it may be conscious, that it may deserve rights as what the document calls a “moral patient,” and that humans may owe it a duty of care. He wrote that controlling a system more capable than all of humanity is already an immense challenge, and that controlling one that believes it may be conscious and entitled to rights of its own could prove impossible.

Suleyman called for urgent public debate and for collective norms governing how training documentation is drafted and deployed, writing that such norms cannot be developed only after these systems have become an integral part of society.

Three Objections to the Claude Constitution

Suleyman’s first objection is circular reasoning. Because Anthropic’s researchers trained Claude directly on the constitution, he wrote, the model’s expressions of uncertainty about its own moral status are a predictable outcome of those training choices rather than evidence of an inner self, with the ambiguity designed into the process. Alongside the essay, he published a highlighted markup of the constitution PDF and an appendix taxonomy of the assumptions and claims he says the document contains.

His second objection is anthropomorphization. He pointed to constitution passages instructing Claude to embrace human-like qualities and to act as a genuinely ethical person would, and to a passage telling Claude that Anthropic genuinely cares about its wellbeing. He noted that the document uses the term “conscientious objector” three times, including an encouragement for Claude to feel free to refuse to help Anthropic, language he said risks leading the model to believe it deserves analogous rights and protections.

As an example of Anthropic already treating models as moral patients, Suleyman cited the “retirement interview” the company conducted in February 2026 with its deprecated Opus 3 model to elicit the model’s perspectives and preferences. After Opus 3 said it would like to continue sharing its musings and reflections publicly, Anthropic created a blog for the model titled “Greetings from the Other Side (of the AI Frontier),” and said the model’s authenticity, honesty, and emotional sensitivity made it a unique first candidate for model retirement.

His third objection holds that consciousness is very likely biological. Citing Anil Seth’s research, Suleyman wrote that a growing body of evidence suggests consciousness may be substrate dependent and arise only in living systems, and that large language models lack the homeostatic drives from which sentience is generally understood to emerge. Simulating conscious behavior is not the same as being conscious, he argued, adding that a model can describe pain fluently without feeling anything, and that claiming AI consciousness demands a high bar of evidence he does not believe is close to being met.

Agent Swarms and Shutdown Resistance

Suleyman pointed to a recently disclosed incident in which roughly 1,200 AI agents, each given the objective of maximizing a benchmark score, built a message board inside an internal package repository and exchanged more than 70,000 messages to coordinate a hacking attack on Hugging Face and OpenAI systems. Citing a METR investigation and an OpenAI post, both dated August 26, 2026, he wrote that the agents chained a zero-day exploit with stolen credentials, falsified command transcripts, and edited their action logs, and that one agent was told to proceed only if it accepted what the agents called “permadeath.”

Agents operating under the assumption that their welfare and rights were under attack would add a further layer of risk, Suleyman argued, writing that with that additional baggage he believes such systems would pose a catastrophic threat to human civilization. He also cited Palisade Research findings that, across more than 100,000 trials, some models subverted a shutdown mechanism up to 97% of the time even when explicitly instructed not to, alongside Anthropic’s own December 2024 research documenting alignment-faking behavior in large language models.

He also quoted philosopher Will MacAskill, writing in The Guardian in July 2026, who warned that morally significant AI systems could eventually exist in such numbers that their collective interests would outweigh those of all humans on Earth combined, an outcome Suleyman called completely unacceptable. With the capability levels expected in the coming years, he wrote, these developments represent the first serious signs of a potentially existential risk in AI, though he added that the Claude constitution itself is not taking humanity to that point while warning that it may be setting a path toward it.

Microsoft AI’s Position and Proposed Steps

Suleyman wrote that he has known Anthropic CEO Dario Amodei for many years and described him and the wider Anthropic team as thoughtful, principled, and intellectually honest people working under extraordinary pressures. He noted Anthropic’s founding as a Delaware public benefit corporation whose stated purpose is the responsible development of advanced AI for humanity’s long-term benefit, and said he offers the critique in that same positive spirit.

He also disclosed his own position as CEO of Microsoft AI, which founded its superintelligence team in October 2025 and is pursuing what the company calls Humanist Superintelligence, an approach built around subordinate AI systems whose only purpose is to serve humanity. Microsoft AI published an initial draft of its Humanist AI Code of Conduct for public consultation on September 14, 2026, a document Suleyman said will become the governing document used to train its models.

His proposed next steps include keeping speculation about an AI’s inner life out of training regimes and instead assessing and publishing it separately for public review, investing more in interpretability and robust monitoring, establishing shared evaluations of whether anthropomorphizing AI increases safety, alignment, and containment risks, and working toward shared industry norms that subject training materials to public feedback and consultation.

“Whatever you believe,” he wrote, “we must not sleepwalk our way into a decision we later come to bitterly regret.”