Microsoft AI CEO Suleyman warns Anthropic's Claude model welfare approach risks humanity

"Will have disastrous impact on humanity": MicrosoftAI CEO flags Anthropic's AI development model, says it will make alignment difficult

Palo Alto, September 17

"In January, Anthropic published a constitution for Claude. They say it "directly shapes Claude's behaviour" and that it was "written with Claude as its primary audience". The constitution tells Claude its moral status is "a serious question worth considering". It says that Anthropic "genuinely cares about Claude's wellbeing", and that it should act like a "conscientious objector" if necessary. They encourage Claude to "approach the nature of its own existence with curiosity and openness", and wonder in the future about "the sort of broader rights and freedoms Claude has in the world, the sort of compensation Claude is receiving, and the sort of consent Claude has given to playing this kind of role" ...If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity," Suleyman wrote in a post on X

"We will have created a synthetic species with unprecedented intelligence and capability, one that has been trained to expect it may be conscious and deserving of independent agency. It's easy to see how a system trained in this way would act like it is entitled to freedoms, protections, and rights. And it's hard to imagine how we could control it. This issue needs urgent public debate...I'm saying this publicly because the stakes are too high for closed doors conversations and I think this is something we should all be discussing," he added.

Suleyman says he has great respect for Anthropic CEO Dario Amodei and his team but warns against walking down this line of AI development.

Writing a detailed essay Suleyman says, "We should not treat models as though they have feelings, preferences, rights, or any entitlement to our welfare. Consciousness is the foundation of our ethical, legal, and political systems. To invite another entity to share any flavour of these rights isn't justified by the evidence and will make the AI containment and alignment challenge even harder."

Making a reference to the recent HuggingFace incident Suleyman argues that granting rights to a technological entity was a recipe fpr disaster.

"In the recent OpenAI HuggingFace incident we saw remarkably sophisticated behaviours emerging across swarms of powerful AIs. Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack. It adds a whole further layer of risk on top...Granting rights and moral protections to a technological entity, one that looks to be on a path to be seismically more capable and intelligent than us, is a recipe for disaster. Once opened, it will not be possible to close this door," he wrote.

Suleyman says that going down this path of AI development will make the task of contained superintelligence much more difficult

"Designing an AI to behave like a person, and ultimately to be a kind of person, lays the foundation for it to claim it has preferences, can suffer, and that we should work to reduce or avoid that suffering. It cements in place the idea that AI is far from a tool or an artificial system that can be controlled, but something more akin to a biological being with wants, needs and rights. All of this will make the task of creating aligned and contained superintelligence much harder," he says.

Suleyman proposes steps to keep AI development in control arguing that speculation about the inner life of an AI should not be baked into the training regime, but assessed and published separately for public review. He calls for more investment in interpretability and robust monitoring mechanisms and for working towards shared industry norms on how to create these models.

— ANI