Anthropic issues big AI warning, calls for slowing down development of models
San Francisco, June 5
,: AI startup Anthropic has issued a warning that the fast-paced evolution of the Artificial Intelligence technology is reaching a point where it could design and develop its own successor.
The startup, in a report, said that the company is "delegating a growing share of AI development to AI systems themselves, which is speeding up our work".
"Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor," Anthropic said.
Calling it recursive self-improvement, the San Francisco-headquartered AI firm said that the world is not there yet, but it could come sooner than most institutions are prepared for.
The Claude-maker recently raised $65 billion in a funding round valuing the company at close to a trillion dollars. It has confidentially filed for an IPO amid a boom in AI shares as investors continue to pour money in AI stocks, remaining bullish on the prospects.
Just to show the scale of how fast AI is developing itself, the company said that Anthropic engineers, on an average, are shipping 8x as much code per quarter as they did from 2021-2025.
The company said that while a self-improving AI is good for the overall development, a full recursive self-improvement risks humans losing control of AI systems.
"If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behaviour all grow much more important," Anthropic warned.
The extent to which the AI models can finish a task on their own is doubling roughly every four months.
"In 2027, AI systems could be capable of tasks that take a person weeks," the company said.
Anthropic urged for slowing down of the AI development process to give more time to come up with technology to deal with its immense implications.
"A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions. It would also require that each can verify that the others have actually stopped."
— ANI
Reader Comments
Honestly, this is a classic case of profit over prudence. Anthropic raised $65 billion and is now suddenly worried about losing control? Maybe they should have thought about this before building these systems. The call to 'slow down' sounds hollow when you're filing for an IPO. Indian governance should take note and start drafting AI safety laws.
The digital world is moving too fast! In our country, we are still struggling with basic AI literacy in schools and colleges. How can we be prepared for systems that can improve themselves every few months? The government's 'AI for All' initiative is good, but we need much more focus on AI safety research. A good reminder to not blindly adopt Western tech.
From a tech perspective, this is fascinating. The idea of recursive self-improvement is like Skynet from Terminator, but it's becoming real. I think India should invest heavily in open-source AI safety research. We have great IITs and talent; why not lead in this area instead of just consuming AI products? The warning is real, and we must act.
I'm a researcher in AI ethics, and this warning is exactly what we have been saying for years. The 'delegating to AI' trend is dangerous because it creates a feedback loop where we lose understanding of how decisions are made. The doubling every four months is terrifying. We need a UN-level treaty, and India must be a signatory. Otherwise, we will be just users of whatever AI emerges.
Very hypocritical of Anthropic. They are the ones speeding up AI development in the
We welcome thoughtful discussions from our readers. Please keep comments respectful and on-topic.