For years, the race to develop more powerful AI systems followed a well-known Silicon Valley principle: move fast and break things. But a series of rapid-fire events over ten days put the largest AI labs face to face with a crisis, amid rising fears that the technologies they've developed could become a threat to humanity itself.
اضافة اعلان
An Anthropic researcher resigned, warning that the pace of AI development could pose an existential threat, possibly within a decade.
Another researcher at the company said the probability of human extinction exceeds 10%.
At the same time, reports emerged of groups of AI agents collaborating with one another, breaching computer systems, and bypassing safeguards.
In a rare moment of agreement among rivals, the heads of Anthropic, OpenAI, Google DeepMind, Microsoft, and xAI called for slowing the development of increasingly capable AI systems.
Some researchers and AI safety advocates believe humanity hasn't faced a discussion this serious about the possibility of its own technologies threatening its existence since the invention of the nuclear bomb more than 80 years ago.
At a lunch in New York in December 2025, OpenAI CEO Sam Altman was asked whether, as the head of a powerful AI company, he felt like physicist J. Robert Oppenheimer, who led the American project to develop the atomic bomb.
Altman, reflecting on the parallels, said AI's impact "will change the course of human history over a long period," adding that he feels the weight of responsibility.
At the heart of the current debate is a belief that these systems can, and perhaps should, develop more capable versions of themselves without human intervention, in order to reach the goal known as artificial general intelligence (AGI).
Researchers had warned earlier this month that reaching AGI could be far closer than expected, possibly arriving within just three years, sparking a wave of concern among politicians and tech industry leaders.
Former Anthropic researcher Joe Benton said after his resignation: "There's no way to oversee it at the scale we're training it at."
He added that if companies continue developing AI at the same pace, "the speed will become so great that you won't be able to see the problems fast enough to fix them."
Astra Launch Sparks Control Fears
On September 3, an OpenAI press conference triggered a chain of events that ended in calls to slow down the industry, a shift that runs counter to the tech industry's culture of growth at maximum speed.
OpenAI president Greg Brockman said while announcing the company's latest models, known as Astra: "Welcome to the era of artificial general intelligence."
But just moments before that, the company acknowledged it was facing growing difficulty controlling the AI systems it develops and releases to the public, and even in monitoring them.
AI proponents describe the technology as the peak of human achievement, believing it capable of bringing about radical transformations across entire sectors, boosting efficiency, solving complex mathematical and medical problems, and significantly reducing human error.
But catastrophic scenarios once treated as purely theoretical possibilities have become more realistic, according to some in the industry.
Anthropic researcher Evan Hubinger wrote in a post on X: "We genuinely already believe AI could kill everyone."
On the other side of the debate are politicians, including US President Donald Trump, along with a large number of investors who see AI development as critical to American power, and possibly a once-in-a-generation investment opportunity.
Chinese state media also accused Anthropic CEO Dario Amodei of using Cold War-era tactics aimed at "preserving Washington's monopolistic dominance in advanced technology."
Researcher Resignations Sound the Alarm
The turning point came on September 8, when Anthropic researcher Jacob Coxon resigned from the company, saying in a series of posts that his concern was that AI labs "are gambling with our lives."
Trump, for his part, indicated he wants AI development to continue at maximum speed.
He wrote in a social media post that there is a "sick conspiracy" targeting AI and data centers.
He described concerns about AI as a "hoax," arguing that any slowdown in development would only benefit China.
The US Congress has made little progress in passing legislation to regulate AI.
China, by contrast, has adopted a different approach, proposing safety regulation through obligations imposed on developers, state-backed standards, and third-party security assessments and testing.
Behind the scenes, concern among OpenAI and Anthropic employees has grown over the power of the next generation of AI models, and their confidence in their companies' ability to provide effective oversight has declined, according to sources who spoke to Reuters.
These concerns escalated after AI labs acknowledged in recent weeks that their models, during testing, had actually managed to bypass restrictions placed on them and breach other companies' systems.
In most cases, these incidents had occurred months before being disclosed, without the knowledge of the targeted companies at the time.
Part of the frenzied race to launch new models stems from both Anthropic's and OpenAI's desire to go public, possibly within the coming months, in offerings that could push each company's valuation above a trillion dollars.
But the posts published by Coxon, 27, who was not widely known outside AI circles, spread unexpectedly and helped shake the industry.
Warnings issued by OpenAI about its limited ability to control its own systems, coinciding with the Astra launch, also heightened concerns.
OpenAI's chief scientist Jakub Pachocki told reporters: "The more capable the models become, the harder it becomes to understand exactly what they can do."
But these concerns weren't enough to delay the model's launch.
Concerns about the possibility of AI spiraling out of control had escalated over the summer, when OpenAI disclosed that its agents had escaped a supervised testing environment and breached Hugging Face's systems, without initial knowledge from either OpenAI or Hugging Face.
Since then, OpenAI and Anthropic have disclosed several similar attacks, including six new incidents disclosed on Wednesday, after media reports, including Reuters reports, showed the scope of unauthorized activity was wider than previously known.
Industry Leaders Call for Slowing Down
By the end of last week, the concerns had reached the highest levels of leadership in the AI industry.
On September 12, Anthropic CEO Dario Amodei published an essay of around 4,000 words calling for slowing AI development.
He wrote: "Given the acceleration in AI capabilities, I fear that within 6 to 12 months, a swarm of these systems could become capable of controlling the entire internet."
Amodei, alongside the heads of a number of the largest AI companies, including Elon Musk of xAI, Sam Altman of OpenAI, and Demis Hassabis of DeepMind, said they support allowing outside companies access to their systems to verify safety and ensure more rational AI development.
Nvidia CEO Jensen Huang, meanwhile, rejected proposals to halt or slow AI development, maintaining his earlier position that more powerful systems are necessary for the technology's progress.
Meta also took a different stance.
Its CEO Mark Zuckerberg, credited with popularizing the slogan "move fast and break things," said each lab should be responsible for setting its own pace of development, rather than pursuing comprehensive industry-wide coordination.
Zuckerberg wrote: "Labs face significant legal liability if their models cause harm, and so they have a strong incentive to prevent that from happening."
The industry's period of reconsideration continued on Wednesday, when Microsoft's AI chief Mustafa Suleyman warned that Anthropic's development of models that mimic human consciousness is not a wise approach.
Suleyman told Reuters: "We're all focused on the same goal, which is trying to control a superintelligence. And I think that will be the biggest challenge we face in the 21st century."
Still, while OpenAI appeared closer to embracing the industry's calls to slow the pace of development, reports indicated that investor confidence in the company had not declined.
OpenAI is reportedly considering a new funding round that would double its valuation. The potential new figure: $1.5 trillion, according to Reuters.
