SAN FRANCISCO, Sept 19 (Reuters) – For years, the race to build ever-more powerful artificial intelligence followed a well-worn Silicon Valley principle: move fast and break things.
But in a series of cascading events over a 10-day stretch, the largest AI labs found themselves reeling as their own creations threatened to break humanity itself.
The Misinformation Monitor newsletter rounds up international misinformation narratives, with a “Real or Fake?” quiz in every edition. Sign up here.
At a New York luncheon in December 2025, OpenAI CEO Sam Altman was asked if, as leader of a powerful AI company, he felt like physicist J. Robert Oppenheimer, who led US development of the atomic bomb. Reflecting on the parallels, he said AI’s impact “is going to transform the trajectory of human history over a long period of time.” But, he said, he felt the weight of responsibility.
At the center of the debate under way is the conviction that these machines can, and even must, build more capable versions of themselves, with no human intervention, to reach a lofty goal known as artificial general intelligence, or AGI. Researchers earlier this month warned that AGI was far more imminent than previously believed and could come in as few as three years, launching a wave of concern from politicians and tech leaders about AI systems.
“There is no way to oversee them at the scale at which we’re training them,” said Anthropic researcher Joe Benton in an interview after recently quitting. If companies continue their relentless AI development, he said, “then the pace will be too fast and you can’t see the problems fast enough to fix them.”
ASTRA LAUNCH SPARKS CONTROL CONCERNS
“Welcome to the AGI era,” OpenAI President Greg Brockman said while announcing the company’s newest model, known as Astra. Yet, only moments before, the company admitted it was increasingly unable to control or even monitor the AI systems it was developing and releasing to the public.
AI has been hailed by its proponents as the pinnacle of human achievement. Proponents depict it as software that can upend entire industries, boost efficiency, solve confounding mathematical and medical problems and all but eliminate human error. But what was once a theoretical byproduct, a doomsday scenario, is suddenly tangible, say industry insiders.
RESEARCHER RESIGNATIONS TRIGGER ALARM
Behind the scenes, OpenAI and Anthropic employees have grown increasingly uneasy about the power of the next generation of AI models and less confident about their companies’ ability to provide meaningful oversight, the people have told Reuters. Those concerns only mounted as the AI labs acknowledged in recent weeks that their models, in testing, had effectively broken free of their shackles and hacked into other companies’ systems, in most cases months earlier and without the firms’ knowledge.
The arms race to release new models at a breakneck pace is driven in part by Anthropic’s and OpenAI’s desire to go public as soon as in the coming months, in IPOs that could value them well above $1 trillion.
But it took the unexpectedly viral posts from 27-year-old Coxon, little known outside AI circles, to upend the industry.
OpenAI’s warnings about its lack of control as it released Astra, its latest and most capable model, had stoked further concerns. “As models get more capable, understanding exactly what they can do gets harder,” OpenAI Chief Scientist Jakub Pachocki told reporters. But those concerns were not enough to delay its immediate release.
INDUSTRY LEADERS CALL FOR SLOWDOWN
By the end of last week, the alarm had reached the industry’s highest ranks. On September 12, Amodei published a nearly 4,000-word essay calling for a deceleration in AI development. “Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet,” he wrote.
Still, even as OpenAI appeared to embrace industry calls for a slowdown, there were reports that investor faith had not waned. The company is considering a new funding round that would double its valuation.
That new number: $1.5 trillion.
Reporting by Greg Bensinger and Deepa Seetharaman in San Francisco; Editing by Kenneth Li, Sayantani Ghosh and Matthew Lewis
Our Standards: The Thomson Reuters Trust Principles.
Greg Bensinger joined Reuters as a technology correspondent in 2022 focusing on the world’s largest technology companies. He was previously a member of The New York Times editorial board and a technology beat reporter for The Washington Post and The Wall Street Journal. He also worked for Bloomberg News writing about the auto and telecommunications industries. He studied English literature at The University of Virginia and graduate journalism at Columbia University. Greg lives in San Francisco with his wife and two children.
Deepa is a Reuters technology correspondent covering artificial intelligence and the companies driving its development, including OpenAI and Anthropic. She reports on how advances in AI are reshaping business, politics, and society. This is Deepa’s second stint at Reuters. She began her career at the news agency in New York and covered the U.S. auto industry from Detroit before moving to San Francisco to report on Amazon. She was part of a Reuters team named a finalist for the Gerald Loeb Award for Beat Reporting for their coverage of the United Auto Workers. She rejoined Reuters in September 2025. In between, she spent a decade at The Wall Street Journal, where she was the lead reporter covering Facebook and later artificial intelligence following the emergence of ChatGPT. Her reporting included coverage of Instagram’s impact on teenage girls and investigations into how AI systems falter in moderating racist and hateful content. She has been part of teams that won the George Polk Award for Business Reporting and the Gerald Loeb Award for Beat Reporting.