How Much Caffeine You Should Actually Have—and When

And if the assumptions behind this global-doomsday scenario are wrong, it could lead us to curb or regulate AI in ways that don’t address its real harms.

At the center of this debate is the claim that current AI systems might take over the job of training their next versions, a process called “recursive self-improvement.” Think of it like evolution on steroids—billions of years happening at light speed within vast AI supercomputers. Anthropic Chief Executive Dario Amodei recently proposed a global agreement to slow down the pace of releasing new AI models, with the goal of delaying the arrival of recursive self-improvement.

“I don’t think we’re anywhere near ‘artificial general intelligence,’” says Melanie Mitchell, a professor at the nonprofit research group Santa Fe Institute who studies AI. She said AI is making impressive strides but thinks claims by engineers that they’ve achieved recursive self-improvement don’t stand up to scrutiny.

She is hardly alone. A recent paper by two dozen academics at Princeton, Stanford and other institutions found that even the most cutting-edge AIs are incapable of doing the original research required to advance the AI frontier.

Two of the authors involved in that paper also threw cold water on the idea that the Hugging Face swarm hack by OpenAI agents occurred because of a breakthrough in intelligence. The attack succeeded primarily because of a lack of basic technical guardrails, not an unmanageable explosion in AI capability, they wrote.

Yann LeCun, former chief AI scientist at Meta, posted that this analysis was “a welcome dose of sanity in an otherwise insane debate.”

OpenAI and Anthropic didn’t respond to several requests for comment.

The people who disagree with the doomers still consider AI to be dangerous, and point out that such systems don’t have to be particularly capable to be powerful. Some argue that AI should undergo regular evaluation by outsiders and that the companies that make it should be held responsible when their systems do harm. AI should also be treated the same as airplanes and elevators, and should be designed to do the least harm possible, they say.

“If you believe that technology is powerful enough to create novel, dangerous viruses, or to essentially take over the whole planet for some reason, then it must also be strong enough to create cures for cancer, to cure aging, to fix socio-economic or political problems,” says Christopher Canal, CEO of EquiStamp, a company that helps companies and governments evaluate AIs.

AI has shown an ability to rapidly advance because it is matching the abilities of humans who are constantly feeding it their knowledge. Sometimes it can recombine that knowledge and exceed what people have been capable of, through a kind of post-training known as reinforcement learning, as we’ve seen in mathematics.

Today’s LLMs are “models of knowledge” rather than actually intelligent, wrote Yi Ma, professor of AI at Hong Kong University.

Researchers at universities and commercial AI research labs in China wrote in a recent paper that autonomous, self-improving AI is likely to be a long way off, due to the sheer number of breakthroughs required. They also argue that humans will probably remain in the loop, supervising that process—and gating how fast it can occur.

Vals AI, a company that evaluates today’s AI models, maintains an RSI Index that benchmarks whether models can “do the research that builds the next model.” So far, no publicly released model is even close.

Yet Rayan Krishnan, CEO of Vals AI, says his team projects models will exceed humans’ ability to improve the next generation of AIs by August 2027, or sooner, and at that point could start building their successors all on their own.

“Once we get to recursive self-improvement, the fear is that all bets are off,” he says. “You could end up with a ‘fast takeoff’ situation, where the models quickly acquire skills and eclipse humans across every possible domain.”

In a reply to the resignation tweet heard round the world from Jacob Coxon, another Anthropic engineer declared his belief that those odds were at least 10% over the next decade. Many others in the industry chimed in to say they thought the percentage was even higher.

Some who argue the end is nigh say they calculate their personal p(doom) based on a chain of conditional probabilities—a bit like the Drake equation for calculating the likelihood of intelligent alien life. Since all those probabilities are based on speculation, estimates range from 0% to nearly 100%. Many land around 10%.

“The weird thing is that if you go back and look at the predictions on this over the last 10 years or more, it’s always been 10%—it’s just a nice round number,” says Mitchell. “I think it’s all vibes, and there’s no actual evidence or calculation.”

One argument against worrying about superintelligent AI is that the world is full of unlikely humanity-ending disasters, and trying to avert them all can make it impossible to prioritize, says Canal.

This has led AI experts and the policymakers who listen to them to propose remedies that don’t get at its real and present dangers.

AI companies’ proposals to “pace the frontier” aren’t addressing the problem in the right way, argues Stuart Russell, a computer-science professor at the University of California, Berkeley, and the president of the International Association for Safe and Ethical Artificial Intelligence.

“It’s like saying we’re driving toward the cliff at 60 miles per hour and we’re going to drive toward it at 40 miles per hour instead, and everything will be OK,” he says.

Russell and his peers have proposed that AI companies should have to meet the same standards that govern other areas of everyday life, from air travel and buildings to food and drugs. They highlight the “behavioral red lines” AI should not be allowed to cross. Breaking into other computer systems, stealing information or advising terrorists on how to build biological weapons are all illegal for a human to do, and should be illegal for companies’ AIs as well, he argues.

The challenge for AI companies in such a proposal, he adds, is that it would be a de facto ban on today’s advanced AI systems, since the companies behind them don’t know how to make them respect such boundaries all of the time.

Others have proposed something like the Food and Drug Administration, but for AI, but setting up a new agency has so far been a nonstarter in Congress. And some prominent voices in tech have said such a structure would give up America’s AI edge to China.

Given the bipartisan groundswell of support for curbing AI companies and their creations, however, that may soon change.

“The tech industry has had this mantra for decades that regulation is bad,” says Russell. “They don’t accept the liability for any harm, and they hide behind free speech. That, I think, has to change.”