Vernor Vinge is possibly my favorite science fiction author of all time.[1] The Singularity, which he famously predicted would happen somewhere between 2005 and 2030, features heavily in most of his stories.
It usually happens the same way: an Intelligence Explosion, of the type envisioned by I.J. Good โ the smarter AI gets, the faster itโs able to make itself even smarter, meaning that AI becomes godlike nearly instantaneously once it reaches a certain threshold of capability.
Almost every discussion about AI safety that I hear in San Francisco or read in the tech blogosphere centers around this sort of event.
The discussions are all about whether recursive self-improvement (RSI) is near, and whether we should โpace the frontierโ โ i.e., deliberately slow down or pause AI progress โ to stop RSI from creating an intelligence explosion.
As Iโve listened to these arguments, Iโve grown increasingly skeptical that this is the right discussion we should be having. But before I explain why, let me quickly summarize the debate itself, to give an idea of what the arguments look like.
The intelligence explosion debate, in brief
In fact, Iโve published some pieces of that discussion on this blog. Back in September, Ramez Naam wrote a guest post arguing that an intelligence explosion is not imminent.
His basic arguments were:
- OpenAI data show that AI is still only able to automate short chunks of the AI research process.
- AI research is subject to diminishing returns, so that each increase in AI capability should buy less of a speedup in the pace of automated research.
- Diminishing returns also mean that accelerating AI progress will require exponentially more of other resources, such as compute and money.
- Thereโs no sign yet of an acceleration in the rate of AI progress.
In a follow-up post, Ramez did a deeper dive on the topic of agent swarms, arguing that theyโre not as effective as many people believe.
Theo Jaffee echoed these arguments, noting that OpenAIโs data shows that AI hasnโt even been able to fully automate smaller research tasks yet:

Luis Garicano and Alex Imas also wrote a post downplaying the likelihood of an intelligence explosion.
Their arguments are more economic in nature. They show that although AI progress has accelerated, it also shows diminishing returns in terms of research spending, meaning that every unit of progress has come with a higher price tag:

They also estimate an economic model of RSI,[2] and conclude that ideas in AI research are probably getting harder to find, reducing the likelihood of an intelligence explosion.
On the other side of the debate, we have the recent METR report about Anthropicโs internal progress in automating AI research. The report uses Anthropicโs own internal measure of AI capability, AECI, rather than the better-known public ECI.
It also relies on nonpublic data, so deciding how much of the report to believe requires a lot of guesswork and assumptions. But the basic story METR tells is that Claude is already speeding up Anthropicโs research process by a lot:
As we describe below, we were able to confidently determine that the acceleration [of AI progress in late 2025] was not due to an acceleration in the rate of growth of training computeโฆWe guess that a majority (perhaps โ80%) of the currentโฆacceleration is due to AI.
Dwarkesh Patel, meanwhile, argues that AI research isnโt that different from math, and that the same factors that made AI suddenly go from useless to godlike at math will happen soon with AIโs own self-improvement:
He writes:
Itโs crazy how fast we went from, in 2023, โThis model canโt reliably add four-digit numbers,โ toโฆabout a month ago, โOkay, it fucking solved a Millennium Prize problem in 88 hours.โโฆ
AI research is actually surprisingly similar to the type of mathematical progress that current AIs are disproportionately good at: unlocking a ton of low-hanging fruit by applying a combination of incredible persistence and a superhuman capacity to connect ideas from lots of adjacent domainsโฆ
In every domain where we can check answers and give AIs thousands of rounds of practice โ for example, competitive programming contests, or well-scoped open math problemsโtheyโve gone from worse than most humans to better than the best humans within a few yearsโฆAI research has those same verifiable propertiesโฆ
Once AI is as good as the best humans at AI research, algorithmic progress will get radically fasterโฆTodayโs AIs already do work at about 10 times human speed. And automated researchers would also be working around the clock, nights and weekends included.
As a result, each individual automated researcher will be cramming in roughly 50x more work every calendar week than a human researcher can doโฆDuring the intelligence explosion, automated engineers will build tens of thousands of times more [reinforcement learning] training environments than we have currently.
So thatโs the debate. Basically, it depends on whether ideas get harder to find faster than AI gets better at finding ideas, and only time will tell which one wins.
But as I said, while this debate is intellectually interesting, Iโm not sure how much it matters.
How useful is this debate?
Whether an intelligence explosion happens is obviously very important. A sudden, unfathomable leap in AI capabilities to godlike levels would be, to put it mildly, a very big deal. But that doesnโt mean an intelligence explosion is the thing we should be thinking and arguing about.
First of all, if an intelligence explosion is possible, itโs going to happen, and thereโs probably just not much we can do to affect how it all shakes out.
Although people disagree about AIโs contribution to its own uplift, everyone agrees that AI capabilities are improving at a steady, rapid rate. Hereโs Epoch AIโs ECI number, the most common measure of AI capabilities:

Most discussions about AI safety policy revolve around the idea of โpacing the frontierโ โ i.e., using government policy and AI labsโ own decisions to bend that trendline of progress downward. The usual reason given is that this will allow alignment research time to catch up, so that more powerful AI is safe by the time it arrives.
But this approach simply canโt deal with an intelligence explosion. If thereโs some threshold at which AI gets good enough to start an intelligence explosion, that threshold will be reached. Even if we โpaced the frontierโ and slowed the rate of AI progress down by half, that would mean weโd hit the intelligence explosion in 2037 instead of 2032, or 2030 instead of 2028, or something like that. It would only buy us a few extra years.
I guess two years or five years isnโt nothing โ we could use it to hang out with our loved ones and try some cool new restaurants while we waited for the arrival of the computer gods. But ultimately, it seems like small potatoes, especially if youโre an AI safety type who thinks an intelligence explosion would be a catastrophic event for humanity.
If an intelligence explosion is possible, the only way to prevent it is to halt AI progress completely, rather than just slow it down. In 1993, Vernor Vinge made a powerful argument as to why thatโs impossible:
But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the โthreatโ and be in deadly fear of it, progress toward the goal would continue. In fiction, there have been stories of laws passed forbidding the construction of โa machine in the form of the mind of man.โ
In fact, the competitive advantage — economic, military, even artistic of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first.
Iโm a little more optimistic than Vinge here โ I think we could engineer an AI slowdown if we got sufficiently scared โ but I donโt for one minute believe we could engineer a complete halt.
Nor do I believe that alignment research will be able to make an intelligence explosion safe. โPacing the frontierโ so that alignment research can catch up might be effective in a world of steady linear AI progress.[3] But in a world of sudden superexponential progress, AI alignment research will immediately be left behind.
Even if there is a Singularity for understanding alignment, in a world with an intelligence explosion it will be the superintelligent AI, not us, that carries out that Singularity and decides on its alignment for itself.
A second point is that itโs not obvious why an intelligence explosion would be more dangerous than โ or even distinguishable from โ the type of rapid, non-explosive progress weโre already getting. Remember, AI going from โcanโt add two numbers togetherโ to โsolves problems too hard for all human mathematicians combinedโ in three years is not counted as an intelligence explosion!
Itโs clear that at the current rate of steady progress, AI will become superhuman in most domains in a fairly small number of years. Even robotics and other physical tasks look conquerable with the current type of AI that powers Claude and GPT.
This means that AI will basically look like a god within our lifetimes, whether or not it looks like the kind of infinitely self-improving gods that populate Vernor Vinge novels.
This is why Iโm not sure what an intelligence explosion means for our economy. Itโs easy to write down a model in which smoothly increasing AI capabilities drive a smooth increase in GDP. But why should we believe a model like that? Other technologies havenโt really worked that way.
Usually, technologies reach some threshold of usefulness where adopting them makes economic sense. When they hit that threshold, their local contribution to productivity growth becomes very rapid, but also finishes fairly quickly.
Fiszbein et al. (2024) shows this for electrification, Bartel (2005) shows it for IT-enabled machines. Each new invention creates a whole bunch of little local Singularities. Itโs only when you add all these up across the economy that you get something that looks like a big smooth wave of growth.
A suddenly superintelligent AI isnโt really limited by locality โ it could, essentially by definition, take over all or most human tasks if it wanted to. Whereas if an intelligence explosion doesnโt occur, these takeovers are likely to be staggered and irregular. But they will still happen, and the economic effect after the first few years probably wonโt be that different.
In other words, I think:
- The AI weโll get from our current trend of improvement will look about as godlike โ and as economically important โ as an AI that suddenly bootstraps itself to godhood.
- Itโs not possible to stop AI improvement completely, and so if an intelligence explosion is possible, itโs going to happen to us soon.
- Slowing down the rate of AI progress will not help us deal with post-Singularity superintelligence.
This is why I think this whole debate is misplaced. We need to be thinking about how to deal with the AI we already have, and the one weโll almost certainly get soon โ an AI thatโs smarter than all humans at most economically important tasks.
We need to think about how to regulate that actually existing superintelligence, how to prevent AI-enabled bioterror, how to make sure the economic benefits of AI are widely shared, and so on. We need to think about competition policy, to make sure the benefits of AI donโt exclusively go to one company.
I think we need to accept that nobody is โpacing the frontierโ enough to prevent the emergence of superintelligence โ and that nobody is likely to do so. If the machine god awakes, then it awakes, and weโll simply deal with it. In the meantime, we need to focus on dealing with the AI we have now.
Notes
1 The other contender would be Lois McMaster Bujold, who writes a very different kind of sci-fi.
2 This estimation is done with only 16 data points, so I donโt really trust it. But weโre basically flying blind here, so this is the best weโve got.
3 In fact, I invested in a company called Goodfire thatโs at the forefront of AI alignment research.
This article was first published on Noah Smithโs Noahpinionย Substack and is republished with kind permission. Become a Noahopinionย subscriberย here.
