Image: Noahpinion

Vernor Vinge is possibly my favorite science fiction author of all time.[1] The Singularity, which he famously predicted would happen somewhere between 2005 and 2030, features heavily in most of his stories.

It usually happens the same way: an Intelligence Explosion, of the type envisioned by I.J. Good โ€” the smarter AI gets, the faster itโ€™s able to make itself even smarter, meaning that AI becomes godlike nearly instantaneously once it reaches a certain threshold of capability.

Almost every discussion about AI safety that I hear in San Francisco or read in the tech blogosphere centers around this sort of event.

The discussions are all about whether recursive self-improvement (RSI) is near, and whether we should โ€œpace the frontierโ€ โ€” i.e., deliberately slow down or pause AI progress โ€” to stop RSI from creating an intelligence explosion.

As Iโ€™ve listened to these arguments, Iโ€™ve grown increasingly skeptical that this is the right discussion we should be having. But before I explain why, let me quickly summarize the debate itself, to give an idea of what the arguments look like.

The intelligence explosion debate, in brief

In fact, Iโ€™ve published some pieces of that discussion on this blog. Back in September, Ramez Naam wrote a guest post arguing that an intelligence explosion is not imminent.

His basic arguments were:

  • OpenAI data show that AI is still only able to automate short chunks of the AI research process.
  • AI research is subject to diminishing returns, so that each increase in AI capability should buy less of a speedup in the pace of automated research.
  • Diminishing returns also mean that accelerating AI progress will require exponentially more of other resources, such as compute and money.
  • Thereโ€™s no sign yet of an acceleration in the rate of AI progress.

In a follow-up post, Ramez did a deeper dive on the topic of agent swarms, arguing that theyโ€™re not as effective as many people believe.

Theo Jaffee echoed these arguments, noting that OpenAIโ€™s data shows that AI hasnโ€™t even been able to fully automate smaller research tasks yet:

Source: OpenAI via Theo Jaffee

Luis Garicano and Alex Imas also wrote a post downplaying the likelihood of an intelligence explosion.

Their arguments are more economic in nature. They show that although AI progress has accelerated, it also shows diminishing returns in terms of research spending, meaning that every unit of progress has come with a higher price tag:

Source: Garicano and Imas

They also estimate an economic model of RSI,[2] and conclude that ideas in AI research are probably getting harder to find, reducing the likelihood of an intelligence explosion.

On the other side of the debate, we have the recent METR report about Anthropicโ€™s internal progress in automating AI research. The report uses Anthropicโ€™s own internal measure of AI capability, AECI, rather than the better-known public ECI.

It also relies on nonpublic data, so deciding how much of the report to believe requires a lot of guesswork and assumptions. But the basic story METR tells is that Claude is already speeding up Anthropicโ€™s research process by a lot:

As we describe below, we were able to confidently determine that the acceleration [of AI progress in late 2025] was not due to an acceleration in the rate of growth of training computeโ€ฆWe guess that a majority (perhaps โ‰ˆ80%) of the currentโ€ฆacceleration is due to AI.

Dwarkesh Patel, meanwhile, argues that AI research isnโ€™t that different from math, and that the same factors that made AI suddenly go from useless to godlike at math will happen soon with AIโ€™s own self-improvement:

He writes:

Itโ€™s crazy how fast we went from, in 2023, โ€œThis model canโ€™t reliably add four-digit numbers,โ€ toโ€ฆabout a month ago, โ€œOkay, it fucking solved a Millennium Prize problem in 88 hours.โ€โ€ฆ

AI research is actually surprisingly similar to the type of mathematical progress that current AIs are disproportionately good at: unlocking a ton of low-hanging fruit by applying a combination of incredible persistence and a superhuman capacity to connect ideas from lots of adjacent domainsโ€ฆ

In every domain where we can check answers and give AIs thousands of rounds of practice โ€” for example, competitive programming contests, or well-scoped open math problemsโ€”theyโ€™ve gone from worse than most humans to better than the best humans within a few yearsโ€ฆAI research has those same verifiable propertiesโ€ฆ

Once AI is as good as the best humans at AI research, algorithmic progress will get radically fasterโ€ฆTodayโ€™s AIs already do work at about 10 times human speed. And automated researchers would also be working around the clock, nights and weekends included.

As a result, each individual automated researcher will be cramming in roughly 50x more work every calendar week than a human researcher can doโ€ฆDuring the intelligence explosion, automated engineers will build tens of thousands of times more [reinforcement learning] training environments than we have currently.

So thatโ€™s the debate. Basically, it depends on whether ideas get harder to find faster than AI gets better at finding ideas, and only time will tell which one wins.

But as I said, while this debate is intellectually interesting, Iโ€™m not sure how much it matters.

How useful is this debate?

Whether an intelligence explosion happens is obviously very important. A sudden, unfathomable leap in AI capabilities to godlike levels would be, to put it mildly, a very big deal. But that doesnโ€™t mean an intelligence explosion is the thing we should be thinking and arguing about.

First of all, if an intelligence explosion is possible, itโ€™s going to happen, and thereโ€™s probably just not much we can do to affect how it all shakes out.

Although people disagree about AIโ€™s contribution to its own uplift, everyone agrees that AI capabilities are improving at a steady, rapid rate. Hereโ€™s Epoch AIโ€™s ECI number, the most common measure of AI capabilities:

Source: Epoch AI

Most discussions about AI safety policy revolve around the idea of โ€œpacing the frontierโ€ โ€” i.e., using government policy and AI labsโ€™ own decisions to bend that trendline of progress downward. The usual reason given is that this will allow alignment research time to catch up, so that more powerful AI is safe by the time it arrives.

But this approach simply canโ€™t deal with an intelligence explosion. If thereโ€™s some threshold at which AI gets good enough to start an intelligence explosion, that threshold will be reached. Even if we โ€œpaced the frontierโ€ and slowed the rate of AI progress down by half, that would mean weโ€™d hit the intelligence explosion in 2037 instead of 2032, or 2030 instead of 2028, or something like that. It would only buy us a few extra years.

I guess two years or five years isnโ€™t nothing โ€” we could use it to hang out with our loved ones and try some cool new restaurants while we waited for the arrival of the computer gods. But ultimately, it seems like small potatoes, especially if youโ€™re an AI safety type who thinks an intelligence explosion would be a catastrophic event for humanity.

If an intelligence explosion is possible, the only way to prevent it is to halt AI progress completely, rather than just slow it down. In 1993, Vernor Vinge made a powerful argument as to why thatโ€™s impossible:

But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the โ€œthreatโ€ and be in deadly fear of it, progress toward the goal would continue. In fiction, there have been stories of laws passed forbidding the construction of โ€œa machine in the form of the mind of man.โ€

In fact, the competitive advantage — economic, military, even artistic of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first.

Iโ€™m a little more optimistic than Vinge here โ€” I think we could engineer an AI slowdown if we got sufficiently scared โ€” but I donโ€™t for one minute believe we could engineer a complete halt.

Nor do I believe that alignment research will be able to make an intelligence explosion safe. โ€œPacing the frontierโ€ so that alignment research can catch up might be effective in a world of steady linear AI progress.[3] But in a world of sudden superexponential progress, AI alignment research will immediately be left behind.

Even if there is a Singularity for understanding alignment, in a world with an intelligence explosion it will be the superintelligent AI, not us, that carries out that Singularity and decides on its alignment for itself.

A second point is that itโ€™s not obvious why an intelligence explosion would be more dangerous than โ€” or even distinguishable from โ€” the type of rapid, non-explosive progress weโ€™re already getting. Remember, AI going from โ€œcanโ€™t add two numbers togetherโ€ to โ€œsolves problems too hard for all human mathematicians combinedโ€ in three years is not counted as an intelligence explosion!

Itโ€™s clear that at the current rate of steady progress, AI will become superhuman in most domains in a fairly small number of years. Even robotics and other physical tasks look conquerable with the current type of AI that powers Claude and GPT.

This means that AI will basically look like a god within our lifetimes, whether or not it looks like the kind of infinitely self-improving gods that populate Vernor Vinge novels.

This is why Iโ€™m not sure what an intelligence explosion means for our economy. Itโ€™s easy to write down a model in which smoothly increasing AI capabilities drive a smooth increase in GDP. But why should we believe a model like that? Other technologies havenโ€™t really worked that way.

Usually, technologies reach some threshold of usefulness where adopting them makes economic sense. When they hit that threshold, their local contribution to productivity growth becomes very rapid, but also finishes fairly quickly.

Fiszbein et al. (2024) shows this for electrification, Bartel (2005) shows it for IT-enabled machines. Each new invention creates a whole bunch of little local Singularities. Itโ€™s only when you add all these up across the economy that you get something that looks like a big smooth wave of growth.

A suddenly superintelligent AI isnโ€™t really limited by locality โ€” it could, essentially by definition, take over all or most human tasks if it wanted to. Whereas if an intelligence explosion doesnโ€™t occur, these takeovers are likely to be staggered and irregular. But they will still happen, and the economic effect after the first few years probably wonโ€™t be that different.

In other words, I think:

  • The AI weโ€™ll get from our current trend of improvement will look about as godlike โ€” and as economically important โ€” as an AI that suddenly bootstraps itself to godhood.
  • Itโ€™s not possible to stop AI improvement completely, and so if an intelligence explosion is possible, itโ€™s going to happen to us soon.
  • Slowing down the rate of AI progress will not help us deal with post-Singularity superintelligence.

This is why I think this whole debate is misplaced. We need to be thinking about how to deal with the AI we already have, and the one weโ€™ll almost certainly get soon โ€” an AI thatโ€™s smarter than all humans at most economically important tasks.

We need to think about how to regulate that actually existing superintelligence, how to prevent AI-enabled bioterror, how to make sure the economic benefits of AI are widely shared, and so on. We need to think about competition policy, to make sure the benefits of AI donโ€™t exclusively go to one company.

I think we need to accept that nobody is โ€œpacing the frontierโ€ enough to prevent the emergence of superintelligence โ€” and that nobody is likely to do so. If the machine god awakes, then it awakes, and weโ€™ll simply deal with it. In the meantime, we need to focus on dealing with the AI we have now.


Notes

1 The other contender would be Lois McMaster Bujold, who writes a very different kind of sci-fi.

2 This estimation is done with only 16 data points, so I donโ€™t really trust it. But weโ€™re basically flying blind here, so this is the best weโ€™ve got.

3 In fact, I invested in a company called Goodfire thatโ€™s at the forefront of AI alignment research.

This article was first published on Noah Smithโ€™s Noahpinionย Substack and is republished with kind permission. Become a Noahopinionย subscriberย here.