How quick and big would a software intelligence explosion be?

Introduction

Summary

How the model works

Image

1. Initial speed-up. When ASARA is initially deployed, how much faster is software progress compared to the recent pace of software progress?
2. Returns to software R&D. After the initial speed-up from ASARA, does the pace of progress accelerate or decelerate as AI progress feeds back on itself?

• a. This is given by the parameter $r$. Progress accelerates if and only if $r>1$.
• b. $r$ depends on 1) the extent to which software improvements get harder to find as the low hanging fruit are plucked, and 2) the strength of the “stepping on toes effect” whereby there are diminishing returns to more researchers working in parallel.

3. Distance to “effective limits” on software. How much can software improve after ASARA before we reach fundamental limits to the compute efficiency of AI software?

• a. The model assumes that, as software approaches effective limits, the returns to software R&D become less favourable and so AI progress decelerates.

Image

Image

Results

Image

• Initial speed-ups of >3X are likely, and the returns to software R&D are likely high enough to prevent progress slowing back down before there is 3 years of progress.
• If effective limits are >10 OOMs away and returns to software R&D remain favourable until we are close to those limits, progress can accelerate for long enough to get ten years of progress in a year. We think it’s plausible but unlikely that both these conditions hold.
• To get 30 years of progress in one year, either you need extremely large efficiency gains on top of ASARA (30 OOMs!) or a major paradigm shift that enables massive progress without significant increases in effective compute (which seems more likely).

1. Retraining new models from scratch. In this variant, some fraction of software progress is “spent” making training runs faster as AI progress accelerates. More.
2. Gradual boost. In this variant, we simulate a gradual increase in AI capabilities from today’s AI to ASARA, with software progress accelerating along the way. More.

Discussion

Limitations

• Assuming AI progress follows smooth trends. We don’t model the possibility that superhuman AI unlocks qualitatively new forms of progress that amount to a radical paradigm shift; nor the possibility that the current paradigm stops yielding further progress shortly after ASARA. So we’ll underestimate the size of the tails in both directions.
• No gears-level analysis. We don’t model how AIs will improve software in a gears-level way at all (e.g. via generating synthetic data vs by designing new algorithms). So the model doesn’t give us insight into these dynamics. Instead, we extrapolate high-level trends about how much research effort is needed to double the efficiency of AI algorithms. And we don’t model specific capabilities, just the amount of “effective training compute”. 
• “Garbage in, garbage out”. We’ve done our best to estimate the model parameters, but there are massive uncertainties in all of them. This flows right through to the results. 

  • You can choose your own inputs to the model here!
• No compute growth. The simulation assumes that compute doesn’t grow at all after ASARA is deployed, which is obviously a conservative assumption.

Structure of the paper

• We explain how this paper relates to previous work on the intelligence explosion.
• We clarify the scenario being considered.
• We outline the model dynamics in detail.
• We evaluate which parameters values to use – this is the longest section.
• We summarise the parameter values chosen.
• We describe the model predictions.
• We discuss limitations.

Relation to previous work

Scenario

• ASARA is deployed internally within an AI developer to fully automate AI R&D. 
• There are no further human bottlenecks on the pace of progress – i.e. no pauses to help humans understand developments, ensure AI is developed safely, or assess legal risk. 
• After ASARA is deployed, the compute used for AI R&D remains constant over time.
• Recent scaling laws on capabilities roughly continue: each doubling of effective compute for developing AI improves capabilities by the same amount as it has in recent years.

Model dynamics

• $S$ is the level of AI software
• $L$ is the cognitive labour used for improving software
• $C$ is compute for experiments, assumed to be constant
• $\lambda$ captures "stepping on toes", whereby there are diminishing returns to applying more research effort in parallel (nine women can't make a baby in a month!).
• $\beta (> 0)$ captures software improvements getting harder to find as software improves
• $\alpha$ captures the diminishing returns of cognitive labour to improving software
• $a$, $b$, $c$ and $d$ are constants that are collectively pinned down by growth rates at the beginning of the model

Pseudocode

• $\text{Doubling-Time}_0$. We calculate this from 1) the recent software doubling time (which we assume is 3 months) and 2) our estimate of the initial speed-up of software progress from deploying ASARA.
• $\text{Doublings-Till-Ceiling}$, i.e. the distance to the effective limits on software.5

   
• $r_0$, the (deflated) returns to software R&D when ASARA is first developed.
• $p$, the diminishing returns to parallel labour.

Parameter values

• Initial speed-up of software progress from deploying ASARA
• Returns to software R&D
• Distance to effective limits of software
• Diminishing returns to parallel labour (less important)

Initial speed-up of software progress from deploying ASARA

• Survey researchers about speed-ups from abundant cognitive labour. This paper surveyed five researchers about how much ASARA would speed up the pace of AI progress. They used the same definition of ASARA as we’re using. 

  • When asked outright about the total speed-up, responses varied from 2X to 20X with a geometric mean of 5X. If software progress accounts for 50% of total progress, then the corresponding speed-up in software progress is 10X.
  • When asked to estimate the speed-ups from different sources separately (e.g. reducing bugs, doing experiments at small scales), their estimates were higher, with a geometric mean of 14X. This translates into a speed-up in software progress of 28X.
• Survey researchers about the per-person slowdown from reduced compute. If you have way more virtual researchers, each one of them will have much less compute. How much more slowly will they make progress? 

  • AI 2027 interviewed AI researchers and found that with 10X less compute they thought they’d make 40% slower progress. From this, along with a few other assumptions, they estimate that ASARA would accelerate software progress by a factor of 21X.8

• Analysing specific sources of speed-up. AI 2027 evaluates specific speed-ups that abundant AI labour would enable — smaller experiments, better prioritisation, better experiments, less wasted compute. They explicitly account for compute bottlenecks in their analysis but still find large gains.

  • This approach forecasts a 5X speed up from the superhuman coder (which is maybe similarly capable to ASARA) and forecasts 417X from a superhuman researcher (which is more capable than ASARA).
• Thought experiment about an AI lab with fewer + slower researchers. ASARA would increase the number and speed of researchers working on AI R&D. We might expect the opposite effect if we instead decreased the number of human researchers and made them much slower. In particular, imagine if frontier AI labs had 30X fewer researchers and they each thought 30X more slowly – how much slower would AI progress be? If you think progress would be a lot slower, that suggests ASARA might speed up progress a lot. Ryan Greenblatt and Eli LIfland explore this idea here.
• Use a simple economic model of R&D (the same model as we use in our simulation)

  • In particular, this model is: $g_S = L^{\alpha} C^{1 - \alpha} S^{-\beta} , L = L_{\text{parallel}}^{\lambda}$
  • Our median model assumptions correspond to $\alpha = 0.5$, $\lambda = 0.6$.
  • ASARA brings two changes: 1) 30X as many researchers in parallel, 2) every researcher thinks 30X faster

    • 1. multiplies $L_{\text{parallel}}$ by 30, which multiplies the rate of software progress by $30^{\alpha^{\lambda}} = 2.8$.
    • 2. multiplies $L$ by 30 which multiplies the rate of software progress by $30^{\alpha} = 30^{0.5} = 5.5$.
    • Multiplying these independent effects together, we get the total speed up of 15X.

Image

Image

Returns to software R&D, $r$

• Upwards for improving capabilities. Improving training efficiency improves capabilities, as you can train a model with more effective compute. Imagine we use a 2X training efficiency gain to train a model with twice as much effective compute. How many times would that double “software”? (I.e., how many doublings of parallel copies would be equally useful?) There are various sources of evidence here:12

  toy ML experiments suggest the answer is ~1.7; human productivity studies suggest the answer is ~2.5. We put more weight on the former, so we'll estimate 2. This doubles our median estimate to $\boldsymbol{r = \, \sim 2.8} \; (1.4 * 2)$.
• Upwards for post-training enhancements. So far, we’ve only considered pre-training improvements. But post-training enhancements like fine-tuning, scaffolding, and prompting also improve capabilities (o1 was developed using such techniques!). These can allow faster thinking, which could be a big factor. But there might also be strong diminishing returns to post-training enhancements holding base models fixed. We'll adjust our median estimate up from 2.8 to $\boldsymbol{r = \; \sim 4} \; (2.8 * 1.45)$.
• Downwards for less growth in compute for experiments. Today, rising compute means we can run increasing numbers of GPT-3-sized experiments each year. This helps drive software progress. But compute isn’t growing in our scenario. That might mean that returns to additional cognitive labour diminish more steeply. On the other hand, the most important experiments are ones that use similar amounts of compute to training a SOTA model. Rising compute hasn't actually increased the number of these experiments we can run, as rising compute increases the training compute required for these SOTA models. And experiments are much less of a bottleneck for post-training enhancements. But this still reduces our median estimate down to $\boldsymbol{r = \; \sim 3}$. (See Eth and Davidson (2025) for more discussion.)
• Downwards for fixed scale of hardware. In recent years, the scale of hardware available to researchers has increased massively. Researchers could invent new algorithms that only work at the new hardware scales for which no one had previously tried to to develop algorithms. Researchers may have been plucking low-hanging fruit for each new scale of hardware. But in the software intelligence explosions we're considering, this won’t be possible because the hardware scale will be fixed. OAI estimate ImageNet efficiency via a method that accounts for this (by focussing on a fixed capability level),13

  and find a 16-month doubling time, as compared with Epoch’s 9-month doubling time. This reduces our estimate down to $\boldsymbol{r = \; \sim 1.7} \; (3 * 9/16)$
• Downwards for the returns to software R&D becoming worse over time. In most fields, returns diminish more steeply than in software R&D.14

  So perhaps software will tend to become more like the average field over time. To estimate the size of this effect, we can take our estimate that software is ~10 OOMs from effective limits (discussed below), and assume that for each OOM increase in software, $r$ falls by a constant amount, reaching zero once effective limits are reached. If $r = 1.7$, then this implies that $r$ reduces by 0.17 for each OOM. Epoch estimates that pre-training algorithmic improvements are growing by an OOM every ~2 years, which would imply a reduction in $r$ of $0.51 \; (3 * 0.17)$ by 2030. This reduces our median estimate to $\boldsymbol{r = \; \sim 1.2} \; (1.7 - 0.5)$.

Image

Image

Image

Distance to the effective limits to software

Gap from ASARA to human learning

Gap from human learning to effective limits

• Not enough data – the brain is severely “undertrained”. Chinchilla optimal scaling suggests that models should be trained on ~20X as many tokens as they have parameters. On this account, the human brain is severely “undertrained”: it has maybe ~1e14 “parameters” (synapses) but only processes ~1e9 “data points” (1 data point per second for 30 years) during lifetime learning. If the Chinchilla scaling laws applied to the brain, then you could train a model with the same performance as the brain but with 4-5 OOMs less training compute – see this appendix. Of course, the brain architecture might have a completely different optimal scaling to Chinchilla. But it is very plausible that the brain is significantly undertrained due to hard biological constraints: organisms don’t live long enough to keep learning for 1000s of years! We'll call this 10-100,000X.
• Low fraction of data is relevant. Take a human ML expert. They only spend a small fraction of their time doing focussed learning that is relevant to doing AI R&D (at most 8 hours a day on average, and plausibly much less given the time spent on poetry, history, etc). This seems like a factor of at least 3-10X.19

   
• Data quality could be much higher. Even when humans are learning relevant material, the quality of the data they would be processing is far from optimal. When an outstanding communicator puts significant effort into organising and presenting content, human learning can be much more efficient. During the software intelligence explosion AI could curate better data sets still, including demonstrations of significantly superhuman performance. It seems like this would be at least 3X and plausibly 300X.

• Brain algorithms must satisfy physical constraints. 

  • Firstly, the human brain is spread out over three dimensions, communications from one part of the brain to the other must physically ‘travel’ this distance, and the speed of communication is fairly slow. This constrains the design space of brain algorithms. 
  • Secondly, the brain cannot do weight sharing (where the same numerical ‘weights’ are used in different parts of the network), which is essential to transformers. 
  • Thirdly, synaptic updates are local, just depending on nearby synapses, whereas AI can implement non-local learning rules like stochastic gradient descent where updates are instantaneously propagated throughout the entire network. 
  • AI algorithms will relax these constraints. On the other hand, the hardware first used to train AGI will have different constraints – e.g. around memory access. Also, evolution coevolved the brain’s hardware and software, which won’t be possible during a software intelligence explosion. 
  • It’s hard to know how significant this will be overall, maybe 3-100X.
• Significant variation between humans. There is significant variation in cognitive abilities between different humans. It may be possible to significantly increase human intelligence (e.g. by >100 IQ points) just by combining together all the best mutations and making more of the same kinds of changes that drive human variation. This might be another 3-10X.20

   
• Fundamental improvements in the brain’s learning algorithm. Evolution is a blind search process and does not always find the optimal solutions to problems – e.g. giraffes have a nerve that loops up and down their neck only to end up right next to where it started. AI could write vastly more complicated learning algorithms than the human brain algorithm, which is encoded within the human genome and so cannot exceed ~1e9 bits. In the limit, AI could hard-code knowledge GOFAI-style rather than learning it. Again, it’s hard to know how big this will be. Maybe 3-30X.
• Coordination. Humans must coordinate via language, which is slow and imperfect. By contrast, AIs could potentially communicate via multidimensional vectors and copies could share their activations with one another. This might significantly raise the collective capabilities achievable by large teams. Maybe 3-10X.

Image

Image

Diminishing returns to parallel labour

• $\lambda$ captures diminishing returns to parallel efforts – nine women can’t make a baby in a month! We use $λ=0.6$.22

   
• $\alpha$ captures the relative importance of cognitive labour inputs to software R&D, as contrasted with inputs of compute for experiments. We assume the share of software progress that’s attributable to cognitive labour is 0.5.

Summary of parameter assumptions

Results

Image

• Retraining cost. This variant models the fact that we’ll need to train new generations of AI from scratch during the intelligence explosion and, if progress is accelerating, those training runs will have to become quicker over time. More.
• Gradual boost. This variant models AI systems intermediate between today’s AI and ASARA. It assumes AI gradually accelerates software progress more and more over time. More.

Limitations and caveats

• We make this assumption when estimating the returns to software R&D $r$, in particular when assessing the benefits of training models with better capabilities.

• You can choose your own inputs here!

Appendices

Derivation of our pseudo code from the semi-endogenous growth model

Additional model assumptions

1. AI software has recently been doubling every 3 months (more).
2. Half of AI progress is due to software and half is due to compute (more).
3. Our sampling procedure for the Monte Carlo (more).

AI software has recently been doubling every 3 months

Calculating the summary statistics

Objection to our sampling procedure

Variants of the model

Retraining time

Image

Image

Image

Gradual boost (from pre-ASARA systems) 

Image

Image

Returns to software R&D: efficiency improvements only

• Downwards for runtime efficiency. To estimate returns to software improvements, we need to convert from training efficiency (the inputs) into runtime efficiency (the outputs). The logic of the Chinchilla paper implies that increasing training efficiency by 4X will increase runtime efficiency by between 2X and 4X.25

  This means we should reduce our estimate of $r$ by up to a factor of 2. We'll reduce it to $\boldsymbol{r = \; \sim1}$.
• Upwards for post-training enhancements. Improvements to pre-training algorithms are not the only source of runtime efficiency gains. There are also techniques like: model distillation; calling smaller models for easier tasks; pruning (removing parameters from a trained model); post-training quantisation (reducing the numerical precision of the weights); more efficient caching of results and activations (especially for agents that re-read the same context multiple times). We'll increase our estimate to $\boldsymbol{r = \; \sim 1-2}$. 

Distance to effective limits: human learning

Image

Discussion of model assumptions

How the returns to software R&D $r$ changes over time

1. By the time software reaches effective limits, $r = 0$. This means that further software progress is impossible.
2. Each time software doubles, $r$ decreases by a constant absolute amount. This absolute amount is chosen to be consistent with #1.

   • So, for example: once software has advanced half the distance to effective limits in log-space, the value of $r$ will have halved. Each doubling of cumulative inputs will double software half as many times as before.

How the speed of software progress changes over time