Once AI systems can design and build even more capable AI systems, we could see an *intelligence explosion*, where AI capabilities rapidly increase to well past human performance.  

The classic intelligence explosion scenario involves a feedback loop where AI improves AI software. But AI could also improve other inputs to AI development. This paper analyses three feedback loops in AI development: software, chip technology, and chip production. These could drive three types of intelligence explosion: a software intelligence explosion driven by software improvements alone; an AI-technology intelligence explosion driven by both software and chip technology improvements; and a full-stack intelligence explosion incorporating all three feedback loops. 

Even if a software intelligence explosion never materializes or plateaus quickly, AI-technology and full-stack intelligence explosions remain possible. And, while these would start more gradually, they could accelerate to very fast rates of development. Our analysis suggests that each feedback loop by itself could drive accelerating AI progress, with effective compute potentially increasing by 20-30 orders of magnitude before hitting physical limits—enabling truly dramatic improvements in AI capabilities. The type of intelligence explosion also has implications for the distribution of power: a software intelligence explosion would by default concentrate power within one country or company, while a full-stack intelligence explosion would be spread across many countries and industries.

 Tom did the original thinking; Will and then Rose helped with later thinking and writing.

Tom1

Tom Davidson

Rose1

Rose Hadshar

Will1

Will MacAskill

Three types of intelligence explosion - main

Three types of intelligence explosion 

In simplified models of intelligence explosions, accelerating AI progress goes to infinity. But in reality, we will hit effective limits on AI intelligence. In this piece, we analyze the room for improvement in three inputs to AI development: software, chip technology and chip production. Setting human constraints aside, we argue that software could increase effective compute by ~10 orders of magnitude or more, chip technology by ~6 orders of magnitude, and chip production by ~5 orders of magnitude using earth-based energy capture, and a further ~9 orders of magnitude capturing all solar energy from space. In combination, we could see effective compute increase by 20-30 orders of magnitude before hitting physical limits - enough to drive immense increases in AI capabilities. 

Tom did the original thinking; Will and then Rose helped with later thinking and writing. Thanks to Max Dalton, Oscar Delaney, Ege Erdil, Daniel Eth, John Halstead, Anson Ho, Fin Moorhouse, Ben Todd and the governing explosive growth seminar group for helpful comments.


Once AI systems can themselves design and build even more capable AI systems, this will create feedback loops where AI builds better AI which builds better AI. More specifically, we can think about three important feedback loops:

* A *software* feedback loop, where AI develops better software. Software includes AI training algorithms, post-training enhancements, ways to leverage runtime compute (like o1), synthetic data, and any other non-compute improvements.  
* A *chip technology* feedback loop, where AI designs better computer chips. Chip technology includes all the cognitive research and design work done by NVIDIA, TSMC, ASML, and other semiconductor companies.  
* A *chip production* feedback loop, where AI and robots build more computer chips.

If one or several of these feedback loops cause AI progress to become very fast, we could call this an “[intelligence explosion](/research/three-types-of-intelligence-explosion)”.

In simplified models of intelligence explosions, progress goes to [infinity](https://www.openphilanthropy.org/research/could-advanced-ai-drive-explosive-economic-growth/#id-618-models-predicting-explosive-growth-have-implausible-implications). But in reality, this is of course implausible. We will eventually hit effective limits on all of software, chip technology and chip production improvements, and so progress from each of these sources will eventually plateau. And progress could slow long before effective physical limits are reached, because of diminishing marginal returns, regulation, or other factors.

In this essay, we set aside human constraints and focus on effective physical limits, as these are easier to analyse.

How far can AI progress before hitting effective physical limits? If we are already close to the limits, then the potential capability increases will be smaller. But if the limits are very high, then ultimate capability levels (and the maximum pace of progress to reach them) could be extreme.

We can operationalize the distance to physical limits in terms of effective compute.

We argue that, setting aside human constraints:

* [**Software**](#software-progress) could likely increase effective compute by **\~13 OOMs**, possibly more.   
* [**Chip technology**](#chip-technology-progress) could likely increase effective compute by **\~2 OOMs** within the [current paradigm](https://epoch.ai/blog/limits-to-the-energy-efficiency-of-cmos-microprocessors), and by a total of **\~6 OOMs** if technology approaches [Landauer’s limit](https://en.wikipedia.org/wiki/Landauer%27s_principle) (a physical constraint on the energy efficiency of computation). [Reversible computing](https://en.wikipedia.org/wiki/Reversible_computing) could conceivably go further still.   
* [**Chip production**](#chip-production-progress) could scale by **\~5 OOMs** using earth-based energy capture, and by a further **\~9 OOMs** if space-based solar could capture *all* the energy emitted by the sun. 

![Effective compute cube](<//images.ctfassets.net/4owxfjx3z3if/23fLSSbvDpSUG4OVxuFAnR/fe8520bb5ccec626e87748252e5cfc02/Amrit_Graphs_-31.png> "There are three ways to increase effective compute in our framework: better software, better chip technology, and more chip production.")

# Software progress

Software could likely increase effective compute by **\~13 OOMs,** possibly more. 

Focusing just on the room for improvement for training efficiency, how much better could software get before it reaches effective physical limits?

* Human lifetime learning takes [1e24 FLOP](https://docs.google.com/document/d/1IJ6Sr-gPeXdSJugFulwIpvavc0atjHGM82QjIfUSBGQ/edit?tab=t.0#heading=h.87mp14r9lgsj). Assuming the relative gap between GPT-4 and GPT-6 will be the same as the gap between GPT-2 and GPT-4, GPT-6 will be trained with 1e29 FLOP.[^1]  
* If each GPT-6 computation were as useful as a human computation, then GPT-6 would be \~5 OOMs less training efficient than human lifetime learning, suggesting at least 5 more OOMs to physical limits.[^2] Of course, GPT-6 might be more or less useful per computation as a human brain, but we’ll use this as a rough estimate.   
* But the human brain is not the physical limit of training efficiency. Firstly, the data used for human learning could be greatly improved upon: the brain is severely undertrained, humans spend only a small fraction of their time on focussed academic learning,[^3] and data quality could be much higher. Secondly, brain algorithms could be improved upon. Brains have to satisfy biological physical constraints, and humans have to coordinate via language. Combined with the significant variation between humans and the fact that evolution is a blind search process, it seems that there is significant room for improvement. Davidson (forthcoming) estimates that these factors add another 4-12 OOMs, so 9-17 OOMs total. The midpoint of this range is \~13 OOMs.

There will likely be other kinds of gain besides training efficiency, which would increase this estimate somewhat.

# Chip technology progress

Chip technology could likely increase effective compute by **\~2 OOMs** within the current paradigm, and by a total of **\~6 OOMs** if technology approaches Landauer’s limit. [Reversible computing](https://en.wikipedia.org/wiki/Reversible_computing) could go further still. 

We’ll measure this in units of FLOP/J – how many floating point operations chips can do per joule of energy. (This is consistent with our calculation of the limits to chip production below, where we consider the scale at which we could increase the energy used by chips.) 

[Epoch estimates](https://epoch.ai/blog/limits-to-the-energy-efficiency-of-cmos-microprocessors) that within the current paradigm, there is room for a \~200X improvement, which is \~2 OOMs.[^4]

But beyond the current paradigm, there could be room for further improvements. [Landauer’s limit](https://en.wikipedia.org/wiki/Landauer%27s_principle) implies that it’s not physically possible to do more than 3e20 bit erasures per Joule. We want to convert this limit to FLOP/J, so we need to estimate the number of bit erasures per FLOP. As a rough ballpark estimate for the limits of non-reversible computing, if we use 8-bit precision numbers, we might have slightly more than 8 bit erasures per FLOP – 8 because a FLOP takes in two 8-bit numbers but outputs only one; and a little more for some intermediate calculations along the way. Let’s round it to 10 bit erasures per FLOP. That implies we could achieve 3e19 FLOP/J.

Today you can get [\~1e13 FLOP/J](https://epoch.ai/blog/limits-to-the-energy-efficiency-of-cmos-microprocessors), which implies \~6.5 OOMs of improvement before reaching effective physical limits.

If [reversible computing](https://en.wikipedia.org/wiki/Reversible_computing) is possible, the ceiling could be higher still: Landauer’s limit only applies to irreversible logical operations, so reversible computing could surpass this limit.

# Chip production progress

Chip production could scale by **\~5 OOMs** using earth-based energy capture, and by a further **\~9 OOMs** if space-based solar could capture *all* the energy emitted by the sun:

* Data centres currently consume [1-1.5%](https://www.iea.org/energy-system/buildings/data-centres-and-data-transmission-networks) of global electricity, and [10-20%](https://www.epri.com/research/products/3002028905) of data centres are used for AI, so AI chips currently consume 0.1-0.3% of global electricity.[^5] This share could perhaps increase 100X, to 10-30%.[^6]   
* Earth-based solar power could increase the current economy’s energy capture by around 3000X.[^7]   
* Taken together this is 300,000X, which is \~5 OOMs.  
* Space-based solar power within this solar system could increase total energy capture a billionfold more again,[^8] which is a further \~9 OOMs.

# Combined progress

These are our estimated limits for each of the three feedback loops:
![Effective phys limits for each feedback loop](<//images.ctfassets.net/4owxfjx3z3if/LGo8bMJpIzW1DDdKqhFWA/ff377bcc4de071235fd3186d6ba2313e/Amrit_Graphs_-20.png> "Our estimates of the total room for improvement for each feedback loop before hitting effective physical limits. The limits for software and chip technology might be higher.")

What does this mean for how far an intelligence explosion could go? [Elsewhere](/research/three-types-of-intelligence-explosion), we argue that these feedback loops could lead to three particularly plausible kinds of intelligence explosion (IE):

* A *software* IE, where AI-driven software improvements alone are sufficient for rapid and accelerating AI progress.  
* An *AI-technology* IE, where AI-driven improvements in both software and chip technology are needed, but AI-driven improvements in chip production are not.  
* A *full-stack* IE, where AI-driven improvements in all of software, chip technology and chip production are needed.

To see how far these three intelligence explosions could go, we can simply add up the limits of the feedback loops to estimate the limits for each IE:

* **The software IE** could increase effective compute by \~13 OOMs, possibly more.  
* **The AI-technology IE** could increase effective compute by \~19 OOMs or more.  
* **The full-stack IE** could increase effective compute by \~24 OOMs using earth-based energy, or \~33 OOMs using all solar energy.

![Effective phys limits for each IE](<//images.ctfassets.net/4owxfjx3z3if/PUudWciYiFAOadnIeEiyg/56fcdc7a22622dd3254c744cd533ba15/Amrit_Graphs_-22.png> "Our estimates of the total room for improvement for each intelligence explosion before hitting effective physical limits. All the limits might be higher.")

[^1]:  [Epoch estimates](https://epoch.ai/data/notable-ai-models) that GPT-4 was trained with 2e25 FLOP, 10,000X more compute than GPT-2. In which case, GPT-6 would be trained with 2e29 FLOP.

[^2]:  Thanks to Carl Shulman for this argument.

[^3]:  In our context, we’re concerned with AI’s ability to do R\&D, so we can take human learning that improves their later R\&D capabilities as a baseline to compare to. 

[^4]:  Epoch’s calculation is for 16-bit precision. Lower precision would allow further gains.

[^5]:  Working [here](https://docs.google.com/spreadsheets/d/1Hkk-Gvv5JMMRgMW6yS8fpQlaMRbWMgUKjvX1cdW_r1s/edit?gid=304983116#gid=304983116). The amount of energy spent on producing advanced AI chips is much lower (see estimates [here](https://docs.google.com/spreadsheets/d/1Hkk-Gvv5JMMRgMW6yS8fpQlaMRbWMgUKjvX1cdW_r1s/edit?gid=0#gid=0)), so the bulk of the energy to consider is the energy from actually running chips.

[^6]:  This is implausible if energy supply remains static, but as per the following point, energy supply could expand dramatically. So spending 20-60% of energy on AI data centres wouldn’t mean cutting off the power to other things we use energy for today \- it would mean using most of any new energy supplies to power data centres.

[^7]:  Workings [here](https://docs.google.com/spreadsheets/d/1Hkk-Gvv5JMMRgMW6yS8fpQlaMRbWMgUKjvX1cdW_r1s/edit?gid=185396384#gid=185396384). Fusion power could increase that further but only at the cost of dramatically increasing Earth’s temperature.

[^8]:  Workings [here](https://docs.google.com/spreadsheets/d/1Hkk-Gvv5JMMRgMW6yS8fpQlaMRbWMgUKjvX1cdW_r1s/edit?gid=185396384#gid=185396384). 


How far can AI progress 1

How far can AI progress before hitting effective physical limits?

Today, AI progress is driven by humans, and the rate of progress is roughly constant over time. But once AI itself drives AI progress, this feedback loop could cause the rate of AI progress to accelerate, getting faster and faster over time. In this piece, we explain the conditions under which progress accelerates, and then evaluate whether these conditions hold for three feedback loops by which AI will improve AI: software, chip technology, and chip production. Setting human constraints aside, we argue that software might sustain accelerating progress, chip technology is likely to , and chip production is very likely to sustain accelerating progress. Overall AI progress is even more likely to accelerate, as it can involve all three feedback loops.

Tom did the original thinking; Will and then Rose helped with later thinking and writing.

Today, AI progress is driven by humans, who write software and design and produce AI chips. But in the future, more and more human work will become automated. Eventually, AI progress will be driven by AI, not humans. 

We can imagine a graph which plots capabilities over time, and shows the transition to AI-driven progress as well as the era of AI-driven progress:  
![AI-driven progress rates 1](//images.ctfassets.net/4owxfjx3z3if/2cA9q9rK1E9OxGjc2vPgrl/434e7a092a81e856e69659b2de98897a/Amrit_Graphs_-14.png)
In the graph AI progress is initially driven by human efforts (green line). Then over time AI increasingly automates the work for improving AI (orange line). Eventually, AI does almost all the work for improving AI (blue line). 

The graph raises several different factors which are relevant to understanding the landscape of the transition to AI-driven progress:  
<ol style="list-style-type: lower-roman;">
  <li><strong><a href="/research/how-suddenly-will-ai-accelerate-the-pace-of-ai-progress">Suddenness</a>:</strong> how many months or years does it take to transition from human-driven progress to AI-driven progress?  </li>
  <li><strong><a href="h/research/how-suddenly-will-ai-accelerate-the-pace-of-ai-progress#initial-speed-up-from-ai-automation">Initial speed-up</a>:</strong> how much faster is the <em>initial</em> period of AI-driven progress, compared to the final period of human-driven progress?</li>
  <li><strong>Acceleration:</strong> after the transition, will AI-driven progress <em>accelerate</em> over time?</li>
  <li><strong><a href="/research/three-types-of-intelligence-explosion#appendix-how-fast-could-ai-progress-eventually-become">Peak growth rate</a>:</strong> how fast could AI progress eventually become?</li>
  <li><strong><a href="/research/how-far-can-ai-progress-before-hitting-effective-physical-limits">Plateau</a>:</strong> how far can AI progress before hitting effective physical limits?</li>
</ol>

In this post, we’re going to discuss (iii): once AI has fully automated the work to improve AI, will AI progress accelerate over time? 

This is an important question, because if progress accelerates, we could end up with much larger and faster increases in AI capabilities. 

First, we explain the [conditions under which progress will accelerate](#under-what-conditions-does-progress-accelerate) in general terms. 

Then we evaluate [whether these conditions hold](#will-ai-progress-accelerate) for three separate feedback loops via which AI will improve AI:

* A *software* feedback loop, where AI develops better software. Software includes AI training algorithms, post-training enhancements, ways to leverage runtime compute (like o1), synthetic data, and any other non-compute improvements.  
* A *chip technology* feedback loop, where AI designs better computer chips. Chip technology includes all the cognitive research and design work done by NVIDIA, TSMC, ASML, and other semiconductor companies.  
* A *chip production* feedback loop, where AI and robots build more computer chips.

We argue that, putting aside any human actions to slow AI progress:

* **The software feedback loop** *by itself* might well be enough to sustain accelerating progress (\~50%).   
* **The chip technology feedback loop** *by itself* is probably enough to sustain accelerating progress (\~65%).   
* **The chip production feedback loop** *by itself* is very likely enough to sustain accelerating progress (\~80%). 

We conclude by arguing that [overall AI progress is even more likely to accelerate](#will-intelligence-explosions-accelerate), as it can involve all three feedback loops.

(This piece gives qualitative explanations only, and is based on forthcoming mathematical work by Basil Halperin and Thomas Houlden.)

# Under what conditions does progress accelerate?

AI improving AI is not the only example of a positive feedback loop. Before discussing AI in particular, we’ll consider the general conditions under which a positive feedback loop leads to accelerating growth.

Let’s imagine a simple system that takes inputs and produces outputs. For example, we can think of the global economy taking inputs of human labour and physical capital, and outputting goods and services.

If the outputs can be used to create more inputs, then there’s a positive feedback loop. (For example, goods and services are used to build more physical capital.)

![Feedback loop](//images.ctfassets.net/4owxfjx3z3if/6K4vOO5snnU2b89f4PByAe/255d3dcdbc5581b6053cbf5065432f29/Amrit_Graphs_-26.png)

The existence of a positive feedback loop, however, doesn’t mean that you get accelerating progress. For acceleration, each doubling of inputs must take less time than the previous doubling \- even though the volume of inputs is twice as large.[^1] 

The key issue is: **when you double the (cumulative) inputs,[^2] how many times do the outputs double?** 

If outputs double exactly once, then you have constant exponential growth – no acceleration.

![Feedback loop at times](<//images.ctfassets.net/4owxfjx3z3if/7wEFWlVKKgpPPU6R8M7u9u/8f20bde09e56b38ee27ac86ed5e1d1bf/Amrit_Graphs_-27.png> "At the earlier time, 200 units of input produced 20 units of output. 2 of these units of output are reinvested as inputs (a 10% reinvestment rate), so inputs have a growth rate of 1%. To grow at the same rate once inputs have doubled, you need to produce double the outputs (assuming a constant reinvestment rate).")

And if outputs *more than* double, you get acceleration: at the later time you have twice as many inputs (so growing them by 1% takes twice as much effort) but you have *more than* twice as many outputs.   
![Feedback loop outputs more than doubling](<//images.ctfassets.net/4owxfjx3z3if/4EWpsjXoBTnABzUm7tKKqq/65a85452d61e975cff09256764d04767/Amrit_Graphs_-28.png> "When inputs double, outputs more than double. As a result, the growth rate increases.")

In [some growth models](https://epoch.ai/blog/do-the-returns-to-software-rnd-point-towards-a-singularity), the number of times outputs double when cumulative inputs double is called *r*.[^3] If r\>1, outputs double more than once and you get accelerating growth. If r is below one, outputs less than double when you double cumulative inputs, so growth doesn’t accelerate.

For example, over the long-run course of history, doubling the number of humans has (on average) led to *more* than a doubling of output. (With twice as many humans, you can have twice as many people farming. But those people can *also* be coming up with new ideas for better farming methods, so that the amount of food produced \- and humans that can be supported in future generations \- more than doubles.) And this meant that (on average) the next doubling of humans happened more quickly, as r was greater than 1\.

| Year | Human population | Years to double |
| :---- | :---- | :---- |
| 0 CE | \~200m | 1100 |
| 1100 | \~400m | 660 |
| 1760 | \~800m | 140 |
| 1900 | \~1.6bn | 60 |
| 1960 | \~3.2bn | 40 |
| 2000 | \~6.4bn | \- |

<figcaption><div data-component="Text" data-variant="caption" class="font-serif font-[300] text-[15px] leading-[135%] italic text-center" -mt-2 mb-2>Each doubling of the <a href="https://ourworldindata.org/grapher/population?time=1400..latest&country=~OWID_WRL">human population</a> has taken less time, even though each increase was twice as large.</div></figcaption><br><br>

In domains where doubling inputs *does* lead to more than a doubling of outputs, the returns to doubling inputs will eventually start to diminish more steeply, r will drop below 1, and growth will stop accelerating — for example once the system approaches some sort of physical limit. So, in practice, accelerating growth is always just one part of an s-curve:  
![Growth curve](//images.ctfassets.net/4owxfjx3z3if/6DORM75UKtEQj7RQ3QTzlL/7b5e97ab35e127404dcf389c9c4a5c4c/Amrit_Graphs_-29.png)

# Will AI progress accelerate?

For AI progress, we’re thinking about the same accelerating growth dynamics, but with cognitive rather than economic inputs and outputs. 

Currently, humans do almost all the cognitive labour for improving AI, and so there isn’t a strong feedback loop of better AI → more cognitive labour for improving AI → better AI. But if AI replaced humans at the task of improving AI, the resulting feedback loop could lead to accelerating AI progress.  
![Effect of AI on cognitive labour](//images.ctfassets.net/4owxfjx3z3if/5fghWBq28XQcgrncPKNXq7/e0257aaed2d6b490260a10dd1b9c123a/Amrit_Graphs_-30.png)

We can define “accelerating AI progress” as “each increment of capability advancement (e.g. GPT-2 → GPT-3 or GPT-3 → GPT-4) happens more quickly than the last”. 

As in the general case of accelerating growth, the key question is how many times the outputs double for each doubling of AI inputs:[^4] i.e. what is r? If r\>1, AI progress accelerates. If r\<1, it doesn’t. 

We can get empirical evidence on r for each of the three feedback loops, and use this to guess the likelihood of acceleration. 

We argue that:

* For the **software feedback loop**, r is around 1, and possibly slightly higher.  
* For the **chip technology feedback loop**, r is comfortably above 1\.  
* For the **chip production feedback loop**, r is robustly above 1\.

Note that in this analysis we focus on *technological* likelihood only, and ignore any human bottlenecks like regulation or cultural constraints which might limit the pace of acceleration. The all-things-considered likelihood of acceleration will be lower.

## Software feedback loop

The question we care about is, when we double the human researcher population (i.e. double cognitive inputs), how many times does software double?[^5] If the answer is \>1, then progress is accelerating.

The most relevant data is on efficiency improvements:

* **Training efficiency of ImageNet.** The most relevant estimate is Epoch’s [analysis](https://arxiv.org/abs/2405.10494) of Imagenet training efficiency, which estimates r \= \~1.4.  
* **Other data on efficiency gains.** Epoch also [analysed](https://arxiv.org/abs/2405.10494) runtime efficiency in other domains:  
  * Computer chess: r \= 0.8  
  * Problem solvers: r \= 3.5  
  * Linear programming: r \= 1.1

But we don’t just care about efficiency gains – getting the same capabilities for less compute – we also care about improving capabilities – the ability to do tasks we previously couldn’t do.[^6] This increases the estimate significantly. After considering this and downwards adjustment factors like limited compute for running experiments, Davidson (forthcoming) estimates r \= 1.2, with a range of 0.4 to 3.6.

Eth and Davidson (forthcoming) review the evidence for software improvements, and argue that, after accounting for compute-experiment bottlenecks, a reasonable range for r today is 0.5 \- 2\.

**We think it is \~50% likely that the software feedback loop drives accelerating progress,** absent human bottlenecks**.**

## Chip technology feedback loop

Again we can ask the question, when we double the human researcher population, how many times does chip technology double? 

* Evidence from Davidson’s [compute-centric framework report](https://docs.google.com/document/d/1rw1pTbLi2brrEP0DcsZMAVhlKp6TKGKNUSFRkkdP_hs/edit?tab=t.0#heading=h.9vgai87ecxln) suggests that recently r \= \~5 when considering all inputs to hardware R\&D.[^7] If we assume that human cognitive labour is 60% of all hardware R\&D inputs, then r \= \~3.   
* As with software, doubling chip technology is actually better than doubling the human researcher population (as you don’t just double the quality of your hardware, you consequently train a model with higher capabilities). This might approximately double r, to \~6.[^8]  
* We should adjust downwards as r will be falling over time (as the low-hanging fruit is picked), perhaps to r \= \~4.   
* Experts argue that we are approaching fundamental limits to the current paradigm.[^9] We can make a further downward adjustment in deference to these experts, but r still seems likely to be comfortably above 1\.

**We think it is likely (very roughly \~65%) that the chip technology feedback loop drives accelerating progress**, absent human bottlenecks**.**

## Chip production feedback loop

Because software and (our definition of) chip technology are virtual, better AI straightforwardly leads to better software and chip technology. But chip production also involves physical labour, so there is a key additional step in the feedback loop: robots (and other machines and infrastructure) which can fully automate the process of building chip factories. 

With robots, a doubling of inputs would straightforwardly result in a doubling of outputs. That’s because if you have twice as many robots, they can *build* twice as many. If robots also improve robot technology at all, which seems likely,[^10] then output would more than double for each doubling of input. 

On the other hand, doubling inputs might *less* than double outputs if scarce natural resources take more and more work to extract, but historically when raw materials have become scarce this has been more than compensated for by innovation.[^11] 

(This is simply the argument given for accelerating economic growth – that once you fully automate labour the economy displays increasing returns to scale. See [Erdil and Besiroglu (2023)](https://arxiv.org/abs/2309.11690) and [Davidson (2021)](https://www.openphilanthropy.org/research/could-advanced-ai-drive-explosive-economic-growth/).)

**We think it is highly likely (very roughly \~80%) that the chip production feedback loop drives accelerating progress,** absent human bottlenecks. 

# Will intelligence explosions accelerate?

What does this mean for whether an intelligence explosion would accelerate? [Elsewhere](/research/three-types-of-intelligence-explosion), we argue that these feedback loops could lead to three particularly plausible kinds of intelligence explosion (IE):

* A *software* IE, where AI-driven software improvements alone are sufficient for rapid and accelerating AI progress.  
* An *AI-technology* IE, where AI-driven improvements in both software and chip technology are needed, but AI-driven improvements in chip production are not.  
* A *full-stack* IE, where AI-driven improvements in all of software, chip technology and chip production are needed.

When thinking about how these feedback loops combine to create intelligence explosions, we can simply add up the r values for each feedback loop involved.[^12] If the total r is greater than 1, then progress will accelerate.

This means that it’s more likely that the intelligence explosions accelerate, than that the feedback loops in isolation do.

Another important point here is that there’s some correlation between the r values: 

* If ‘extra compute’ doesn’t translate to extra intelligence, that weakens both the chip technology and chip production feedback loops.  
* If extra intelligence doesn’t actually help with research (e.g. because humans are already close to hitting non-cognitive bottlenecks in many domains), that weakens all of the feedback loops.

So lower r values for one feedback loop is correlated with lower r in others. This makes the extremes of very fast acceleration or no acceleration at all more plausible.

Overall, we think that:

* **A software IE might well accelerate over time,** because the software feedback loop *by itself* might well be enough to sustain accelerating progress (\~50%).   
* **An AI-technology IE would likely accelerate.** The chip technology feedback loop *by itself* is probably enough to sustain accelerating progress (\~65%). This means the *combination* of the software and chip technology feedback loops are likely jointly strong enough to drive accelerating progress (\~75%).   
* **A full-stack IE is highly likely to accelerate.** It’s likely that the chip production feedback loop *by itself* can sustain accelerating progress (\~80%). So in combination with the other feedback loops it is highly likely that an industrial IE would accelerate (\~90%).

*Thanks to Max Dalton, Oscar Delaney, Daniel Eth, Fin Moorhouse and the governing explosive growth seminar group for helpful comments.*

[^1]:  This assumes that there is some power law relationship between inputs and outputs.

[^2]:  We're interested in cumulative inputs because current levels of technology (the relevant meaning of ‘output’) build on the total effort that's gone into the problem so far. In areas of technology, if you double your research workforce overnight, the level of technology won’t immediately double. So we look at how many times output doubles when *cumulative inputs*, that is, inputs added up across time, double. On this definition, having 200 people work for a year constitutes twice as much cumulative input as having 100 people work for a year. 

    To get a feedback loop, the inputs also need to be accumulable: if you get more outputs you can convert them into more inputs.

[^3]:  Technically speaking, r is defined as follows: if your inputs grow exponentially at g%, your outputs will grow exponentially at g\*r%. 

[^4]:  For the software and chip technology feedback loops, AI inputs are cognitive only. For the chip production feedback loop, which includes robots, AI inputs include physical inputs too.

[^5]:   What does it mean for software to “double”? We can think about improvements in training efficiency leading to capability gains. To incorporate such improvements precisely, we can translate them into an “equivalent” runtime efficiency gain. If some capabilities improvement increases our AI’s ability to make research progress by the same amount as doubling efficiency, we’ll say that capabilities improvement “doubled” software. In other words, all software improvements will be measured in the same units, where a “doubling of software” corresponds to any improvement that’s “as useful” as doubling runtime efficiency. 

[^6]:  If we double the quality of our software, our AI models will be more efficient by some factor, but they will also be more capable.

[^7]:  This estimates r for FLOP/$, which includes both improving the energy efficiency of chips (FLOP/Joule) and making chips cheaper to make and run ($/chip and $/Joule). 

[^8]:  [Davidson estimates](https://docs.google.com/document/d/1DZy1qgSal2xwDRR0wOPBroYE_RDV1_2vvhwVz4dxCVc/edit?tab=t.0#bookmark=id.eqgufka8idwl) that doubling training size doubles output-per-FLOP at runtime between once and twice. If we are conservative and take the lower number, then doubling hardware 1\) doubles your runtime FLOP and 2\) doubles your output-per-FLOP at runtime, i.e. it doubles your total output twice. 

[^9]:  For instance, [here](https://ieeexplore.ieee.org/abstract/document/7878935) and [here](https://royalsocietypublishing.org/doi/full/10.1098/rsta.2019.0061).

[^10]:  More robots can build more chips which can be used to run more AI models to work on improving robot technology. So having more robots seems likely to lead to improved robot technology.

[^11]:  This will eventually cease once all the materials on earth or the solar system are used up, but there is a lot of room for acceleration before that point.

[^12]:  See Houlden, Thomas, presenter. [Is Automating AI Research Enough for a Singularity?](https://drive.google.com/file/d/1jPG-vSLkje3RmjLk2Yz__ObsGGAGEfHM/view?usp=sharing) By Tom Davidson, Basil Halperin, Thomas Houlden, and Anton Korinek. 15th Oxford Workshop on Global Priorities, December 2024\. 

Will AI progress accelerate? - main

Will AI progress accelerate once AI drives AI progress? 

Today, human efforts drive AI progress. But at some point, all AI progress will be driven by AI. This piece analyzes the transition from human-driven to AI-driven progress. Firstly, we discuss the suddenness of the transition: how many months or years does it take to transition from human to AI-driven progress? Secondly, we turn to the size of the initial speed-up: how much faster is the initial period of AI-driven progress compared to the final period of human-driven progress? We argue that the transition from human-driven to AI-driven progress could be very sudden and big for improvements to AI software, but less so for improvements to chip technology and chip production.

Today, AI progress is driven by humans, who write software and design and produce AI chips. But in the future, more and more human work will become automated. Eventually, AI progress will be driven by AI, not humans. 

What might the transition from human-driven to AI-driven progress look like? We can imagine a graph which plots AI capabilities over time:  
![AI-driven progress rates 1](//images.ctfassets.net/4owxfjx3z3if/2cA9q9rK1E9OxGjc2vPgrl/434e7a092a81e856e69659b2de98897a/Amrit_Graphs_-14.png)
In the graph, AI progress is initially driven by human efforts (green line). Then, over time, AI increasingly automates the work of improving AI (orange line). Eventually, AI does almost all the work of improving AI (blue line). There is a continuous transition from today’s era of human-driven progress to a future era of AI-driven progress.

The graph raises several different factors that are relevant to understanding the landscape of the transition to AI-driven progress:  
(i) [**Suddenness**](#suddenness)**:** how many months or years does it take to transition from human to AI-driven progress?   
(ii) [**Initial speed-up**](#initial-speed-up-from-ai-automation)**:** how much faster is the *initial* period of AI-driven progress compared to the final period of human-driven progress?   
(iii) **Acceleration:** after the transition, will AI-driven progress *accelerate* over time?   
(iv) **Peak growth rate:** how fast could AI progress eventually become?  
(v) **Plateau:** how far can AI progress before hitting effective physical limits?

(i) and (ii) happen during the transition period; (iii)-(v) happen during the era of AI-driven growth.

:::textbox
How should we define when the transition period starts and ends?  

We can think about this period in terms of the elasticity of AI R\&D output from humans and AI for each feedback loop: how much the marginal pace of progress increases if you increase the population of humans or AIs by 1%. We can say that the transition period begins when AI elasticity is first \>1/10 that of human labor and ends when it’s \>9/10. The specific start and end points are arbitrary, but for any such choice, there’s a wide range of possibilities for: 
* How sudden the transition period could be: it could take anything from months to decades. 
* How fast the initial speed-up from AI-driven progress could be: the pace of AI progress could increase by 2X or 100X over the course of the transition.
:::

This post explores the dynamics of this transition by analyzing (i) and (ii). 

The transition will likely play out differently in different areas of AI progress. We think it’s helpful to distinguish between three separate feedback loops in which AI improves AI:

* A *software* feedback loop, where AI develops better software. Software includes AI training algorithms, post-training enhancements, ways to leverage runtime compute (like o1), synthetic data, and any other non-compute improvements.  
* A *chip technology* feedback loop, where AI designs better computer chips. Chip technology includes all the cognitive research and design work done by NVIDIA, TSMC, ASML, and other semiconductor companies.  
* A *chip production* feedback loop, where AI and robots build more computer chips.

We argue that the transition from human-driven to AI-driven progress has the potential to be the most sudden and biggest for software, followed by chip technology and then chip production.

# Suddenness

How sudden could the transition from human-driven to AI-driven progress be?
![Suddenness of AI acceleration](//images.ctfassets.net/4owxfjx3z3if/5dfBtk2RVrJ3Yw8BFwMRaw/4b172dac411dda8faeb08fb71881e42c/Amrit_Graphs_-16.png)

## Time lags

One reason things are unlikely to be extremely sudden is that all the feedback loops have inbuilt time lags.

![Time lags](//images.ctfassets.net/4owxfjx3z3if/5HP847xI9hNoUbBy8pFsJj/5c9f5b7e4ee63dcc37e13208d83973c3/Amrit_Graphs_-32.png)

Software has the shortest time lags, and chip production probably has the longest: 

* It takes [\~3 months](https://epoch.ai/data/notable-ai-models#training-time-growth) to train new SOTA AI, which is the main time lag in the software feedback loop (though [post-training enhancements](https://epoch.ai/blog/ai-capabilities-can-be-significantly-improved-without-expensive-retraining), such as fine-tuning, take *much* less time).  
* For the chip technology feedback loop, it’s also necessary to integrate new technology into chip factories and print a stock of new chips, which would take many months.[^1]  
* It takes years to build new fabs, which would be needed for the chip production feedback loop.

![Time lag timeline](<//images.ctfassets.net/4owxfjx3z3if/5jdWR4L2h4mzrGMZYRXkYW/040db1da8f50b7f8b2060849900fa64d/Amrit_Graphs_-33.png> "We can subdivide the three feedback loops further and visualize the duration of the time lags for each subdivision.")

## Gradual automation

Another reason is that tasks for each feedback loop won’t be automated all at once, but gradually. The more gradual the automation, the less sudden the transition.

Automation will be more gradual where:

* The tasks to automate are varied, so we gradually automate more and more of them (rather than automating all tasks at once).  
* There isn’t already data on the tasks, so data needs to be collected in the physical world.  
* The tasks take a long time, and so gathering data takes longer. Tasks in the physical world will tend to take longer and sometimes much longer than virtual tasks.

These considerations suggest that software automation will be more sudden than the automation of chip technology and chip production:

* The tasks are less wide-ranging.  
* There’s much more online data (and easy access to data in the heads of human experts who already work at the relevant companies).  
* The tasks are entirely virtual. 

The automation of chip technology would likely be less sudden than software but more sudden than chip production:

* According to our definition, chip technology tasks are also entirely virtual, in contrast to chip production tasks, which are predominantly physical.  
* But there’s much less online data for chip technology than for software, and the tasks are more varied.

## Rapid AI progress

The software feedback loop is likely to happen first.[^2] If the software feedback loop leads to a large and rapid increase in capabilities, then automation of other areas could happen very suddenly, and the time lags in other areas (e.g. time to print chips) could be reduced. That could mean the chip technology and chip production feedback loops begin very suddenly.

# Initial speed-up from AI automation

As well as considering how sudden the transition from human-driven progress to AI-driven progress might be in calendar time, we can also ask how much faster AI-driven progress will initially be compared to human-driven growth.  
![Initial speedup](//images.ctfassets.net/4owxfjx3z3if/4JKspnAhcpsdYaHnxtnwSO/f65774f1e582139ffb95d4cadff85835/Amrit_Graphs_-17.png)

For the **software feedback loop**, the initial speed-up from automation could be very large. Currently, frontier labs have only a few hundred people working on software R\&D, but once AI can automate these jobs, that number could [quickly rise to millions](https://www.planned-obsolescence.org/continuous-doesnt-mean-slow/) of human-equivalent software R\&D workers. This gain won’t be bottlenecked by physical constraints like constructing fabs and printing chips, though it would be limited by the time taken to run AI experiments and train new models. Estimates of the size of the initial speed-up from automating software vary from **2x to 20x** faster progress than pre-automation, though they are very speculative.[^3]

The initial speed-up for the **chip technology feedback loop** is highly uncertain. The innovations that could be incorporated most quickly are probably in chip design. Interviewees from [this chip design report](https://docs.google.com/document/d/1DaaisT7cZDwuLPzd4y468Up2tZWJ0FII0d-aw8-J1YM/edit?tab=t.0#heading=h.fa42z66cy9cr) were asked how much AI might speed up progress. They thought that:

* A narrow AI chip designer that thought 100x as fast as human designers might increase FLOP/$ by 1.5-4X over the next year, increasing the pace of hardware progress by **2X-4X**.   
* A ‘perfect’ GPU design might increase FLOP/$ by 10X-100X or more. If this happened very quickly over 2 years, that would be a speed-up of **4X-14X**.

| Capability level | Time period | FLOP/$ increase | Increase in the pace of hardware progress  |
| :---- | ----- | ----- | ----- |
| Narrow AI chip designer, 100X human speed | 1 year | 1.5-4X | **2X-4X** |
| A perfect GPU design | 2 years | 10X-100X | **4X-14X** |

Including other types of chip technology innovation beyond chip design would increase this factor, but their effect would be limited by the time it takes to make new machines and fabs.

The **chip production feedback loop** would probably have the smallest initial speed-up:

* It takes years to build new fabs, so initial gains will be limited.  
* Just before this feedback loop kicks in, the growth in the number of chips being used for training might already be very high, which would mean that it would be hard to increase that growth rate much further.   
  * Specifically, this pre-chip-production-feedback growth might happen via an unsustainable route: we might simply increase the proportion of chips used in AI training. But you can only increase this percentage up to 100%, and beyond that point, you need to build more fabs. 

# Conclusion

There will really be [multiple gradual transitions](https://www.lesswrong.com/posts/BTApNmv7s6RTGxeP4/cyborg-periods-there-will-be-multiple-ai-transitions) from human-driven to AI-driven progress as different domains of progress transition at different times. 

The transition from human-driven to AI-driven progress has the potential to be the most sudden and biggest for software (especially for post-training enhancements), followed by chip technology and then chip production.

*Thanks to Max Dalton, Oscar Delaney and the governing explosive growth seminar group for helpful comments.*  

[^1]:  New chip designs can begin printing within weeks. But it would take longer to print significant quantities of chips. As a rough estimate:

[^2]:  See here for this argument.

[^3]:  Leibowich, Jurkovic and Davidson (forthcoming), ‘AI accelerating the Pace of AI Progress: Researcher Perspectives’. [Aschenbrenner estimates](https://situational-awareness.ai/from-agi-to-superintelligence/) a 10x acceleration in algorithmic improvement during an intelligence explosion.

How suddenly will AI accelerate the pace of AI progress?

There have been recent discussions of centralizing western AGI development, for instance through a Manhattan Project for AI. But there has been little analysis of whether centralizing would actually be a good idea. In this piece, we explore the strategic implications of having one project instead of several. We think that it’s very unclear whether centralizing would be good or bad overall. We tentatively guess that centralizing would be bad because it would increase risks from power concentration. We argue that future work should focus on increasing the expected quality of either a centralized or multiple projects, rather than increasing the likelihood of a centralized project.

Tom Davidson did the original thinking; Rose Hadshar helped with later thinking, structure and writing. 

Should there be just one western AGI project?

Once AI fully automates AI R&D, there might be a period of fast and accelerating software progress – a software intelligence explosion. One objection to this is that it takes a long time to train AI systems from scratch. Would retraining each new generation of AI stop progress accelerating during a software intelligence explosion? If not, would it significantly delay a software intelligence explosion? This post investigates this objection. Using a theoretical analysis and some spreadsheet models, I argue that retraining won’t stop software progress from accelerating over time, but does make it less likely that a software intelligence explosion happens in shorter than 10 months.

# Summary

Once AI fully automates AI R\&D, there might be a period of fast and accelerating software progress – a software intelligence explosion (SIE).

One objection to this is that it takes a long time to train SOTA AI systems from scratch. Would retraining each new generation of AIs stop progress accelerating during an SIE? If not, would it significantly *delay* an SIE? This post investigates this objection. 

Here are my tentative bottom lines:

* **Retraining won’t stop software progress from accelerating over time.**   
  * Suppose that, ignoring the need for retraining, software progress would accelerate over time due to the AI-improving-AI feedback loop.  
  * A simple theoretical analysis suggests that software progress will *still* accelerate once you account for retraining. Retraining won’t block the SIE.   
* **Retraining will cause software progress to accelerate somewhat more slowly.**  
  * But that same analysis suggests that your software progress will accelerate *more slowly.*    
  * Quantitatively, the effect is surprisingly small – acceleration only takes \~20% longer.  
  * However, if acceleration was going to be extremely fast, retraining slows things down by more.   
* **Retraining means that we’re unlikely to get an SIE in \<10 months**, unless either training times become shorter before the SIE begins or improvements in runtime efficiency and post-training enhancements are large.  
  * Today it takes [\~3 months](https://epoch.ai/data/notable-ai-models#training-time-growth) to train a SOTA AI system.   
  * As a baseline, we can assume that the first AI to fully automate AI R\&D will also take 3 months to train.  
  * With this assumption, simple models of the SIE suggest that you’re unlikely to complete the SIE within 10 months. (In the model, the SIE is “completed” when AI capabilities approach infinity – of course in reality they would reach some other limit sooner).   
  * BUT if by the time AI R\&D is fully automated, SOTA training runs have already shortened to one month or less, then completing an SIE in less than 10 months is much more plausible.  
  * And improvements in runtime efficiency and other post-training enhancements (which I’m not modelling in my analysis) could potentially allow very fast takeoff without needing to retrain from scratch. 

How did I reach these tentative conclusions? 

**Theoretical analysis**  
First, I conducted a very basic [theoretical analysis](#theoretical-analysis) on the effect of retraining. I took a standard [semi-endogenous growth model](https://en.wikipedia.org/wiki/Jones_model) of tech development, and used empirical estimates for the diminishing returns to software progress. This is the simplest model I know of to estimate the dynamics of an SIE – and by default it doesn’t account for retraining. 

To understand how fast software progress[^1] accelerates, we can ask: how many times must software double before the *pace* of software progress doubles? This is a measure of how quickly software progress accelerates: lower numbers mean faster acceleration.

Without including retraining, my median parameters imply that once AI R\&D is fully automated,  software must double **\~five** times before the *pace* of software progress doubles. (There’s large uncertainty.)[^2] So progress accelerates gradually. 

Accounting for retraining increases the number from five to **six**. Training runs get progressively shorter over time, and the SIE still accelerates but slightly more slowly. (See [below](#theoretical-analysis) for more explanation.)  

![Doubling time 1](//images.ctfassets.net/4owxfjx3z3if/4W6aXQwGusFtLR3C8QPOOl/3ab769fbef8723f4a57f5b390e6a757d/Amrit_Graphs_-34.png)

In a very aggressive scenario where software must only double **once** before the pace of progress doubles, retraining makes a big difference by increasing this to **twice.** So retraining makes a bigger difference to these aggressive scenarios, making them significantly less extreme.

**Simple spreadsheet models**  
Second, I made very simple [spreadsheet models](#toy-model-of-the-software-intelligence-explosion) of an SIE – again based on semi-endogenous growth models – one without retraining and one with retraining. Both sheets use the same parameters (other than whether to include retraining) and both calculate the time between the AI R\&D being automated and AI capabilities going to infinity. I assumed that AI algorithms become 2X as efficient every month – that’s about \~10X faster progress than today.

Results:

* If the initial training run lasts 100 days:  
  * The SIE takes about **3X** longer with retraining  
  * The SIE lasts \>10 months unless very extreme parameter values are used.  
* If the initial training run lasts 30 days:  
  * The SIE takes about **2X** longer with retraining.  
  * The SIE lasts \>7 months unless very extreme parameter values are used.

(These slowdowns are longer than the slowdown predicted by the theoretical analysis because the theoretical analysis assumes that training runs get gradually shorter as the SIE gets closer, so are already very short at the point at which the spreadsheet model begins, and so have less of a slowing effect. See [below](#synthesis) for details.) 

# Theoretical analysis

Let’s estimate how quickly software progress will speed up *excluding* and *including* accounting for re-training. 

**Excluding retraining**  
We can ask: how many times must software double before the *pace* of software progress doubles? This is a measure of how quickly software progress accelerates: lower numbers mean faster acceleration.

In a separate piece (forthcoming), I estimate that the answer is very roughly 5\. It could be as low as 2, or higher than 10\. 

**Including retraining**  
When we were excluding retraining, we were implicitly modelling AI progress as simply consisting of one step: improving AI software. Doubling the pace of AI progress simply involved doubling the speed of that step.  
![Training loop 1](//images.ctfassets.net/4owxfjx3z3if/1w0Jug7yIGcsloBfIcHJD1/44c8f1008d0095053c6f82f37925fb4d/Amrit_Graphs_-03.png)
Now we add a second step: retraining AI from scratch.

![Training loop 2](//images.ctfassets.net/4owxfjx3z3if/4VRiLjtpbu5LVxp8KbAyoB/cd5ceb1f49c78560bbe79d45f021d490/Amrit_Graphs_-04.png)
<figcaption><div data-component="Text" data-variant="caption" class="font-serif font-[300] text-[15px] leading-[135%] italic text-center" -mt-2 mb-2>We expect the ‘improving AI software’ step take longer than the ‘retraining’ step. Today training runs currently take <a href="https://epoch.ai/data/notable-ai-models#training-time-growth">~3 months</a> but the gap between releasing frontier models is closer to <a href="https://docs.google.com/spreadsheets/d/1v_IFQ2okXL2ivUqzz4s7ACKWpiqV71pHvuVtfNAx0dk/edit?gid=0#gid=0">1-2 years</a>. For this basic theoretical model, I’m assuming that the steps are serial. In reality, while you’re training the next model, you can still be making software progress, so the steps would overlap.</div></figcaption><br><br>

Now to doubling the pace of overall AI capabilities improvement requires doubling the speed of both steps.[^4] We must not just double the pace of software progress (as discussed above, this might take \~5 software doublings) but we must also halve the time taken to retrain AI.  
![Doubling time 2](//images.ctfassets.net/4owxfjx3z3if/5BZahwUv6BYH8yy1t6i804/1bd6aec08176d49cba2adeff786540f1/Amrit_Graphs_-35.png)  
How can we halve the time to retrain AI? We can do this by 'spending' a 2X gain in training efficiency to make training faster rather than using it to make models smarter. That is, normally efficiency gains are used to train a model with more ‘effective training compute’, resulting in a smarter model. But instead we could use half as much training compute and train an equally smart model. This would allow us to train the model in half as much time (assuming we can parallelise training sufficiently – if we can’t, retraining could slow down an SIE much more). 

So rather than doubling the pace of software development every \~5 software doublings, the pace will only double every \~6 doublings. The first 5 double the pace of software progress. The 6th isn't actually used to improve AI capabilities at all, but is just used to make training twice as fast.[^5]

![Retraining half time flowchart](//images.ctfassets.net/4owxfjx3z3if/5ZpedaSItepGPwgAJJGBOK/8004fb3c281235501c46081183016b24/Amrit_Graphs_-05.png)

All of this assumes my best-guess median parameters for accelerating software progress, in which it takes 5 doublings to double the pace of software progress. In a very aggressive scenario software would only need to double **once** before the pace of progress doubles, and retraining would increase this to **twice.** This makes the scenario significantly less aggressive. So accounting for retraining makes a small difference to the median scenario, but a significant difference to very aggressive scenarios. 

The bottom line here is:

* If software progress was going to accelerate before taking retraining into account, it will still accelerate when retraining is accounted for.  
* But software progress will accelerate *more slowly* than you would naively think.  
* The slow down here is not very big unless the pace of acceleration would have otherwise been very rapid. 

# Toy model of the software intelligence explosion

I model a scenario where:

* AGI has just been trained using 1e29 FLOP.  
* The training run took 100 days.  
* The compute used for retraining is then reinvested to further improve capabilities.

In the model [with continuous retraining](#software-explosion-with-continuous-retraining---model-explanation), half of this compute is used to improve AI software (by finding better algorithms), and half is used to continually retrain the AI.[^6]

In the model [without continuous retraining](#software-explosion-with-continuous-retraining---model-explanation), all of the compute is used to improve AI software (better algorithms can be applied instantaneously).

## Software explosion with continuous retraining \- model explanation

In this [model](https://docs.google.com/spreadsheets/d/1BsFQErjhQXZXiBCm1pVVmPzEd2XYln-to6z29ZwHrUQ/edit#gid=1905930355), if all of the training compute were used to find better algorithms, cumulative software research would double in \~30 days. In fact I assume that only half of the training compute is used for this, so it would actually take \~60 days. 

As training algorithms improve, the effectiveness of continuous training also improves. So initially each FLOP spent on training adds 1 effective training FLOP to the cumulative training run size, but after 60 days each FLOP adds 4 effective training FLOP to the cumulative training run size. There's little change to cumulative training run size over the first 60 days, because it's hard to make a big difference to the 100 days initially spent on training. After \~100 days, the effective training FLOP has increased more than 3X, because of the improved efficiency with which additional FLOP are increasing effective training FLOP.

| Days after training AGI | Cumulated effective training FLOP |
| :---- | :---- |
| 60 days | 1.6E+29 |
| 100 days | 2.5E+29 |
| 200 days | 1.3E+30 |

The productivity of the AI workforce increases in lock-step with the total cumulative effective training FLOP.[^7] So 100 days in the AIs are \>3X more productive, and they improve training algorithms faster, which in turn allows them to increase the cumulative effective training FLOP faster, which in turns makes the AIs more productive. 

Better training algorithms → faster increase in cumulative effective training FLOP → faster increase in AI productivity → faster discovering better training algorithms → faster increase in cumulative effective training FLOP.

After 200 days effective training FLOP has increased by 100X and the AI is 100X more productive.   
![Effective training FLOP per FLOP 1](//images.ctfassets.net/4owxfjx3z3if/1IPEO82ds3P6YyE2lxZMoP/89a2e49c10d52e1ab81831d1beabb70f/Amrit_Graphs_-01.png)

Ege Erdil has made a [version of this model](https://colab.research.google.com/drive/19-ZRsBv5PtS7i7DqB5RjoHFzxyG2LXp3?usp=sharing) where you must retrain new models from scratch.

## Software explosion without retraining – model explanation

This [sheet](https://docs.google.com/spreadsheets/d/1gW8uVQZQxbJ-Y95lXbhXxM7u6sDN9skXSk16f3YoChY/edit#gid=0) models the same SIE dynamics without retraining. Here, algorithmic innovations are applied instantaneously to improve the productivity of AI labour. This is appropriate for things like better prompting, better tool use, Auto-GPT, better inference efficiency via quantization, and perhaps also fine-tuning.   
![Effective training FLOP per FLOP 2](//images.ctfassets.net/4owxfjx3z3if/74VYGN0b8zP2dilzmjMmZf/8be3f487f14bba26f71c028721be34aa/Amrit_Graphs_-06.png)

## Comparing the two scenarios

I used the same parameters in each sheet. In both cases, if you took the amount of cognitive labour available on the first day and held it fixed, it would take 35 days to double software.

Without retraining, it takes \~5 months for software to go to infinity. With retraining, it takes \~16 months. This is an increase of about **3X**. 

Other parameter values gave similar results:

|  __Software must double N times before the pace of improving AI software doubles__[^8]  | __N \= 4.3__ | __N \= 3__ | __N \= 2__ | __N \= 1.5__ |
| :---- | :---- | :---- | :---- | :---- |
| Initial software doubling time in months (holding cognitive labour fixed) | 1.4 | 1.2 | 1.1 | 0.5 |
| Months for software to go to infinity **without retraining** | 8 | 5 | 3 | 1.3 |
| Months for software to go to infinity **with retraining** | 27 | 16 | 10 | 4.6 |
| Multiplier that retraining applies on months to infinity | 3.4 | 3.2 | 3 | 3.5 |

# Synthesis

Why are these slowdowns so much bigger than predicted in the theoretical analysis (which suggested retraining would only slow the number of doublings needed to double the pace progress from 5 to 6)?  

My toy model assumes that training runs still take 3 months when we fully automate AI R\&D and software progress is 10X faster.

But my theoretical model assumes that we gradually make training runs shorter as software progress speeds up. So my theoretical model would predict that training runs would have already become significantly shorter by the time software progress is 10X faster. And if training runs are already shorter, they will have less of a slowing effect on an SIE.

The toy model is more valid if we’ll suddenly and unexpectedly automate AI R\&D, without shortening training times in advance. The theoretical analysis is more valid if you think we’ll gradually automate AI R\&D, software progress will gradually speed up, and we’ll make our training runs gradually shorter and shorter as we do so. 

So the more you expect sudden jumps, as in the toy model, the more you should expect retraining to slow the SIE. The more you expect gradual progress, the less of an impact retraining will have on the SIE.

Redoing the toy model with a shorter initial training run time is consistent with this story. We find that the results are more in line with the theoretical model. 

Below are results with using a 30-day initial training run time instead of 100 days (and all other parameters the same):

| Each time cumulative software research doubles, software capabilities double… | __N \= 4.3__ | __N \= 3__ | __N \= 1.5__ |
| :---- | :---- | :---- | :---- |
| Months for software to go to infinity **without retraining** | 8 | 5 | 1.3 |
| Months for software to go to infinity **with retraining** | 16 | 10 | 3.2 |
| Multiplier on duration (with shorter initial training time) | 2 | 2 | 2.5 |

*Thanks to Max Dalton, Daniel Eth and Rose Hadshar for review.*

[^1]:  Software includes AI training algorithms, post-training enhancements, ways to leverage runtime compute (like o1 and o3), synthetic data, and any other non-compute improvements.

[^2]:  What does it mean for “software” to double? A simple way of thinking about this is that software doubles when you can run twice as many copies of your AI with the same compute. But software improvements don’t just improve runtime efficiency: they also improve capabilities. To incorporate these improvements, we’ll ultimately need to make some speculative assumptions about how to translate capability improvements into an equivalently-useful runtime efficiency improvement. 

[^4]:  Alternatively, you could more-than-double the speed of one step to compensate for less-than-doubling the other. The net effect needs to be a doubling of the combined speed of both steps. 

[^5]:  I’m assuming here that each “software” doubling doubles training efficiency. In other words, I’m ignoring software improvements that don’t apply to pre-training. 

[^6]:  Probably the optimal strategy isn’t half and half, so we should expect progress to actually be somewhat faster than this model suggests.

[^7]:  This is a simple assumption. A more general functional form is to assume that increasing effective training compute by a factor of 2 increases AI output by a factor of 2^v, v\>0. My model assumes v=1.

[^8]:  [This sheet](https://docs.google.com/spreadsheets/d/14SgV6V_cLjvMZCHYfZhoGZredetJP_pBh9amNdz57NQ/edit) calculates N from the parameters of the semi-endogenous growth model used in the sheets.

Will the need to retrain AI models from scratch block a software intelligence explosion?

If there is an international project to build artificial general intelligence (“AGI”), how should it be designed? Existing scholarship has looked to historical models for inspiration, often suggesting the Manhattan Project or CERN as the closest analogues. But AGI is a fundamentally general-purpose technology, and is likely to be used primarily for commercial purposes rather than military or scientific ones. 

This report presents an under-discussed alternative: Intelsat, an international organization founded to establish and own the global satellite communications system. We show that Intelsat is proof of concept that a multilateral project to build a commercially and strategically important technology is possible and can achieve intended objectives—providing major benefits to both the US and its allies compared to the US acting alone. We conclude that ‘Intelsat for AGI’ is a valuable complement to existing models of AGI governance. 

Thanks to Max Dalton, Joao Fabiano, John Krige, Joseph N. Pelton, the governing explosive growth seminar group and several anonymous reviewers for thoughtful criticism and feedback.

Intelsat part 1

Intelsat part 2

Intelsat as a model for international AGI governance

# How should we navigate explosive AI progress?

homeTitleSection

#### We are a research nonprofit focused on how to prepare for rapid AI-driven technological change. 

We try to understand both the challenges and opportunities this change will bring, from extreme concentration of power and new weapons of mass destruction to economic abundance and enhanced collective decision-making. We ask: how can we reach as good an outcome as possible? And what can we do, now, to prepare?

homeSplashSection

AI systems might soon be much more capable than humans, quickly leading to rapid technological development, and many follow-on opportunities and challenges.

So not only do we need to align AI systems: we also need to avoid extreme concentrations of power, figure out how AI can improve rather than degrade our collective reasoning and coordination, work out what rights to give digital minds, manage races to grab off-world resources, and more.

We study challenges like these and how to navigate them.

Right now, we’re thinking about how to avoid AI-enabled coups, about which applications of AI can most help with other challenges, and about what an ideal post-AGI future would look like. We’re also interested in the rights of digital beings, AI persuasion capabilities, space governance, and the structure of projects to build AGI.

How should we navigate explosive AI progress?

We are a research nonprofit focused on how to prepare for rapid AI-driven technological change.