Abstract

0 Summary

0.0 The claim

1. Humanity will almost certainly go extinct in the next million years.
2. Under Darwinian pressures, intelligent life will spread throughout the stars and rapidly evolve toward maximal reproductive fitness.
3. Through moral reflection, intelligent life will reliably be driven to pursue some specific “higher” (non-reproductive) goal, such as maximizing the happiness of all creatures.
4. The choices of intelligent life are deeply, fundamentally uncertain. It will at no point be predictable what intelligent beings will choose to do in the following 1000 years.
5. It is possible to stabilize many features of society for millions or trillions of years. But it is possible to stabilize them into many different shapes — so civilization’s long-term behavior is contingent on what happens early on.

1. It will be possible to preserve highly nuanced specifications of values and goals far into the future, without losing any information.
2. With sufficient investments, it will be feasible to develop AGI-based institutions that (with high probability) competently and faithfully pursue any such values until an external source stops them, or until the values in question imply that they should stop.
3. If a large majority of the world’s economic and military powers agreed to set-up such an institution, and bestowed it with the power to defend itself against external threats, that institution could pursue its agenda for at least millions of years (and perhaps for trillions).

• The possibility of stable institutions could pose an existential risk, if they implemented poorly chosen and insufficiently flexible values.
• On the other hand, if we want humane values or institutions such as liberal democracy to survive in the long-run, some types of stability may be crucial for preserving them.
• The possibility of ultra-stable institutions pursuing any of a wide variety of values, and the seeming generality of the methods that underlie them, suggest that significant influence over the long-run future is possible. This should inspire careful reflection on how to make it as good as possible.

0.1 Preserving information

• Ensure that future civilisational decisions are made democratically.
• Enforce a ban on certain weapons of mass destruction (WMD)
• Make sure that reverence is paid to some particular religion.
• Always do what some particular group of humans would have wanted.

0.2 Executing intentions

• Will it be possible to construct AGI systems that (with high probability) are aligned with the intended values?
• Will such systems stay aligned even over long periods of time?

0.2.1 Aligning AGI

• With sufficient understanding of how to induce particular goals, AI systems could be designed to more single-mindedly optimize for the intended goal, whereas most humans will always have some other desires, e.g. survival, status, or sexuality.
• AI behavior can be thoroughly tested in numerous simulated situations, including high-stakes situations designed to elicit problematic behavior.
• AI systems could be designed for interpretability, perhaps allowing developers and supervisors to directly read their thoughts, and to directly understand how it would behave in a wide class of scenarios.

0.2.2 Preventing drift

• Whenever there’s uncertainty about what to do in a novel situation, or a high-stakes decision needs to be made, the institution could boot-up a completely-reset version of an AI system (or a brain emulation) that acts according to the original values.

  • This system will have had no previous chance of value-drift, and so only needs to be informed about anything that is a prerequisite for judging the situation.
  • In order to reduce contingency from how these prerequisites are learned, the institution could bring back multiple copies and inform them in different ways — and also let some of the copies opine on how to inform the other copies. And then have them all discuss what the right option is.
• AI systems designed to execute particular tasks could be motivated to do whatever the more thorough process would recommend. They could be extremely well-tested on the types of situations that most frequently come up while performing that task.

  • For any tasks that didn’t require high context over a long period of time, they could be frequently reset back to a well-tested state.
  • If the task did require a larger amount of context over a longer period of time, they could be supervised and frequently re-tested by other AI systems with less context. These may not be able to correctly identify the value of the supervisee’s every action, but they could prevent the supervisee from performing any catastrophic actions. (Especially with access to transparency tools that allow for effective mind-reading.)
• Value drift that is effectively random could be eliminated by having a large number of AI systems with slightly-different backgrounds make an independent judgment about what the right decision is, and take the majority vote.

• endorse options that less-informed versions of themselves disagree strongly with,
• have irresolvable disagreements with AI systems which have somewhat different previous experiences,
• exhibit thought-patterns (detected with transparency tools) that show doubt about the institutions’ original principles.

• An extreme version of this would be to prevent all reasoning that could plausibly lead to value-drift, halting progress in philosophy.

  • It doesn’t seem impossible that all philosophically ambitious institutions would eventually converge to some very similar set of behavior, from a very wide range of starting points. This might be the case if some form of moral realism holds, or perhaps if something like Evidential Cooperation in Large Worlds works. If this were the case, our claim that it’s feasible to stabilize many different value-systems would be false for philosophically ambitious institutions. It would only apply to institutions that refused to conduct some philosophical investigations. (Which we hope wouldn’t be very common.)
• A further extreme would be for the institution to also halt technological progress and societal progress in general (insofar as it had the power to do that) to avoid any situation where the original values can’t give an unambiguous judgment.

  • This would largely eliminate the issue motivating this subsection — that continual learning could lead to value drift — since complete stagnation wouldn’t require much in the way of continual learning.
  • But depending on when technological progress was halted, this could limit the institution's ability to survive in other ways, e.g. by preventing it from leaving Earth before its doom.

0.3 Preventing disruption

• prevent others from building weapons of mass destruction,
• prevent others from building a competitive institution of similar economic or military strength, and
• prevent others from leaving the institution’s domain by colonizing uninhabited parts of space.

0.4 Some things we don’t argue for

• technological maturity1

  (preventing new technologies from shaking things up),
• human immortality (reducing drift from generational changes),
• the ability to cheaply and stably align AGI systems with any goal, and
• such AI systems being equally good at pursuing instrumental goals regardless of what terminal goals they have. (Thereby mostly eliminating the tendency for some values to outcompete others, c.f. decoupling deliberation from competition.)

0.5 Structure of the document

• In section 1, we clarify our assumptions about AGI, discuss what we mean by “stability” and “lock-in”, and clarify how confident we are that societies could be made how long-lasting.
• In section 2, we give some remarks on the desirability and probability of highly stable institutions.
• In section 3, we go through some sources of instability from the past, and how some of them seem less likely in the future.
• In section 4-8, we go through the arguments sketched in the summary in more detail.

1 To be more precise

1.1 What do we assume about “AGI”?

A note on anthropomorphism:

• it is clearly possible to build societies and institutions out of agent-like beings (as demonstrated by human society),
• the analogy to humans helps clarify how agents might act and interact with each other, and
• even without the analogy to humans, agents are easier to reason about since we can use the intentional stance and assume that they’ll do (easy) things that are helpful to accomplish some goal (instead of having to reason about their internal structure).

1.2 What do we mean by “lock-in”?

• Before that time, there is notable uncertainty about how that feature will turn out in the long-run.
• After that time, the uncertainty has been significantly reduced. In particular, there is a much smaller set of possibilities that have non-trivial probabilities.

• Before that time, there are many different values that might end up being adopted by powerful actors.
• After that time, all powerful actors hold values from a much-reduced subset of the original possibilities, and it is very unlikely that any powerful actor in that civilization will adopt values from outside that subset.

1.3 How confident are we that stability is feasible, for how long?

• Among the arguments in the summary, components 0.1 and 0.3 seem solid.
• Component 0.2 (both 0.2.1 and 0.2.2) seems somewhat less certain. There are many arguments for why AI systems could be made with more stable values than humans, and we don’t know any particular, credible objections to total stability. But we don’t have arguments that knowably identify and exclude all possible sources of drift.
• We’re especially confident that it would be possible to build a civilization that was stable for millions of years on Earth, since this would be compatible with complete stagnation. If a civilization doesn’t mind sacrificing things like economic growth and technological progress, then it also seems quite easy to avoid value changes.
• Being stable for billions or trillions of years would cause additional challenges of:

  • Possibly encountering alien civilizations.
  • Space travel (to avoid the end of the Earth) would require some significant amount of technological progress from our current state.
  • Unpredictable physical events — the universe has only existed for 13 billion years so far, we’ve only been studying it scientifically for a few hundred, and we’re not that confident about what will happen in the next trillion.
• Out of these, alien civilizations seem most likely to upset an otherwise locked-in scenario. Except for that, we think it seems more likely than not that a trillion-year stable society is possible, and that the arguments fairly robustly point towards them being at least plausibly possible (say, that their feasibility is worth at least 20% subjective probability).

2 Desirability and probability

2.1 Would it be good for highly stable institutions to be built?

2.2 How likely is this?

• Some past authoritarian leaders seem to have desired stable influence over the future.5

• Given any preference about what should happen in the future, a group of humans could be tempted to make sure that the future didn’t stray too far from those ideals. This could apply to many different kinds of groups. For example, a libertarian group might seek to create very long-lasting constraints against interfering with individuals’ freedom.
• Some groups of humans may be extremely uncertain about what they want the long-term future to look like, such that they could want to defer to the deliberation of future people. Even so, they might want to create stably favorable conditions for this deliberation (which should perhaps be stable in the face of everything but the results of the deliberation).
• There are some catastrophic outcomes that many groups would agree should not be allowed to happen even once. For example; people might want to permanently prevent the use of weapons so powerful that they could cause the collapse of civilization (Bostrom, 2019). Similarly, in a world where global value lock-in is possible, it could be important to permanently stop bad, inflexible values from becoming locked-in.

• It would not be necessary to pay the entire cost for stability up-front. Initially, a dominant actor might just invest enough in stability that it could expect to stay stable for a few decades. Throughout those decades, it could gradually invest more in stability, perhaps extending the expected lifespan to a few hundred years, etc…
• It would not be necessary to have all technology necessary for total stability from the beginning. Knowledge and experience of how to design e.g. increasingly stable AGI could accumulate over time.
• It would not be necessary to decide exactly what to lock-in up-front. A group of actors who were initially reluctant to fully lock-in some specific set of values could start by making society very stable in other ways. As they reflected more on what they wanted, over hundreds of years, they might then gradually become more and more confident about their values, making it less and less likely that they would ever deviate far from them.

  • In these cases, there might not be any sharp point where “value lock-in” happens, but the eventual outcome could nevertheless be highly stable values.
  • If all of humanity participates in decision-making, this might look like achieving existential security and then deliberating on what to do with the universe during a long reflection (as discussed in e.g. The Precipice (Ord, 2020)). But the same dynamic could also apply in situations where a smaller group holds power.

• AGI could do cognitive work much faster than humans. On a pure hardware-level, transistors can send signals orders of magnitude faster than human neurons can send signals.
• Major power-shifts and wars are largely caused by social processes and by technological progress, both of which would be accelerated by faster cognition. If the rate of such events were proportionally sped up, biological humans would see a lot of turmoil during a single year.8

• Thus, if human decision-makers wanted to preserve their power and be safe from violent conflict just during their own lifetimes, they might have to restructure things to be significantly more stable than they would be by default.

  • It is also possible that radical new biotechnologies could make human maximum life-spans far longer, which could further increase selfish benefits to stability.

3 Past sources of instability

3.1 Foreign intervention

• The most central example of this occurs when states are invaded by other states.

  • For example, Hitler’s attempt at a “Thousand-Year Reich” ended after just 12 years, when Germany lost World War 2.
  • The Aztec empire lasted for about a century before being invaded by the Spanish conquistadors and their native allies.
• States can also be destabilized by external actors in more subtle ways.

  • Bryan Caplan argues that the existence of non-totalitarian societies where people were obviously “richer and happier” caused party-members in communist Soviet to lose faith in their own system. (Caplan, 2008.)
  • For a while, when Japan was quite isolated, it almost completely suppressed the use of guns. But when a US fleet visited in 1853, the Japanese government was motivated to reequip with firearms, presumably due to fearing conflict with the well-equipped outsiders. (Diamond, 1997, p. 258)

3.2 Aging and death

3.3 Technological or societal changes favoring new values

• New technologies might favor certain types of values or norms, over time causing societies to switch to those new values. (E.g. because people or sub-groups with such values tend to accumulate influence, and because people tend to switch to more useful practices.)

  • As an example of this, the invention of agriculture led greater groups of people to live close to one another, which favored more hierarchical systems of political organization. This probably caused a shift from values that favored egalitarianism to values that were more accepting of hierarchies.
• New technologies might change how people communicate with each other, which might directly influence what sort of ideologies can easily spread through society.

  • For example, the printing press played a significant role in the spread of Protestantism (Rubin, 2014).

3.4 Internal rebellion

• some essential supporters of the state (such as the military or security forces) launched the coup, or allowed the coup to carry on when they could have stopped it,
• the rebellion was feasible even when consistently opposed by supporters of the state.

4 Preserving information, and baseline stability

• With AGI, it will be possible to accurately preserve even quite complex values, by storing large amounts of data, including whole minds.
• By using digital error correction, it is possible to virtually eliminate the problem of small-scale hardware errors during storage, communication, and computation.
• Larger-scale errors (such as a data-center being destroyed by an earthquake) can be made harmless by storing relevant information in many different places.

4.1 Preserving complex goals

4.2 Digital error correction

• Today, almost all AI systems are run using discrete numbers, and there’s no sign that this will change in the near future.

  • It is conceivable that analog computation could be more efficient, for some purposes, but discrete computations can always be used to simulate analog computations to arbitrary precision, so regardless, it should be possible to build digital AGI.
  • Digital error correction for quantum computing isn’t as well developed as for non-quantum computing, yet. But we know of no credible reason for why AGI would necessarily need to run on quantum computers. In particular, quantum phenomena do not seem to play a large role in human cognition. (See footnote 75 in Carlsmith (2020) for some discussion.)
• Independence of errors seems like a fair assumption for small-scale hardware-level errors, but can’t be assumed for larger scale interference.

  • For example: if a meteor hits a computer — affecting all bits simultaneously — then no way to represent information within that computer could preserve the information.

4.3 Baseline stability via redundancy

5 Aligning with goals

5.1 What do we mean by “goals”

• On average, they are good at achieving the goal. When it’s obvious what action will best further the goal, they will take it; and even when it’s difficult to tell, they will mostly take one of the better actions that are available to them. They are at least as good at this as an exceptionally competent human who was highly motivated to accomplish that goal.
• When they act suboptimally, their actions are accidental mistakes. In particular, this means that they never take actions that are optimized for destabilizing the institution that they are aligned with. (In the context of alignment, a similar property has been referred to as avoiding malign failures, while acknowledging that there may be benign failures (Christiano, 2018a).)

5.2 Alignment

• That humans have partly agent-relative goals. This means that copies of a human brain could end up in conflict with each other, as each copy would pursue e.g. its own happiness.
• That humans display many unnecessary forms of value-drift.

• They wouldn’t need to align significantly superhuman AI systems, since human-level AI seems sufficient for lock-in.
• The aligned AI systems wouldn’t need to be competitive with unaligned AI systems, because a dominant institution could prevent unaligned AI systems from being used. Without the necessity of being competitive, alignment strategies could make use of highly inefficient methods, such as spending large amounts of time and resources on thoroughly understanding an AI system’s cognition (using the best interpretability tools available).
• They wouldn’t necessarily need to create AI systems with values that were identical to their own, as long as they were similar enough.

  • For example, if some group of humans wanted to lock-in some combination of their values, handing over control to an AGI system that mostly shared their values (and that would itself never have to do an imperfect hand-over again) could be preferred to leaving them up to many future generation of humans with imperfect transfer between every generation.
  • Similarly, for a system that e.g. tried to prevent the use of WMD, it might not be important that it handles every edge-case exactly like its creators would have, as long as it is still stops all egregious uses of WMD (and that it interferes with very few normal human activities).

6 Stability of goals

6.1 Why do humans’ goals drift?

• Humans’ values and goals can change when their biological conditions change, such as if they are starved of food.
• Humans' values and goals predictably change in some ways as they age.
• Humans’ values and goals can change in response to social pressure to adopt certain values.
• Humans’ values and goals can change when they learn new information, or think about a new argument.

6.2 Why might AGI goals drift?

6.3 Institutional drift

• In general, dealing with possibly-sentient AI systems raises many ethical problems. It’s almost certainly possible to cause huge harm by handling them incorrectly.
• What is harmful for AGIs with very different minds could differ a lot from what is harmful for humans, and it might well be feasible to design AGI systems for which it’s harmless to run them in the ways outlined in this section.
• Regardless, this document isn’t only trying to understand whether ethical actors could create stable institutions, but also whether there’s a danger of unscrupulous actors doing it. Thus, the below discussion is important regardless of how the ethical issues shake out.

• Arranging the systems into hierarchies of supervision, where more reliable systems continually check that more-experienced systems aren’t doing anything catastrophic.
• Continuously testing systems, as they learn more, to ensure that they’re still loyal to the institution’s goal.

• The verifiers can aim to have a good understanding of what the goal is supposed to be, and check that the supervised or tested systems’ own goals don’t drift too far from the original.
• The interpretation of the institution’s goals can be relatively centralized, and the verifiers could check that the supervised or tested systems are intent aligned (see Christiano (2018b)) with the central source — i.e., that it intends to do what the central source wants it to do.

• The institution could then afford to spend a lot of resources on getting the interpretation right.
• Supervisors could separately verify that systems lower in the hierarchy both (i) have decently accurate beliefs about what the interpreters would say about various situations, and (ii) that they’re motivated to act according to those beliefs.

  • This decomposition seems nice since (i) has a clear source of ground truth, and (ii) is a more crisp and precise motivation than checking that their motivations align sufficiently well with a very complex set of values.
  • In addition, it’s ok if an intent aligned system doesn’t understand how the values apply to all situations, since if it encounters an unknown situation, it would be motivated to ask for clarification. This might not be true for a system which just (mostly) shares the institution’s goals.
• If it’s genuinely unclear how to apply values to some situations, a centralized source can make a judgment call that everyone will coordinate around, without impinging on other parts of the institution’s values. Whereas if everyone different parts of the system maintain separate values, it seems more likely that AIs in different parts of the system would acquire slightly different values — perhaps resulting in conflict that causes much bigger problems.

• how a stored goal could be interpreted in a wide variety of situations, and
• how an institution’s actions could be stably aligned with those interpretations.

6.4 Interpreting goals

1. If the original version’s values are clearly and unambiguously articulated in a way such that any sensible listener would recognise that the more-informed version’s desires are in contradiction with it.
2. If there’s some limited (but faithful) summary of a more-informed version’s desires (or thought-process) such that a less-informed version, upon hearing that summary, would be confident that the more-informed version has deviated from a rule that they never wanted broken.19

3. If the judgment of the more-informed versions differ depending on what order they’re exposed to various arguments, or are otherwise contingent on facts about the judgment process that ought to be irrelevant. (As judged by themselves.)

1. In this case, it should be fairly clear that value-drift happened. Drift could be identified by the algorithm in the next bullet points.
2. Insofar as less-informed systems can recognise drift, have more-informed interpreters be supervised and tested by less-informed versions of the same systems.

   • Perhaps by tasking some limited AI systems with the job of searching for accurate summaries of a situation that would cause a less-informed AI system to reject it.
   • Perhaps by letting less-informed AI systems thoroughly interrogate and understand (parts of) the more-informed system’s reasoning. See section 6.6 for more discussion of this.
3. When making high-stakes decisions, always bring back many copies of the original AI system, and give each of them a different summary of the situation, recounting evidence and arguments in different orders.

   • Those copies could themselves have opinions on what summaries are appropriate to give to further copies of themselves, and they could discuss and compare their perspectives with each other.
   • Such a process may not remove all arbitrariness, but it could reduce it by a lot.

6.5 Aligning with that interpretation

• Thoroughly test and understand how each system will behave after foreseeable inputs.
• Reset systems back to a thoroughly-tested state often. (This gives the systems less time to stray from a trusted state, and allows testing to focus on shorter inputs.)
• Continuously check that each system does its job reasonably well, and in particular doesn’t pose any danger to the institution as a whole, through a combination of:

  • Using very similar systems to perform exactly the same task; to cover cases where sheer randomness would cause one of them to drift.20

  • Using supervisors who don’t engage as deeply with the inputs as the supervisee, but who check that the outputs are acceptable.
  • Continuously re-testing the system’s motivations.

• For every fact or skill that’s necessary in the relevant field, extensive experimentation could identify the minimal changes to a trusted AI system that would give them that skill. The AI granted those skills could then be thoroughly tested using the methods discussed below.
• Researchers and engineers may then have to spend significant amounts of time thinking about their intended topic. But they could be supervised by copies that spend much less time, who need to approve every insight or new design that the researcher produces.

6.6 Verifying loyalty

• AI systems could test copies of themselves, to better understand how robust they are, and in which situations they could fail.
• If there is some important piece of information that needs to be internalized by many copies, they could expose some copies to that information, and then test them to ensure that it didn’t cause any changes in their goals.
• If a supervising AI sees some reason to suspect that a supervisee may have been corrupted, they could have them tested.
• If the AI systems develop some other, more specialized systems to fill particular niches, they may want to test that those systems behave well in the situations they are to be implemented in.
• AI systems could test copies of themselves that they had tweaked to increase capabilities or robustness, or could test entirely new AI systems that they wanted to replace themselves with.

• The more extreme tests could be quite expensive. But the escalation could be gradual, depending on how likely it is that the tested AI system is dangerous based on both prior information and the results of cheaper tests.
• Some of these testing schemes could cause the testers themselves to experience value drift, by being exposed to the same information that the testee was exposed to. But if just a few of the testers suspected that this had happened (without themselves being affected), they could sound the alarm and reset both the tested system and the testers to a previous state.

• Typically, the tested AI wouldn’t be any more competent than the testers. Therefore, if there’s any possible flaws in a test that would let the testee deduce that they were being tested, the testers would be aware of this, so that they could avoid such flaws. This means that the testers would know how the tested AI could deduce that it was being tested, which would allow them to design tests where the AI was genuinely unsure whether it was currently being tested or not. The testers being at least as competent as the testee also means that — if the tested system chooses some action because it can see that it would harm the institution — the testers would also be able to understand and recognise that that action could harm the institution.

  • Cases where the tested AI system is more capable than the testers would be avoided if the institution never improved its AI technology.
  • But if they wanted to build stronger AI systems, one technique they might use is to have the tested systems’ capabilities only be slightly stronger than the individual testers, and then use multiple testers with more processing power to check each of the stronger system’s decisions; with the hope that many fast AI systems would be more capable than a single, slower slightly stronger system (Christiano et al., 2018).
• There could be significantly more tests of high-stakes situations than low-stakes situations. For example, testers could simulate situations where the AI system sees suspicious behavior that it should report, or it could simulate a crisis situation where the AI system plays a pivotal role.

• Transparency tools could be used during the testing-procedures above, for at least two purposes:

  • First, they could clarify why an AI system does what it does in the tested cases. For example, if it’s trying to figure out whether it’s in a test situation (presumably with an intention to behave differently depending on the answer) that’s a red flag.
  • Second, it could potentially be used to modify an AI system's beliefs directly. For example, this could be used to test what the AI system would do if it really believed that it was not currently being tested.
• Separately from behavioral testing, sufficiently strong transparency tools could also directly look for important beliefs, goals, or intentions in the mind of the given AI system.

  • Some testers could try to understand as much as possible about the AI system’s mind. If that was an unfeasibly large task, or if analyzing the entire mind would have a high risk of corrupting the analyser, there could instead (or in addition) be many AI systems analyzing many small bits of the AI’s mind.
  • Relatedly, transparency tools could be used during supervision as a way to detect big changes of goals without the supervisor being exposed to the reasons that the goals changed. I.e. the supervisor could monitor certain parts of the supervised system’s cognition, chosen such that if a supervised system encounters information that induces value drift, the supervisor can notice this without themselves processing that information.

• By default, we might expect AI systems of the future to become more complex and thereby more difficult to analyze.

  • For some reasons to think that smart AI might be uninterpretable, see Sharkey (2022).
• But we could also expect progress in transparency techniques. In addition, we should expect better AI capabilities to directly contribute to automating the process of interpretability.

  • For example, with a lot of work, a team at OpenAI managed to understand a small part of a neural network in great detail (Cammarata et al., 2020). Having humans take equally much time to look at every part of a large neural network is not feasible. But it could become scalable with the help of AI tools.
  • By the time we have AGI, achieving a human-level understanding of how each component of a large neural network works might be possible by using many copies of such AGIs to analyze them.21

• This still leaves significant uncertainty about how interpretable future AGI systems will be. But if we consider an institution that could develop and select their systems partly for being interpretable, and who were willing to pay large costs to analyze individual systems, it seems more likely than not that they could eventually develop AGI where a high degree of transparency was possible.

• The testers could have many thousands of conversations with the human, each instance with a different conversation partner. Some of these could take place in highly stressful or highly relaxing situations. In some, the human could be given drugs. In some, the testers could directly influence the human’s digitally-implemented neurochemistry. Some simulations could last for a long time, seeing what happens if the human spends years immersed in various environments.

  • That procedure alone seems like it should give an excellent understanding of a human’s goals, desires, and behavior.
• In addition, throughout all these diverse situations, neural activations in the person’s brain could be monitored by humans or by ML models trained to predict whether the human is experiencing various emotions, whether they’re lying, what topics they are thinking about, etc. This could then be used to check for signs of dishonesty during questioning or testing.
• Of course, some of these methods are human-specific. But the human case is a useful data point, insofar as we don’t expect humans to be a huge outlier in how easy it is to get insight into our psyche.

7 Robustness to natural disasters

• Local disasters do not pose a problem, given high baseline stability achieved by redundant storage of everything important.
• Global disasters are rare, and ones that cause widespread comprehensive destruction are extremely rare.
• In the long run, the institution could also spread to space.

7.1 Natural disasters

• Biological pandemics are not a threat; primarily because AI would not operate on biological hardware (and an AI civilization would not need to rely on humans for anything), but also because it seems likely that future technologies would make pandemic mitigation extremely effective.

  • Intentionally designed computer viruses would be an analog to engineered pandemics, but they could be prevented by preventing anyone from constructing them. See section 8.
  • In favorable circumstances, something analogous to computer viruses could probably arise without intentional actions, just like biological viruses arose from pure evolution, without being designed. But an institution trying to be stable could make this very improbable.

    • The evolution of biological diseases rely crucially on occasional errors in DNA- or RNA-copying, which generate variation on which natural selection can act. But via the use of digital error correction, the probability of such low-level errors could be driven to effectively 0.
    • There would still be some variation induced by the use of different AI systems in different contexts, or different AI systems receiving different histories of inputs. But this would be relatively little variation, and extensive supervision and testing protocols (see section 6) could both reduce the variation (e.g. by frequently resetting AI systems) and notice if any bug seemed to be self-replicating.
• The main reason for which asteroid/comet impacts and supervolcanic eruptions can become global (rather than local) catastrophes is because they can throw up enough dust to blot out the sun for a period of years, killing plant life that relies on the sun and animal life that relies on those plants. But an AI civilization could easily survive such catastrophes by using energy-sources that don’t depend on the sun (such as nuclear power or enough stored electrical or chemical energy to last for several years).

  • Sufficiently large astronomical objects would pose more extreme dangers, including liquifying the Earth’s surface into magma. But the largest asteroid to hit the Earth in the last 100 million years (the asteroid that killed the dinosaurs) mainly had global impact via the atmosphere, and asteroids much larger than that one are significantly rarer.
  • In addition, future technologies would probably enable very effective asteroid detection and deflection.

7.2 Astronomical stability

8 Robustness to non-aligned actors

• has uncontested military and economic dominance,
• has the ability to produce any number of stably-aligned agents, and
• that they will try to survive and maintain coercive power as long as possible.

• For any task that the institution needs done, they can use one of their stably-aligned AI systems for it. Thus, actions taken by humans and non-aligned AIs would not be necessary for the continued existence of the dominant institution.
• The institution could control far more stably-aligned AGI systems than there are humans or non-aligned AI systems, and could therefore do any amount of surveillance necessary to ensure that no-one causes the institution any catastrophic harm.

8.1 In-principle feasibility

• For human members of the population, this would (by assumption) be at least as effective as having each human be constantly watched by another human, the latter of which was loyal to the dominant institution.
• Part of the population could also consist of AI systems that are not robustly aligned, which raises some new questions. In particular, it may be difficult for an aligned AI system to surveil a different, unaligned AI system that was much more capable, at some tasks, since this means that the aligned AI doing the surveillance may not understand the intention behind or consequences of some of the surveillee’s actions. Thus, the dominant institution might have to be on the cutting-edge of AI technology (possibly by prohibiting all superior forms of AI).

• Currently, it costs ~$10/h to rent 8 A100 GPUs (Amazon, 2022), collectively capable of 1e15-1e16 FLOP/s (Nvidia, 2022). This is probably similar to or greater than the FLOP/s used by a human brain (Carlsmith, 2020). There aren’t currently enough GPUs to run more than a few million models of this size, but supply could be expanded if there was demand.
• Technological progress has historically led the price of computation to fall by ~6-300x per decade (AI Impacts, 2015, 2017). While the current paradigm of computer hardware is approaching physical limits, some further gains remain for cheaper AI hardware. (And more speculatively, the price of computation could fall many additional orders of magnitude with the use of novel computation technologies like reversible, optical, quantum, or superconducting computing.)
• And AGI could both accelerate technological progress by a lot (leading the price of computation to fall even further) and massively increase wealth, making expenses more affordable (Davidson, 2021; Trammel, 2020).

8.2 How much control is needed?

• (i): The most clear way to pose a concrete threat to a dominant institution would be to amass enough defensive or destructive power that it would no longer be dominant.

  • Since the dominant institution could make itself militarily dominant with a very large margin, the defensive approach could be very difficult, and likely easy-to-detect even without any significant degree of surveillance.
  • And since a dominant institution could make many redundant copies of everything important, and spread it across the world, the destructive approach would require truly worldwide destruction. It might be the case that preventing such worldwide destruction would require extensive surveillance (Bostrom, 2019). But note that every civilization that wanted to avoid imminent destruction would have to handle such threats; so these costs would not be unique to lock-in.
  • Our main uncertainty here is about cybersecurity. Perhaps there are hacks which could cause worldwide problems in a world with sufficiently centralized power, that wouldn’t cause problems for a more decentralized world.
• (ii): A dominant institution may also need to prevent unauthorized space travel, since anyone who left the institution’s purview would be able to build their own civilization, until they posed a threat.

  • Though this probably wouldn’t be a danger if the institution had already sent out its own colonization wave and already controlled all accessible resources in space.
• (iii): In human authoritarian societies, it’s common for regimes to try to limit what kind of ideas and ideologies can spread throughout society. This seems much less necessary in an AGI-controlled society — if a dominant institution’s essential supporters were truly stably aligned, then there would be little or no reason that a dominant institution would have to do this, since the remaining population couldn’t threaten it. However, there are two possible partial exceptions to this:

  • First, it is possible (though we think it’s unlikely) that AI systems’ stability could depend on what information the systems are exposed to. In section 6, we bring up the possibility that some AI systems might be stable only if they are not exposed to certain information or arguments. If this were the case, then in order to maintain stability in the long-term, it could be necessary to prevent such ideas from becoming too common. This would require enough surveillance and control to prevent those ideas from being invented and/or from spreading.
  • Second, it is possible that a dominant institution could find it easier to maintain stable control if some particular technology isn’t widely adopted. The most salient examples of this are technologies of mass destruction, which we discussed under point (i) above. But perhaps some technologies could be uniquely problematic for a locked-in society. For example, perhaps the dominant institution relies on some less-than-maximally effective form of AI (e.g. because they have more stable goals), and would be threatened by the development of superior AI systems. If so, it may need to do enough surveillance to limit technological development to this lower level.
• But it seems fairly likely that (iii) wouldn’t be a problem, in which case a dominant institution may only need to really robustly prevent (i) extremely destructive capabilities, and (ii) unauthorized space travel to regions the institution does not yet govern.

8.3 Alien civilisations

References