AGI and World Government

Introduction

• Not much. Bottlenecks to AI progress bite hard, competitors soon catch up, real world impacts are slow to come online, and current governments take all of this in their stride.
• Misaligned AI takes over.
• Society slowly goes off the rails.
• There’s an enlightened, globally coordinated response.

1. The first project to develop AGI is:

   • Government-led rather than private.
   • Multilateral rather than single-government.
   • Governed by a coalition of democratic countries rather than all countries.
   • Governed by an explicitly interim and time-bound arrangement, with definitive governance arrangements to be made at a later date.
2. Different countries in the coalition are given fixed voting power, and neither one-person-one-vote nor one-country-one-vote are used.
3. Countries that are not part of the project receive major benefits from the development of AGI and credible reassurances that they won’t have their sovereignty violated later on.

Why expect the first AGI project to evolve into a world government?

• Assuming that alignment is achievable, the first project to build AGI would need to decide what constitution or model specification to align the AGI to. The most obvious approach would be that the governing board of the project has ultimate authority, including in cases where any constitution provides unclear guidance, or if the constitution is to be changed. If this happened, the project would by default have ultimate control over the behaviour of its AGI systems, and the future systems they build.
• If there’s a large enough intelligence explosion, then AGI would quickly lead to superintelligence (“ASI”). This would give the project such a huge capabilities advantage over the rest of the world that — unless quickly checked by other actors — they could effectively achieve a decisive strategic advantage — a level of technological and other advantages sufficient to enable it to achieve complete world domination.2

  • To get more concrete, the first project to develop ASI might be able to quickly do things like: automate 99%+ of the economy; build up its own army very rapidly; bloodlessly disarm all other military powers; strategically manoeuvre such that other actors do what it wants without the use of military force; prevent other actors developing ASI through sabotage, persuasion or force; unilaterally seize control of space resources and block other actors from following suit. Of course, this is speculative — but it’s what I think is plausible given how rapid technological progress could become.

(Weakly held) implications for AGI governance

1. The first AGI project is government-led, rather than private. 

   1. Corporate governance structures are (a) not designed to govern political power, (b) not tested at governing political power. 

      1. A privately-developed AGI by default will be aligned to the CEO or to the company’s governance regime. If the former, de facto autocracy is likely. If the latter, it is at least a major risk: the CEO could potentially outwit or collude with the Board and largest shareholders, or simply start ignoring their demands post-AGI, and thereby become de facto dictator. And, even if that doesn’t happen, power over the de facto world government would essentially be in the hands of the company’s largest shareholders — who probably represent a small fraction of society. 
      2. In contrast, democratic governance is the best approach to political power that has actually been tried. Governments also have far more legitimacy than companies to exercise political power (though more on this below). 
   2. What’s more, if the first AGI project is private, we expect that the relevant government will intervene, and we’ll end up with a government-led project anyway, but one that was set up in haste and without multilateral involvement.
2. The project is multilateral, rather than single-government.   

   1. If the first AGI project is to evolve into a world government, then avoiding the risk of the project becoming an autocracy is extremely important.3

      Having multiple governments with some meaningful control over the project reduces this risk considerably: even if one government becomes more authoritarian, the others can oppose this.4

   2. Moreover, if the project becomes a world government, it seems desirable for many governments and people to have a stake in the project, and for all people to receive benefits from it.
3. The project is governed by a coalition of democratic countries, rather than as a global democracy.  Here are the arguments for this, from least to most controversial:

   1. Global democratic governance is unlikely to be feasible, because it would involve the US giving up a lot of power. Pushing hard for global democratic governance may make a multilateral project of any kind less likely, increasing the chances that the US government goes it alone, and that we end up with something like autocracy.
   2. From the perspective of ensuring a flourishing future over the long term, the gains from global democratic governance may be quite small.

      1. For one thing, most beings with moral status wouldn’t be represented by either a coalition of democratic countries or global democratic governance — as most beings are future beings (also, animals and digital minds). So there aren’t big gains on that front.
      2. For another, going from a coalition of democratic countries to all countries matters much less than going from autocracy to a coalition of democratic countries, in terms of increased moral diversity.5

         The gain of going from hundreds of millions of people being represented to 8 billion is only an order of magnitude. In contrast, the gain from going from a single person in charge to a hundred million people being represented is 8 orders of magnitude.  
   3. Global democratic governance might increase the risk of authoritarianism. In a survey of citizens from 24 countries,6

      64% of people said that rule by a strong leader or the military would be a good way of governing their country. Of those countries which would be most likely to take part in a multilateral AGI project,7

      only 31% of people agreed to the same claim.
4. The project is governed under an explicitly interim arrangement. 

   1. For example, the project could be governed by some time-bound governance structure, with a binding agreement that this structure will be renegotiated after a certain number of years (as was the case for Intelsat). The case for this is that designing the ideal world government post-AGI is very hard, and we’ll do a much better job of it after we’ve thought more about it, with the help of AGI and ASI. 
5. Different countries in the coalition are given fixed and weighted voting power, rather than using a one person one vote or one country one vote system. 

   1. The reason to fix voting power is that post-AGI, rapid population growth will become possible (whether of digital citizens, or biological ones via artificial wombs and robot child-rearers). If project voting were one-person one-vote, then whichever country grew its population the fastest could seize power.
   2. The reason to weight voting, rather than use one-country-one-vote, is that otherwise small countries would get disproportionate amounts of power, in a way that seems arbitrary and very non-democratic. For example, each of around 100 smaller countries would have at least 100x the voting power per person as the US. And, pragmatically, weighting would also make the arrangement more palatable to the US, making an international project more feasible.
   3. There’s some tension here: if the voting is weighted so that countries are proportionately represented, but fixed so that runaway population growth can’t be used to seize power, then the weights between countries could eventually become very disproportionate.8

       
6. Countries that are not part of the project receive major benefits and credible reassurances that they won’t have their sovereignty violated. 

   1. The prospect of world government makes it more likely that non-participating countries will take drastic action (stealing model weights, short-cuts on safety, kinetic strikes) in order to prevent that from happening. This puts more importance on ensuring that countries that are not part of the first AGI project receive major benefits from the development of AGI and credible reassurances that they won’t have their sovereignty violated in a post-AGI world. That said, we believe we should still be reluctant to give much in the way of formal governance power to authoritarian countries.