Should We Lock in Post-AGI Agreements Under Uncertainty?

Citations

13th March 2026

Abstract

It’s a widely-held view that we should delay locking-in consequential or long-lasting agreements until we know much more about the world after a possible intelligence explosion, and until we can engage in extensive reflection on our beliefs and values. But some mutually-beneficial deals depend on shared uncertainty about the future. If we prevent those deals, we’re potentially leaving a lot of expected value off the table. So we need to make those deals before we have time to seriously reflect. I discuss when this could be true, and what the practical implications could be.

Introduction

Here are two sketches of how the near-future could go:
Early deals. In the run-up to an intelligence explosion, society lays the groundwork for credible, high-stakes agreements about uncertain post-AGI outcomes: new AI tools for arbitration and deal-making, institutional commitments to honouring deals, perhaps legal precedents. More risk-tolerant people insure those who are more risk-averse about their wealth. Some make deals to concentrate their influence in worlds where they think the moral stakes are highest, increasing their expected impact. On the international scale, major powers like the US and China agree to share (at least some) future resources, while they’re both unsure who might dominate without a power-sharing agreement. Enforcement relies on a system of iterated growth, plus AI-enabled self-commitment and third-party enforcement mechanisms. The result is a more pluralistic future, and one shaped more by moral motivations.
No early deals. In the run-up to an intelligence explosion, no such agreements are made about post-AGI outcomes. Nobody can buy insurance on their post-AGI wealth, nor insure others. Those who care most about worlds with the highest moral stakes have no way to stake resources on their convictions, so their influence on those outcomes reflects only their pre-AGI wealth. When one major power eventually pulls ahead, it’s too late to negotiate, and the losers end up empty-handed.
The main question I’ll discuss is whether we should prefer a near-future more like the “early deals” scenario.
On one hand, early deals miss later opportunities for reflection and understanding. On the other hand, some deals require uncertainty. By trading between possible states of the world, both parties to a deal can improve their own prospects, according to their own beliefs and values. The classic example is insurance: while your house hasn’t been struck by lightning, you and your insurer can improve each other’s prospects. But once your house gets struck by lightning, it’s too late to make a deal. So there’s a tradeoff.1

I consider three kinds of deal which stand out as most promising to enable early relative to an intelligence explosion: deals about the relative stakes, about the material stakes, and about the normative stakes. For relative-stakes agreements, I focus on a deal between major powers. For the two other kinds, I switch to focusing on individuals.
The basic case for enabling all these agreements is the same basic case for any voluntary commitment: all parties improve their prospects by their own lights, and nobody else is hurt. Moreover, agreements between major powers to share resources could make the future meaningfully more pluralistic and morally diverse, which seems better under moral uncertainty than a more unipolar future. And agreements between individuals could give more influence to those who staked their wealth today on future outcomes as a credible show of their beliefs or values, and were vindicated.
It’s not clear that the kinds of deals I discuss will be possible to make. We’ll need much more powerful mechanisms to enforce international agreements for a power-sharing deal to hold. Similarly between individuals, contracts involving future income might be unenforceable. More importantly, it’s not clear which of these deals really are worth trying to enable: because the stakes are high, the downsides from rash or unwise early agreements are also high.
Still, my guess is that early resource-sharing deals between major powers (like states or companies) look most promising, albeit least tractable. With appropriate safeguards, I also come down positively on enabling private deals based on the material and normative stakes. To make the sensible and impartially desirable early deals more likely, we might consider encouraging commitments from private institutions to honour small-scale deals, building diplomatic frameworks for resource-sharing treaties, thinking more carefully about voiding mechanisms and safeguards, and even setting up infrastructure for trading on post-AGI outcomes.

Three kinds of early agreement

In this section, I’ll consider different kinds of potentially long-lasting or very consequential2

deals3

which are only feasible to make before or relatively early in an intelligence explosion, because they depend on later outcomes being unknown or uncertain.4

I’m not assuming it’ll be easy to make these deals, or overall desirable to make them: I discuss those questions in the “Objections” section. None of these deals depends on disagreement over the odds of outcomes.5

Before I discuss each kind of deal, here’s an example of each, so you have something to picture:
  • Deals about the relative share of resources (share of the pie): The US and China6

    agree to share some fixed division of future resources, while they’re still unsure who would otherwise achieve a decisive strategic advantage to grab all future resources. They agree to install systems for automated monitoring and enforcement.
  • Deals about the material stakes (size of the pie): Beth agrees to pay Bill if space resources are distributed for private ownership by 2060, and vice-versa. Bill cares more linearly about space resources, so he effectively insures Beth: he gets to own more space resources in expectation, while Beth de-risks her future wealth.
  • Deals about the normative stakes: Adam agrees to pay Annie if extensive AI reflection concludes that moral realism is roughly correct, and vice-versa, since Annie is more motivated by her impartial impact in these worlds.
These three examples track the three central kinds of deal, which I now turn to.

Deals about the relative share of resources

The first kind of deal is about the share of the pie that both parties get; the relative material share. When the wealth of two parties isn’t perfectly correlated, then those parties can agree on a way to exchange wealth contingent on how their relative wealth changes.
When two (or more) parties are risk-averse about their own wealth, and they face imperfectly correlated risks to their wealth, then in principle there will be some risk-sharing deal which they will both prefer to no deal. Mutual insurance is a real-world example: two or more people can agree to pay into a pot, which then pays out if anyone suffers a loss (like if they become ill, or their house burns down). The risk is shared, rather than transferred to a third party.7

Great power resource-sharing

Here’s a concrete scenario where I think this kind of deal could matter. Let's say there’s some point before an intelligence explosion where two great powers, both the US and China agree they both have a 50% chance of achieving a decisive strategic advantage (DSA) without any power-sharing deal.
I assume that both the US and China are risk-averse about their share of resources in the future, that they both understand one side will achieve a DSA without a resource-sharing deal,8

and they both agree on the chance either side has of securing it. Then there will be some division of future resources where both the US and China would prefer a guarantee of their side of that split, over some shot at securing a full DSA, and some shot at losing almost everything.9

For example: say both sides currently control 10 arbitrary units of resources,10

and they could control 10,000 arbitrary units with a DSA, or otherwise fall back to 1 unit. With a power-sharing deal, they could agree to control 5,000 units each. Both the US and China care about the order of magnitude of resources they end up controlling,11

so the same marginal resources are worth less on top of a larger existing stack. Then the even-split deal is far more attractive than the no-deal scenario. To see this, ask: what guaranteed amount of resources would be just as good as the no-deal gamble for both powers? Given log utility with resources, the answer is just 100 units — fifty times less than the 5,000 guaranteed by the deal.12

Area chart showing the surplus from a power-sharing deal (pink) is largest at even odds (p=0.5) and shrinks to near zero as one side's probability of achieving a decisive strategic advantage approaches 1.

Image

I focus on the possibility of a deal between the two great powers, here the US and China, because the stakes and consequences would be so high.13

But what I’ll say could apply to deals involving companies, other states, coalitions, groups, or individuals.
At least in theory, both sides should prefer to strike an early power-sharing deal over any later one, because the expected surplus from a deal shrinks over time (from any starting odds).14

Waiting to learn more is costly in expectation.
Moreover, deals under more uncertainty (with odds closer to even) would also seem more attractive from an impartial perspective, since they result in less concentration of power and more moral diversity.
On the first point: a unipolar future could fall far short of the best feasible futures, because a single autocrat could enforce their own parochial values, with no opportunities for exit. Even having two alternative systems might open up the possibility for comparison, healthy competition, and moral trade.
On the second point: if you are currently morally uncertain, then from an impartial perspective you should probably prefer a more morally diverse future to betting the entire future on one value system, because of the opportunity for moral trade between alternative value systems.
So if both sides are starting close to even odds, those might be impartial reasons for enabling early power-sharing agreements.
Line chart showing the zone of possible US-China resource-sharing agreements across probabilities of the US achieving a DSA. Some deal is always mutually preferred to a DSA race, but an even split is only feasible between roughly p=0.08 and p=0.92.

Image

Enforcing a great power resource-sharing deal

Substantial international agreements are notoriously hard, because powers like the US and China can’t appeal to an even more powerful entity to enforce their agreements. Reputational costs and one-off sanctions are often not enough to hold together a deal where defection is attractive. The question is what else could.
One approach could be to effectively turn the power or resource-sharing agreement into an iterated game, where both sides maintain a balance of strategic forces as they both grow and expand, such that neither could achieve a decisive strategic advantage throughout. Advanced AI itself could enable mechanisms which aren’t workable today, like credible self-commitment using AI systems designed to enforce the terms even against the interests of the party that built them, or mechanisms for verifying compliance without passing on compromising information. A final option is just to make resource-sharing deals more modest: by giving them an expiry date, or limiting them only to certain kinds of resources.
One issue is that, if these deals are to be struck relatively early in an intelligence explosion, then mechanisms for monitoring and enforcement will be relatively untested. See the ‘Feasibility objections’ section for more discussion.

Deals about the material stakes

In the previous section, the uncertainty was about who gets the bigger share of future resources. In this section, both parties keep the same share, but they value resources differently depending on how the world turns out. More generally, when one party cares more about having resources in a particular state of the world — say, because their moral goals are more tractable in that state — they can trade for more resources in that state by giving up resources in states they care less about.
Structurally, this kind of deal is very common. Take the case where a farmer sells a wheat future today, to guarantee a sale price at the time of harvest. On the other side, a speculator accepts the price risk in exchange for higher expected returns. The key enabler is that the farmer and the speculator value a guaranteed dollar differently: the farmer’s livelihood depends on the harvest price meeting some minimum level; the speculator is happy to bear the farmer’s risk, for a premium.
Unlike deals over relative share, where the key enabler is that both parties are risk-averse about who ends up on top, the key enabler here is that parties value resources differently across states of the world — maybe because they differ in risk-aversion, or for any other reason.
For example: Annie and Bob might both be uncertain about whether hugely valuable future resources are auctioned to private actors by some future date. For the sake of concreteness, imagine permits to capture some fraction of the Sun’s energy, or fractional titles to stars in our galaxy. Annie is less risk-averse than Bob about future wealth. So, Bob agrees to transfer some of his upside to Annie in the case where stars or sunlight are auctioned, in exchange for Annie guaranteeing Bob some minimal resources if the resource pie isn’t massively expanded in that way.15

But two parties don’t necessarily need different levels of risk-aversion to strike a deal over the material stakes. They might instead value the same absolute amount of resources differently contingent on different empirical outcomes. For example, Clara might care more than David about holding resources in worlds where the same resources can alleviate more wild animal suffering. So they could agree that David pays Clara if that turns out to be the case, and vice-versa.
More generally, someone might have impartial values which scale more linearly with resources; and they could make early deals to increase their expected impact (by insuring people with more risk-averse preferences). This could be good from an impartial perspective, even if you’re morally uncertain about how linearly you should value the goodness of outcomes.16

This is similar in spirit to mission hedging: choosing investments which pay off most in outcomes where you expect your dollars to go furthest. For example, a philanthropic actor focused on improving the course of AI progress might buy call options on AI companies, which pay off when AI progress is fastest, and philanthropic spending on AI goes furthest.17

Enforcing deals about material stakes

Because individuals don’t face the same problem of international anarchy that great powers face, the simplest way to enforce a deal between individuals or groups is to enter into a contract enforced by a third party.
The enforcing agent could be a private actor, like today’s private arbitration and escrow services. Advanced AI could make such systems more transparent and trustworthy, because — in principle — both parties could agree to the terms of the deal, plus voiding conditions, relying on judgement from the AI arbitrator to decide which conditions are met.18

But many of the most interesting deals involve committing future income — potentially a large windfall of resources that doesn’t yet exist. Escrow can’t hold wealth you don’t have yet, and it’s unclear whether private arbitration can enforce such commitments without the coercive power of courts.
Ideally, then, the agreements would be legally enforceable. But it’s also not clear if existing common law would recognise deals to commit future income (or many other unusual details of the kind of agreements at stake). See the ‘Feasibility’ section for more discussion.

Deals about the normative stakes

The third and final kind of deal I’ll consider is when two actors agree to trade contingent on learning new normative or theoretical knowledge.
For an example of a deal hinging on new theoretical knowledge: suppose Ava’s mathematical research depends on the still-unknown XYZ conjecture being true; Ben’s depends on it being false. There’s a chance XYZ is soon proven true or false. If so, they agree that whoever’s work is vindicated pays the other, with a trusted friend as arbiter. Such deals might be fairly rare today, but they’re totally possible.19

Instead, two parties could make a deal about theoretical knowledge for moral reasons. Suppose Ava is (psychologically speaking) very motivated to do the most good if it turns out there are facts about how to do good. Ben’s moral motivations, on the other hand, aren’t contingent on whether something like moral realism is true. Thus, Ava cares much more than Ben about having resources in the world where it turns out something like moral realism is correct.20

The issue at stake doesn’t have to be moral realism. Ava might care more than Ben about having resources if a totalist axiology turns out to be roughly correct (according to the arbitration process they agreed on), or a hedonic conception of wellbeing, and so on. Neither does it need to be an explicitly moral question; just a morally relevant one. The wager could be over which theory of consciousness is vindicated by much more reflection, for example, or whether personal identity is preserved through certain kinds of mind uploading. In any case, there is a deal to be made in principle; and the question is how to set the terms.

Enforcing deals about moral stakes

Ultimately, Ava and Ben need to agree on empirical resolution criteria for their wager,21

which stand a reasonable chance of resolving their question reliably enough that they’re prepared to bind themselves to the result. If they’re lucky, they might know a wise and trusted mutual friend, who can tell them whenever she reaches a strong conclusion either way. Ben could then agree to pay Ava if their friend comes down in favour of moral realism, and vice-versa, nullifying the deal if their friend doesn’t decide.
But it’s more likely that Ava and Ben are not so lucky. Unlike mathematical claims, open moral questions are rarely settled so firmly as to compel agreement, so shared resolution criteria are hard to find today.
Advanced AI could change that, by acting as a mutually trustworthy arbiter. Ava and Ben could agree to take a (perhaps future) model from a developer they both roughly trust, and pose it a question whose phrasing they agree on in advance, at an agreed date. They could also agree that the model passes pre-agreed tests for epistemic fairness, openness, and rigour. The model can assess the state of intellectual progress at the time, and do its own substantial reasoning about the question. The model can pronounce in either Ava or Ben’s favour, or nullify the deal. That conclusion becomes the resolution criterion. Of course, the choice of model and prompt itself becomes a negotiation, but the idea is that it should be possible to agree on terms even when the underlying question is still unknown.
Two features make this kind of deal seem promising. First, AI systems may have structural advantages as impartial arbiters: no personal relationship with the deal-makers, potentially more legible and auditable biases than a human arbiter, and an existing reputation for answering many similar questions fairly.22

Second, an intelligence explosion will plausibly involve an acceleration in intellectual progress, perhaps resolving long-running normative questions in just a few years, so such deals can resolve quickly enough to make them attractive.23

Morally motivated deals can also have positive externalities. If Ava makes a deal to increase her impact in high-stakes worlds, other people who share her moral motivations benefit too, as if they had made the deal themselves. This suggests that the impartial value of morally motivated deals could be much larger than the private gains to the parties involved. These positive externalities could also open up collective deals that wouldn’t happen between two people. That is, Ava might find it too costly to shift any marginal resources into the morally high-stakes state; but if many similarly-motivated people each agree to shift a small amount at the same time, then Ava can multiply the impact caused by her own contribution.24

The difficulty is that collective deals are harder to negotiate: any one person can threaten to hold out for better terms, and coordination costs grow with the number of parties.25

Are early deals feasible?

I’ve considered three kinds of deals which hinge on shared uncertainty about the future. Because that uncertainty will resolve throughout an intelligence explosion, these are deals which have to be made early or never. The question now is whether we can actually make them.

Issues with enforcing deals between major powers

Great powers like the US and China can’t appeal to a more powerful entity to enforce their agreements. But once one party to a resource-sharing deal becomes confident they could take most resources for themselves if they defected from the deal, then they would have a strong self-interested reason to defect.
One present-day reason for a state to stick to international agreements is to avoid reputational damage and sanctions. But if a great power can win a decisive strategic advantage over other powers by reneging on their resource-sharing deal, then those threats don’t matter. The question is how else two major powers can enforce a deal to share power and resources throughout or beyond an intelligence explosion.

Iterative growth

Firstly, the major powers could try to turn the agreement into an iterative process, with opportunities to retaliate against small defections. Both parties could agree to grow proportionally, always maintaining roughly the balance of power and resources they initially agreed on. They form a mutual understanding that if one party grows too fast, or takes actions to grab a larger share of future resources for themselves, then the other party retaliates. If both parties always value future cooperation enough relative to the one-time gains from defection, cooperating is an equilibrium.
The key requirements for cooperation to hold are that (i) each party’s resources and strategic capabilities are sufficiently observable, (ii) growth is balanced26

in such a way that no single act of defection can be decisive, and (iii) both parties are patient enough to care about future cooperation. This is roughly the idea behind many arms control verification agreements, like the SALT and START treaties for limiting nuclear arsenals, which relied on mutual verification with satellites and on-site inspections.
Whether (ii) is true depends on the offence-defence balance of technologies either side could use to achieve a DSA, and prevent the other from achieving one. Instructively, the Anti-Ballistic Missile treaty also limited defensive capabilities against nuclear weapons: to preserve mutual vulnerability, thus making defection less attractive. This suggests it could initially be in both parties’ interests to keep open a way of disabling the other’s capabilities, at the cost of their own capabilities. In theory, one actor would then need to establish a much larger gap in capabilities before it’s in their interest to fully defect from the deal. One version of this idea is developed in “Mutual Assured AI Malfunction” (Hendrycks et al. 2025).27

But maintaining a balanced path of sustained cooperation through mutual deterrence could be too fragile. In particular, the key technologies required for one actor to break away from a binding deal by achieving a decisive strategic advantage could be relatively easy to amass in secret, and correspondingly difficult to monitor.

Staged renegotiation

Relatedly, the terms of the deal itself could be renegotiated in stages. The initial deal could only concern the use of resources likely to be useful within (say) 20 years, with an explicit commitment to negotiate beyond that point nearer the end of the period. One advantage of a time-limited agreement is to reduce the downside risk from uncertainty about signing a more all-encompassing deal.
For example, each side might be unsure about how its collective ‘values’ or decision-making process will evolve, especially if human decision-makers responsible for the deal don’t fully trust or understand the AI advice they’re getting. The original 1959 Antarctic Treaty included a provision for review after 30 years, partly because the signatories didn’t know how valuable Antarctic resources would turn out to be for them. Arguably, that made the treaty feasible in the first place.
One issue with a fixed-term agreement is if one party is more powerful at the point of renegotiation, then the new terms will reflect that sharper asymmetry in power, losing much of the ex-ante gains that could have been achieved with a longer-run deal. Also, when both parties know the deal is soon set to expire, they might choose to race for the strongest bargaining position for renegotiation, and the original deal might then unravel.28

Self-commitment devices

Transformative AI could enable much more binding kinds of credible self-commitment, making both fixed-term and iterative agreements easier.
Each side could place decision-making authority in the hands of AI systems programmed to retaliate, even when retaliation would no longer make sense as soon as it fails to deter an initial defection. This is similar to the “Dead Hand” mechanism for mutually assured destruction, but far less crude, and able to meet small instances of defection with an appropriately sized response.
Each party could use similar self-commitment mechanisms to bind themselves to the terms of the deal itself, not just to a policy of retaliation for the other party defecting. For example, both parties could install the terms of the deal into AI-controlled systems able to inflict enough damage on themselves to disincentivise defecting from the deal. They could then agree on mutual inspection and verification to ensure those systems are working as described.
The same observability worries apply here as in the “iterative growth” proposal, just one layer removed: although one actor’s own AI self-commitment system could be designed to spot any attempt at breaking the deal, it could be difficult for the other actor to verify that’s what the other actor’s self-commitment system is actually doing.

Third-party commitment and merging

Rather than building self-commitment devices with mutual inspection and verification, both parties could contribute to jointly building a third-party enforcing entity. Concretely, the US and China could create an impartial military with enough force to deter defection from a deal, controlled by AI systems bound to enforce the terms of the deal, and otherwise maintain order.
A treaty-enforcing “world police” hasn’t been at all feasible historically: if the enforcer is powerful enough to be credible, then capturing or influencing it becomes the highest-stakes game in town. But that could be far less relevant for an enforcer controlled by AI systems, since the motivations of the controlling AI systems — being jointly designed — could in principle be tamper-proof and easy to verify.
Beyond mutually building a third-party enforcing agency, both parties could begin to do something more like merging: where they have overlapping or duplicated functions, agreeing to centralise those functions, so they are increasingly reliant on each other.
Relatedly, Goldstein and Salib (2025)29

propose that frontier AI itself — the critical technology sustaining an intelligence explosion — be jointly developed by major powers who might otherwise face incentives to race.
In the 20th century, early member states of the European Union progressively merged functions like trade policy, competition regulation, currency (for eurozone members), and freedom of movement. One explicit goal was to become more mutually dependent, to make conflict less thinkable.
Reflecting on the possible mechanisms for enforcement, I end up fairly unsure about whether such a high-stakes resource-sharing deal would actually be possible, but I’m more hopeful than not. It’s worth noting these mechanisms can work together: the deal could be enforced by some mix of iterative tit-for-tat, self-commitment devices, and merging or third-party enforcers. On the other hand, even if they would work in theory, the most promising AI-enabled enforcement mechanisms might just seem too strange, unfamiliar, or untested to the human decision-makers in charge.

Issues with contracting over future income

The previous section considered how early deals between major powers face enforcement problems because no higher authority exists. Deals between private actors don't face this problem — in principle, they can rely on courts and contracts.
But they face a different challenge. As mentioned, one reason to think that early deals before an intelligence explosion could be a big deal is that an intelligence explosion could generate a massive amount of new wealth, and newly valuable resources — like shares of sunlight, or other stars.30

But when private actors are trying to make a deal about future resources, they may need to commit to handing over resources they don’t yet own, and such agreements may be very hard to enforce. So the feasibility of early deals about future resources depends on exactly how those resources are distributed.

Auctioned resources

One possibility is that future resources are auctioned to private actors, and can then be traded. As long as the auction is fair and frictionless, then wealth at the time of the auction can be converted into ownership of stars: twice as many dollars before the auction can buy you twice as many stars. We can also assume wealth at the time of the deal translates fairly straightforwardly to expected wealth at the time of the auction. The result is that two deal-makers can contribute the same amount to a wager, and the winner can expect to buy twice as many stars than they could with what they wagered. This makes deals over future resources relatively easy: for instance, by placing staked wealth in escrow.
There’s historical precedent for newly valuable resources being auctioned to private actors. In the early 20th century, no one could legally own fractions of the electromagnetic spectrum. In fact, when the idea of electromagnetic spectrum auctions was first seriously proposed,31

it was treated as a joke.32

But the first spectrum auction in the US raised more than $600 million, and subsequent auctions have raised tens of billions of dollars. And today spectrum licenses are routinely bought and sold on secondary markets. Hypothetically, a cell carrier startup (looking for entry into the market) could have made a wager with an incumbent cellular carrier before it was clear whether spectrum auctions would happen by some date, with the startup winning if auctions went ahead, and the incumbent winning otherwise. I’m not sure such deals were ever made, but that would have counted as a deal about the “material stakes”.
In the auction case, there are still questions about figuring out workable resolution criteria, whether enforcing mechanisms survive between making the deal and realising it, and so on. But the deals are structurally similar to agreements and contracts people make all the time today.

Distributed resources

Another possibility is that future resources are distributed after the window for making a deal closes, where deal-makers can’t directly control how they are distributed. For instance, imagine the US credibly commits to defending and administering some number of stars as property of US citizens, and then distributes the same fractional claim to those stars to every US citizen, in the form of a “star credit” certificate which they can freely sell and trade.
If these star credits ultimately account for most wealth in the US economy, then current wealth becomes a small fraction of total wealth after the windfall. Deals staked in current wealth would be correspondingly small. The only way to make deals at the right scale is to commit future income — agreeing now to spend some share of your post-windfall wealth according to the terms of a deal.
However, it’s not clear that such agreements to commit uncertain future income are legally enforceable. In most common law jurisdictions, you generally cannot assign rights to future income expected to arise from a contract that doesn’t yet exist — such as a possible distribution of star credits that hasn’t yet been confirmed.
Moreover, even if a future distribution of star credits is confirmed by contract, signing away guaranteed future income is still highly restricted for paternalistic reasons.33

For deals which require uncertainty about whether rights to future resources will be disbursed at all, I think it’s fair to conclude that those deals would not be recognised by common law courts today, so they’d have to be made and enforced by private means, outside of the legal system.

Untradeable resources

Finally, future resources might just not be tradeable at all, even once they’re being used for valuable purposes. It could be that property rights are recognized, but they’re inalienable (they can’t be transferred). Or it could be that future resources aren’t treated as property: maybe they’re collectively owned and used, or administered by a public agency, or by some kind of AI-mediated collective decision-making process.
This scenario obviously rules out deals where one actor wants to own more future resources given different outcomes.
In this case, though, deal-makers could still potentially trade other kinds of influence over how future resources are used. For instance, if future resources are collectively administered, individuals might trade voting rights or commitments to use their political influence on each other’s behalf.
These arrangements face their own problems. Selling votes is already illegal in most democracies,34

on the grounds that political influence should be distributed equally, not according to wealth. Similar restrictions would presumably extend to any arrangement that effectively allows future political influence to be bought and sold.35

That said, the deals I’m considering here are more like (outcome-contingent) vote swapping, where no material wealth changes hands. Standard vote swapping between voters (“I vote for your candidate in your state, you vote for mine”) is effectively legal in the US,36

and of course it’s standard between legislators (“I vote for your bill, you vote for mine”). I’m not aware of precedent for trading influence contingent on uncertain outcomes, but I’d guess it would be allowed. Moreover, money obviously still influences political outcomes, so two parties could agree on how to spend on political influence.

In sum: if space resources (or any other ownable future asset)37

are auctioned and freely tradeable, early deals can plausibly influence who controls vast resources in the future, albeit with some practical challenges like figuring out unusual resolution criteria, or general social disruption undermining the enforcing entity. If resources are distributed as windfalls, early deals look less feasible, since they need to be staked on future income, which the law doesn't clearly support. I’m unsure what privately enforced alternatives could look like, but I place some weight that they will exist.
Finally, if resources aren’t treated as property at all, deal-makers are limited to trading political influence, which is more constrained but probably still quite possible. That would rule out many of the self-interested gains. Overall, the kinds of deal I’m more interested in might not be legally enforceable, but seem unlikely to be legally prohibited.
Early deals over future resources under ignorance about an intelligence explosion are going to be complicated and probably controversial, but if there’s enough will, I lean towards thinking they are feasible. Finally, note that a deal doesn’t need to be guaranteed to work. If it’s attractive enough, both parties can accept some chance that it falls apart, especially if there’s a fairly robust way to re-collect whatever was staked.

Are early deals desirable?

So far, I’ve presented the case for enabling early deals about uncertain outcomes around an intelligence explosion, from an impartial perspective;38

and I’ve asked whether they’re feasible to enable. The question now is whether these deals have downsides, and under what circumstances they override the case in favour.

Deciding before we’re ready

The main worry is that even if people make deals which they feel good about at the time, we would be trusting each other to make extremely consequential decisions before we’re wise enough.
Some people will make decisions which they themselves later come to regret even as ex ante decisions. For instance, children are not allowed to sign long-term employment contracts, take on debt, get married, or get tattoos: not because they never want to do these things, but because we suspect they don’t fully appreciate what they’re giving up. We generally think it’s good that society doesn’t allow some of these options.
To be clear, many people will regret making the deals I’ve been discussing ex post: wagers and insurance contracts have a clear winner and a loser in hindsight, and both parties know this entering into them. The worry is that either party to the deal might become wiser or better-informed enough to realise, even setting aside how things turned out, that their earlier reasoning was flawed. For example, someone might trade away their rights to future resources because they didn’t think they’d value having many more resources than they currently have, but later realise all the ways they could have spent those resources for moral, ideological, or self-interested reasons.
This worry applies to any way to trade away future resources early on, whether or not the trade is made behind the veil of ignorance, just as long as the trade is made before reflecting adequately. For instance, one person might sell their rights to almost all their future resources for earlier consumption, and later regret being so impatient;39

or some just might not understand their eventual value, or feel pressured or confused into selling. The worry bites especially hard when people can’t as easily earn back wealth through their own labour;40

since bad early decisions are practically unrecoverable.
Furthermore, we should be careful not to only imagine moral motivations which we endorse. Because people are making deals behind the veil of ignorance about moral knowledge, they might make such deals based on moral mistakes which, with the benefit of AI-driven intellectual progress, they come to regret.
Another issue is that early-on, understanding of the value of claims to future resources might be concentrated among the wealthy and well-connected, in a way which is procedurally unfair or results in worse outcomes. In the 1990s, the Russian government distributed vouchers equally to all citizens, each representing a share of national wealth. Most people, struggling through hyperinflation and unfamiliar with equity markets, sold their vouchers almost immediately for trivial sums. The result was extreme concentration of wealth among a small number of insiders.

Guardrails on early deals

Of course, whether to enable these deals isn’t an all-or-nothing question. There could be ways to preserve much of the impartial upside of early deals, while mitigating the paternalistic worries above.
One approach may be to impose requirements on who can enter these early deals.41

You could imagine a fairly rigorous test, where only parties who can demonstrate real understanding of the stakes and meaning of a given deal would be allowed to enter it. Ideally such a test would screen for something like wisdom, good judgement, and reflectiveness over impulsiveness.
Of course the difficulty is figuring out a broadly acceptable way to cash those things out. Consider contexts like organ donation, euthanasia, and some forms of elective surgery, where reasonable people disagree about how paternalistic to be, and the question of where to set the bar for understanding and consent is hotly argued.
A second idea is to set up impartial mechanisms for voiding deals after they have been made, if it becomes clear that at least one party seriously misjudged the terms ex ante, given what was knowable at the time. The basic principle that sufficiently lopsided or ill-conceived contracts can be unwound is well-established: courts can release a party on grounds of unconscionability, duress, or mistake.42

But these doctrines only apply in fairly extreme cases: poor judgement alone isn’t enough.43

But early deals over future resources could be voided on much more expansive and opinionated grounds. The questions are who adjudicates, by what standard, and how do they distinguish genuine ex ante mistakes from mere ex post regret. But I see two reasons to think more substantial reasons for voiding contracts could be feasible. First, an intelligence explosion will be a period of fast-increasing wisdom and understanding, meaning society might develop more substantial protections against making bad deals, which they could retroactively apply. Second, AI could help judges weigh claims more fairly, by drawing on a collective understanding of “wisdom”.
Beyond retroactive protections, parties could also include catch-all voiding mechanisms into the deals themselves, by agreeing to void their deal if some agreed-on future model decides that the deal they are currently making was poorly conceived. By default, that would still be a voluntary choice, which doesn’t help those most vulnerable to making catastrophically imprudent deals in the first place. But the voluntary version still seems like an easy win, and broadly sensible voiding mechanisms could be made a requirement for (certain or all) deals, especially deals over future income.
Perhaps a more robust approach is just to cap the fraction of future resources that can be committed through early deals, at least early on. I see two ways to do this: capping the fraction of resources which can be owned by anyone; or capping the tradeable fraction of each person’s resources. If society agrees that no individual can stake more than, say, 10% of their eventual claim on future resources,44

then the worst-case cost of bad decisions is bounded, while still leaving room for the gains from trade.
Finally, restrictions could be placed not on the deals themselves but on how the resulting resources can be used, to prevent outcomes that are collectively harmful even if individually preferred, like building weapons or causing what others view as major harms or wrongs. In fact, early deals about future resources could be explicitly limited to the questions of ownership I’ve been discussing, and not allowed to extend to plans about how to use those resources.
None of these safeguards is going to exactly sort between impartially bad and impartially good deals. A regime with strict requirements of understanding, cap on stakes, and adjudication mechanisms, would lock out many people from making veil-of-ignorance deals who wanted to. But I do think we should be fairly cautious about the downsides of early deals on the eve of an intelligence explosion, and since many such deals don’t currently seem legal anyway, I lean towards an attitude of ruling in the most promising and robust versions gradually. But I am confident that enabling some deals, with the right guardrails, seems much better than allowing none.

Conclusion

I've considered three main kinds of deals which hinge on shared uncertainty about the future. Roughly speaking: uncertainty about the share of the pie (especially between major powers), about the size of the pie, and about the moral stakes. In each case, all parties to the deal improve their own prospects, or make the future overall better, according to their own lights. Since those gains come from trading between possible outcomes, the window to make them closes once the period of shared uncertainty ends. We make them early, or never.
There’s a very general case for welcoming these early deals: if some arrangement improves the ex ante prospects of everyone involved by their own lights, and makes nobody worse-off, that seems like a generally compelling reason to prefer the arrangement from an impartial point of view.
More specifically, each kind of deal has its own impartial appeal. Resource-sharing deals between great powers could prevent massive imbalances in power and preserve social and political diversity; keeping open the possibility of further moral trade down the line. Deals over the size of the pie let those more motivated by owning future resources to own more in expectation, in return for insuring those with less appetite for risk against losing their wealth. Other deals over the material stakes could be morally motivated, giving more expected influence in worlds where people want it. And similar deals could be struck over existing theoretical kinds of uncertainty. For instance, if there is a correct moral view, people who are more motivated to act on it gain more leverage when it turns out to be correct. So early moral-stakes deals tend to shift resources toward people acting on the true moral view, whatever it is. And again, the window of opportunity to strike each deal closes when the uncertainty resolves.
Against these upsides, the main case against enabling deals behind the veil of ignorance is that people make unwise decisions — making hugely consequential and hard-to-reverse decisions based on reasons they or others will later recognise as flawed. Guardrails could help, like accreditation requirements, caps on the fraction of future resources that can be staked, mechanisms to void deals that were clearly unwise at the time they were made, and floors on possible losses. But none seem totally watertight, and they could also block deals which should happen.
A tentatively promising approach would be to allow at least a small subset of deals with the highest impartial upsides. In particular, I’m most excited about making possible power-sharing deals between great powers, while both powers are more uncertain about their relative share of resources in the future, since the likely result would be a less concentrated distribution of power and more opportunity for moral trade (even if conflict is prevented anyway). Morally motivated deals — both over material and moral stakes — also stand out to me as especially worth enabling from an impartial point of view. In both cases, deals staked in current resources seem relatively safe; deals committing large shares of future income or political influence are going to be higher-variance.
Overall, I think there is a surprisingly strong case in favour of enabling at least some early deals behind the veil of ignorance which specifically depend on early uncertainty, even if you think it’s generally important to prevent early kinds of lock-in, and delay major decisions.
It also seems likely that actors will want to make such deals during the early stages of an intelligence explosion, before the key normative and empirical uncertainties are resolved. If that's right, the question isn’t just whether to enable early deals, but how to shape their terms.

Appendix

Deals about the relative share of resources

Here’s a toy example of a risk-sharing deal. There are two agents, A and B, and two equally likely states of the world:
State 1 (p = ½)State 2 (p = ½)
A’s resources10025
B’s resources25100
We assume both agents agree on these probabilities and payoffs, and both have the same concave (risk-averse) utility function in resources u(r)=ru(r) = \sqrt{r}. Without any deal, each agent’s expected utility is then:
E(U)=0.5100+0.525=0.510+0.55=7.5\mathbb{E}(U) = 0.5\cdot\sqrt{100} + 0.5\cdot\sqrt{25} = 0.5\cdot 10 + 0.5\cdot 5 = 7.5
Now suppose they agree to split the resources equally, whatever state they end up in. In both states, total wealth is 125, so each gets 62.5 units of wealth whatever happens. With the deal, each agent’s utility is:
E(Upool)=62.57.91\mathbb{E}(U_{\text{pool}}) = \sqrt{62.5} \approx 7.91
Both agents are strictly better off by agreeing to the deal.
It turns out it isn’t important for both agents' resources to be negatively correlated across states. Whenever two or more risk-averse parties face imperfectly correlated risks to their resources, a mutually beneficial risk-sharing deal always exists. This is due to Jensen’s inequality. For any concave function uu and random non-constant variable RR:
u ⁣(E[R])  >  E ⁣[u(R)]u\!\bigl(\mathbb{E}[R]\bigr) \;>\; \mathbb{E}\!\bigl[u(R)\bigr]
In words: a risk-averse agent prefers receiving their expected resources with certainty over a gamble with the same expected resources. Resource-sharing deals do exactly this: they reduce variance in resources without changing expected resources for both agents, raising expected utility for both.

Great power resource-sharing

For the sake of concreteness, here’s an example of a risk-sharing deal between two great powers.
Without a deal, assume the US achieves a decisive strategic advantage (DSA) just in case China doesn't. The US has probability pp of achieving a DSA without a deal, and China has probability 1p1 - p. The DSA winner controls 10,000 units of resources; the loser falls back to 1 unit. Both have log utility in resources: u(r)=log10(r)u(r) = \log_{10}(r).
Expected utility without a deal is then:
E(UUS)=plog10(10,000)+(1p)log10(1)=4p,E(UChina)=4(1p)\mathbb{E}(U_{\text{US}}) = p \cdot \log_{10}(10{,}000) + (1-p) \cdot \log_{10}(1) = 4p, \qquad \mathbb{E}(U_{\text{China}}) = 4(1-p)
Each side’s certainty equivalent is the guaranteed amount of resources that would make them indifferent to the no-deal gamble:
CEUS=104p,CEChina=104(1p)\text{CE}_{\text{US}} = 10^{4p}, \qquad \text{CE}_{\text{China}} = 10^{4(1-p)}
When p=0.5p = 0.5, both certainty equivalents are 102=10010^2 = 100 units. Since both sides only need any amount more than 100 guaranteed resource units to prefer a deal over the no-deal gamble, a resource-sharing agreement only needs to split any amount over 200 units to be preferred by both actors; leaving a surplus of 9,800 units to be divided however they negotiate (including to be spent towards moral public goods).
The minimum total resources needed for a mutually preferred split to exist is therefore just the sum of certainty equivalents. For a given pp, this is just:
CEUS+CEChina=104p+104(1p)\text{CE}_{\text{US}} + \text{CE}_{\text{China}} = 10^{4p} + 10^{4(1-p)}
This sum is smallest when p=0.5p = 0.5 (just 200 units) and grows as the odds polarise, reaching 10,000 — the entire pie — as pp approaches 0 or 1.
A natural measure of the combined gains from trade from a deal is the surplus: the total resources minus the sum of certainty equivalents. This is easy to work out:
Surplus=10,000104p104(1p)\text{Surplus} = 10{,}000 - 10^{4p} - 10^{4(1-p)}
This is the pool of resources left over after using the minimum resources needed for a resource-sharing deal which both powers prefer to seeking a DSA. Put differently, it’s the total amount both sides would collectively be willing to pay to secure a deal.
Importantly, the surplus is largest when p=0.5p = 0.5, and approaches zero when the odds polarise. We can see just how much the surplus shrinks as both sides become more confident that one particular side would succeed in securing a DSA, in the table below.
p(US achieves DSA)Surplus from dealSurplus as % of total
0.509,80098.0%
0.906,01660.2%
0.953,68836.9%
0.998798.8%
As you can see, when p is closer to 0.5, the surplus is larger and more equal deals are feasible. At p=0.5p = 0.5, the range of mutually acceptable splits runs from 1% to 99%.
The US prefers an even split of resources whenever log10(5,000)4p\log_{10}(5{,}000) \geq 4p, that is whenever p0.925p \leq 0.925, and symmetrically for China. So a 50/50 split is acceptable to both parties for any pp between about 8% and 92%.
For 0<p<10 < p < 1, some mutually preferred deal always exists, but it has to become increasingly lopsided.. At p=0.99p = 0.99, the US demands at least ~91% and China demands at least ~0.01% of the available resources, so a deal still exists — but only barely, and overwhelmingly on the US's terms.45

Will both sides prefer an early resource-sharing deal?

When the odds are close to even, it’s intuitive that both sides would prefer to strike a deal earlier rather than later, since the surplus is largest now and can only shrink. But if pp is already far from 0.5 — say, the US already has a 80% chance of securing a DSA — it’s not obvious that both sides will prefer to make a deal early, rather than wait.
Under some natural assumptions, it turns out that both sides always prefer to make a deal early rather than late, and the expected surplus always shrinks with time. So, all else equal, an impartial onlooker hoping to maximise the surplus should also hope for an early-as-possible deal.
Since I assume both sides are rational and Bayesian, I assume pp is a martingale: both sides’ shared current best estimate of tomorrow’s probability is just today’s probability:
E[pt+1pt]=pt\mathbb{E}[p_{t+1} \mid p_t] = p_t
If both sides expected pp to drift in some particular direction, and they’re updating in a Bayesian way on common information, they’d have already updated to account for that.
Recall that the surplus from a resource-sharing deal is:
S(p)=10,000104p104(1p)S(p) = 10{,}000 - 10^{4p} - 10^{4(1-p)}
This function is strictly concave in pp. For all pp, its second derivative is:
S(p)=16ln2(10)(104p+104(1p))<0S''(p) = -16\ln^2(10)\bigl(10^{4p} + 10^{4(1-p)}\bigr) < 0
Now we can combine these two facts. Since pp is a martingale, the expected future value of pp equals its current value. But since SS is concave, gains in surplus from pp moving toward 0.5 with some probability are always more than compensated by the losses from pp moving the same distance away from 0.5 with corresponding probability. So the expected surplus always shrinks over time, by Jensen's inequality.
E[S(pt+1)pt]S(pt)\mathbb{E}[S(p_{t+1}) \mid p_t] \leq S(p_t)
In words: the expected future surplus is always weakly less than the current surplus, regardless of the current value of pp. The surplus is a supermartingale, meaning it drifts downward over time in expectation. This means that even if one side is already well ahead, both sides should expect the gains from trade to shrink from here.
So, an impartial onlooker hoping to maximise the total surplus should always prefer the earliest possible deal, regardless of the current odds. However, the original intuition — that the loser might prefer to delay a deal to give time for the odds to swing back in their favour — was wrong. Each side’s no-deal option (racing for a DSA) is a convex function of pp, in certainty-equivalent terms. Thus, it improves in expectation over time. But the disadvantaged side (here China) gains relatively less from delay, while watching its share of a shrinking surplus shrink faster than its no-deal option improves from waiting. So fixing how the surplus is split (e.g. evenly), it’s the disadvantaged side that prefers to lock in a deal early, and the winner that has reason to hold out.
However, this doesn’t mean deals will actually get delayed. Since the surplus shrinks in expectation from waiting, the disadvantaged side can offer the advantaged side enough additional share of the surplus to compensate the advantaged side for giving up the improved leverage they’d expect from waiting. Since the surplus that would otherwise be destroyed funds the side payment, both sides can be made better off by dealing now (by the Coase theorem). The advantaged side receives enough to prefer an early deal over holding out, and the disadvantaged side still prefers its smaller share of today’s larger surplus to its expected share of tomorrow’s smaller one. Both parties should prefer to strike early.
Thus, the motivation for making a resource-sharing deal early doesn’t depend on the odds being close to even. At any point before one side has definitively secured a DSA, there’s always a cost to delaying a resource-sharing deal, both selfishly for both powers, and from the impartial perspective of maximising the surplus.

Deals about the material stakes

Deals arising from differences in risk-aversion

In the examples of resource-sharing deals, parties to the deal were uncertain about who gets the bigger share of a fixed total pie. Here, the size of the pie itself is uncertain. Unlike the resource-sharing example, the gains from trade come from the differences in risk aversion between the parties to the deal.
Suppose there are two equally likely states of the world:
Big pie (p = ½)Small pie (p = ½)
Total resources20020
Annie’s share10010
Bob’s share10010
Annie is risk-neutral, while Bob is risk-averse:46

uA(r)=r,uB(r)=log10(r)u_A(r) = r, \qquad u_B(r) = \log_{10}(r)
Without a deal:
E(UA)=55,E(UB)=0.5×log10(100)+0.5×log10(10)=1.5\mathbb{E}(U_A) = 55, \qquad \mathbb{E}(U_B) = 0.5 \times \log_{10}(100) + 0.5 \times \log_{10}(10) = 1.5
Bob’s certainty equivalent is 101.531.610^{1.5} \approx 31.6 units of resources — his risky gamble between 100 and 10 is only worth a guaranteed 31.6 units to him. Annie, being risk-neutral, values the same gamble at 55 units. Thus, Bob would happily give up some of his upside in the big-pie state to insure against the small-pie state. Annie, who doesn’t mind variance, is happy to insure Bob for a premium. Concretely, Bob could then pay Annie 30 units of resources in the big-pie state; and Annie pays Bob 8 in the small-pie state.
Big pie (prob ½)Small pie (prob ½)
Annie’s resources1302
Bob’s resources7018
Annie’s expected utility increases from 55 to 66. Bob’s expected utility also improves:
E(UB)=0.5×log10(70)+0.5×log10(18)1.55>1.5\mathbb{E}(U_B) = 0.5 \times \log_{10}(70) + 0.5 \times \log_{10}(18) \approx 1.55 > 1.5
Thus, Bob’s certainty equivalent rises from 101.531.610^{1.5} \approx 31.6 to 101.5535.510^{1.55} \approx 35.5. And Annie’s rises more straightforwardly from 55 to 66 units of resources.
In general, whenever two parties differ in risk-aversion and face shared uncertainty about the size of the pie, the less risk-averse party can “sell insurance” to the more risk-averse party. The risk-averse party pays a premium (gives up expected resources) in exchange for reduced variance; the risk-neutral party accepts more variance in exchange for higher expected resources. Both end up better off in expected utility terms, because they value the same variance differently.47

Morally motivated deals

The previous deal worked because Annie and Bob effectively valued risk differently. But parties can also gain from trade when they value resources differently across states of the world, even if they have the same general risk-aversion with resources.
Suppose there are two equally informed parties, Ava and Ben, and two states of the world:
  • High stakes (prob 10%): a totalist axiology turns out to be correct, and resources can do 100x more moral good per unit.
  • Low stakes (prob 90%): it doesn’t, and resources do ordinary good.
Both start with 50 units in each state. Ava only cares about expected moral impact, and she values each unit of resources at 100x in High stakes and 1x in Low stakes. Ben has no strong moral view and values resources equally in both states. Both are risk-neutral over their respective objectives:
UA=qmrA,1+(1q)rA,2,UB=E[rB]U_A = q \cdot m \cdot r_{A,1} + (1-q) \cdot r_{A,2},\quad U_B = \mathbb{E}[r_B]
Where q=0.1q = 0.1 is the probability of High stakes and m=100m = 100 is the moral multiplier in High stakes, for Ava.
Without a deal:
High stakes (prob 10%)Low stakes (prob 90%)
Ava’s resources5050
Ben’s resources5050
So Ben’s expected utility is 50, and Ava’s is 0.1(10050)+0.950=5450.1 \cdot (100 \cdot 50) + 0.9 \cdot 50 = 545.48

But Ben could sell all 50 of his High stakes resource units to Ava, in exchange for 6 of Ava’s Low stakes units (a small premium over the “fair odds” price of 50/95.650/9 \approx 5.6).
High stakes (prob 10%)Low stakes (prob 90%)
Ava’s resources10044
Ben’s resources056
Now, Ben’s expected utility is ≈ 50.4, and Ava’s expected utility is ≈ 1,039.6. The deal is mildly good for Ben, and overwhelmingly good for Ava.

Limits on the gains from morally-motivated deals

In this two-person example, note that Ava’s gains are capped at roughly 2x: Ava can at most double her High stakes holdings by buying all of Ben’s. This is relevant for moral trade between two (or a small number of) major powers: even if one power values resources vastly more in one state compared to the other power, they can’t vastly leverage their influence in that state if they’re roughly as wealthy as the other actors combined.
But in a large market of parties who mostly don’t share Ava’s moral motivations, Ava can sell all her Low stakes resources and go all-in on High stakes. In that case, she concentrates her entire endowment into an event that happens with probability q=0.1q = 0.1, multiplying her Low stakes holdings by up to 1/q=101/q = 10x. Her expected moral impact rises by a similar factor.
When Ava is a small enough actor compared to the rest of the market, and her moral motivations are rare enough, the bottleneck to how much leverage Ava can get in High stakes worlds is 1/q1/q (the inverse probability of the morally important state); not the moral multiplier mm. Even if resources are 1010010^{100} times more morally valuable in High stakes according to Ava, her gains from moral trade are capped at roughly 10x, because that's the most she can concentrate her portfolio.
More precisely, if Ava sells all her State 2 resources at market prices and buys State 1 resources, her expected impact increases by a factor of:
mqm+(1q)\frac{m}{qm + (1-q)}
Again, the gains from moral trade are bounded by how unlikely the morally important state is, not by how important it is. And note this is generally true of moral trade — the probabilities could be interpreted as credences in moral theories which aren’t moved by reflection during an intelligence explosion.
Finally, note that moral public goods do not face this problem. Morally motivated trades behind a veil of ignorance still look potentially very important, but the gains seem more capped than the potential gains from moral public goods.

Early deals and the veil of ignorance

I’ll finish with some more sketchy thoughts about the possible analogy between early deals under uncertainty about an intelligence explosion, and the idea of a “veil of ignorance” from Harsanyi (1955) and later Rawls (1971).
Harsanyi (1955) showed that if agents choose social arrangements from behind a complete veil of ignorance — not knowing which position in society they will occupy — and if they are expected utility maximisers, then the arrangement they unanimously prefer maximises a weighted sum of individual utilities. Rawls’ setup does not imply the result maximizes welfare, since he argues agents behind the veil would apply maximin reasoning rather than expected utility maximisation.
Early deals about an intelligence explosion could be instructively similar to the veil of ignorance thought experiment: many parties are all uncertain about how they will fare, so their preferences over deal terms are shaped by improving their overall prospects across all possible outcomes. Assuming people behave roughly like expected utility-maximisers, then the more symmetric the ignorance, the more the resulting agreements resemble welfare-maximising arrangements.
By contrast, once the veil lifts and parties learn their relative positions after an intelligence explosion, then people establish more common knowledge of each other’s bargaining power. Outcomes are then more determined by threat points and outside options rather than by aggregate welfare, where they need not maximise total welfare, and typically won’t when power is distributed unevenly (nor would they most maximise the welfare of the worst-off, per Rawls’ setup).
This gives a further reason to prefer early deals beyond the shrinking-surplus argument: deals struck under greater uncertainty don't just capture more surplus in terms of certainty-equivalent resources, but they might tend to distribute those resources in a way which makes everyone better-off overall.
The analogy is imperfect. Parties to the deals I have been discussing face a partial veil, not a complete one. And it’s contested whether the Harsanyi result can be interpreted as genuinely utilitarian without assuming interpersonal utility comparisons.49

But the approach of an intelligence explosion is plausibly the last period of deep shared ignorance about very long-lasting and consequential outcomes.

Footnotes

Citations

Search

Search for articles...