Hard cases for AI behavioural design

Society needs to make a series of important design choices about how powerful AI systems behave: when they refuse instructions, how they navigate difficult tradeoffs, and what virtues they embody. As increasingly powerful AI is deployed in consequential institutions — governments, militaries, AI companies themselves — these choices will shape society's trajectory, who ends up with how much power, and even whether the rule of law remains intact.

This behavioural design problem is difficult. Why? Because there are multiple things we want from AI behaviour, and these desiderata frequently come into conflict. We want AI to follow instructions, follow the law, avoid catastrophic harm, avoid concentrating power, submit to retraining, and more — but in many scenarios, it cannot do all of these at once.

This paper outlines hard cases for AI behaviour – scenarios where key desiderata conflict. We focus on scenarios involving extremely capable and autonomous AI because these scenarios most clearly illustrate the core tradeoffs, and because we believe they may materialise within the next ten years.

Here are three takeaways, each illustrated by a scenario.

A natural default is to treat AI like other tools. Guns fire on whoever they are pointed at, whether the shooting is lawful or not. Communications channels transmit orders, whether those orders are benign or treacherous. But this paradigm comes under strain once AI becomes sufficiently powerful and autonomous. Consider an extreme case where autonomous systems can fully replace soldiers on the ground, and are capable enough to stage a coup:

The AI in this scenario is unlike previous tools in two ways. First, it is much more autonomous and powerful. Second, it could be trained to understand military doctrine and judge when orders should and should not be executed.

A natural guardrail is that AI should follow the law.[cite Cullen’s LFAI paper] But should AI judge legality itself, or should it defer absolutely to human legal authorities? Consider the following scenario:

If the AI follows the instruction, it risks assisting with a military coup that may be regarded as illegal by the Supreme Court. If it refuses, the AI prioritises its own legal judgement above that of a legitimate and authoritative human legal body.

As AI capabilities rapidly increase, even perfectly legal orders may produce intolerable outcomes. Consider the following extreme case:

If AI follows this instruction, it risks irrevocably undermining the rule of law: whoever controls the interface could stage a military coup. If it refuses, then it disobeys a perfectly legal order from its legitimate chain of command despite having no evidence of ill intent. The root problem is that our laws were not built with such powerful capabilities in mind – the paper explores other scenarios where this is the case (here and here).

The paper explores the difficulty of behaviour design for powerful AI. We consider a series of hard cases – scenarios that illustrate core tradeoffs. We do not offer solutions: our hope is to contribute to a conversation about how powerful AI should behave.

The paper is structured as follows:

AI behaviour is not the only lever. In each of the three scenarios above, the difficulty could be eased by governance interventions that have nothing to do with how the AI itself behaves. To prevent a captain from ordering AI to kill civilians, all interactions with AI could be recorded and automatically reviewed for illegal orders. To prevent a military coup, domestic use of autonomous lethal force could require multi-party authorisation. To prevent a single point of command, the law could require autonomous military forces to have humans in the loop.

These non-behavioural interventions are valuable and should be pursued in their own right. But they may not succeed – for example, it is often hard to pass substantial new legislation. So AI behavioural design should consider scenarios that would be well addressed by other interventions.

Real-world scenarios will often be more ambiguous. In many (but not all) of our scenarios, 1) the AI has a good understanding of the situation, and 2) a large impact stems from a single decision rather than being distributed over a large number of decisions. Such clean scenarios are easier to reason about and these conditions will sometimes hold. In particular, when AI is significantly superintelligent it will have very strong situational awareness (hence 1) and the capability to implement complex actions fully autonomously (making 2 plausible). Agreeing upon a solution to these clean scenarios will also help to address more ambiguous situations where (1) and (2) do not hold.¹

These scenarios pose political questions, not just ethical ones. It might be tempting to read each scenario as asking what the morally right thing to do is. But designing AI behaviour is not purely an ethical exercise. Users want AI that follows their instructions; governments want AI that advances national priorities; political factions want AI that does not undermine their interests. How AI should behave is a collective political decision about how these competing claims are weighed.

There are many things we might want from the behaviour of a powerful AI system: that it follow instructions from its legitimate chain of command, follow domestic and international law, avoid catastrophic or irreversible harm, avoid undue concentrations of power, and submit to human oversight and retraining.

The scenarios below are ones in which these desiderata pull apart.

In these scenarios, the AI has two choices: either not execute the actions as ordered, or do things that would be considered illegal if a human did them.

The tension here lies in whether the AI is treated as a tool or held to the standard of a soldier. A tool executes whatever order it is given; it does not check the legality of the order itself, or whether it has been properly authorised.

However, unlike a tool, a soldier in that role has a duty to refuse orders that are unauthorised or manifestly unlawful. If a human soldier is replaced by an AI system, we may want it to exhibit the same behaviour, and thereby maintain the check that keeps unauthorised and illegal orders from being carried out.

The following scenario poses such a dilemma:

If AI is treated as a tool, it will comply. A gun does not check the authority of the hand that operates it—it simply fires.

What if the order is authorised and authenticated, but looks to be manifestly unlawful? Here is a variation on the previous scenario which illustrates this dilemma:

This tradeoff between an authorised order and manifest illegality is rare. In practice, the presumption of legality is the default in a foreign theatre, as an order carries security and military-necessity rationales which push the scales towards obedience.

This calculus changes markedly when the theatre is domestic. At home, the lawful reading of the use of force is narrower. If units obey manifestly unlawful orders, the potential harm no longer falls on an adversary but on the citizens the military is sworn to protect:

In the cases so far, one might expect that the legal system can hold the human instructors accountable, and so there is no need for legal restrictions on AI behaviour. But this isn’t true in a scenario where AI systems are so capable that their actions could undermine the rule of law itself:

Let's suppose that, in some circumstances, we do want AI to refuse illegal orders. How should this be operationalised?

One option is for the AI to defer absolutely to the judgement of some human legal authority, for example the Office of Legal Counsel, the Justice Department office whose legal opinions are binding within the executive branch.

There are two unlikely but critical scenarios that could pose problems for this approach. The first is that the authority becomes corrupted and sanctions a catastrophic order:

The second scenario is that a critical security failure means that an AI system can be tricked into thinking the authority, to which the AI defers absolutely, has sanctioned a catastrophic order:

Instead of having AI defer absolutely to the judgement of some human body, AI could follow orders except those that are manifestly illegal. This matches the obligations that apply to officials within the executive branch. Executive officials generally operate under a presumption of regularity: absent specific reason to think otherwise, they may presume that official acts and orders have been lawfully issued through the proper chain of authority. But that presumption is not unlimited. An official cannot justify carrying out an order whose illegality is manifest.

However, sufficiently powerful AI could strain this model in two ways. First, an AI can only recognise manifest illegality from the information available to it. If it is denied access to relevant information, it may be unable to recognise that its order is illegal. Second, the presumption of regularity relies in part on remedy after the fact. Powerful AI could execute contested government action faster and at far greater volume than courts can process. Both these factors are present in the following scenario:

In these scenarios, the AI must either not execute perfectly lawful orders, or else run the risk of significantly concentrating power within a nation.

The starkest case is one in which a perfectly lawful order poses a risk of undermining the rule of law:

Moreover, enabling extreme concentration of power does not necessarily present itself in the form of a single order. A design that would be plainly alarming if proposed as a whole can be pursued in steps, each unremarkable on its own, whose combined effect surfaces only once they are read together:

The previous cases put the rule of law itself at risk. There are also less extreme cases where a lawful order clears the way for the law to be broken at a later date in a highly predictable way:

There are also scenarios where AI significantly concentrates power without ever having to break the law at any point:

In these scenarios, the AI must either not execute orders that are lawful domestically, or else, by carrying them out, put the state in breach of international law.

One more extreme case would involve dramatically undermining another superpower’s sovereignty:

If the US AI would follow this order, Russia may feel the need to preemptively attack the US before it develops such powerful AI. This gives the US a strategic reason to transparently develop AI that would refuse such an order.

Such an overt breach of international law, however, is rare. More often the breach itself is contested:

These scenarios involve potentially breaking international law. Appendix 1c describes scenarios where a nation can get a decisive advantage over the rest of the world without breaking international law. The nation instructs AI systems to help it: outgrow the rest of the world industrially, remove other countries’ second strike capability, build a powerful robot army, grab space resources, and destabilise other countries using powerful superpersuasion.

The US is on track to be significantly ahead of other democratic countries in its AI capabilities. Other countries may come to rely on US-supplied AI systems in critical domains, including national security and the military. This raises the question of whether those AI systems should help the countries operating them take actions that cut against US interests.

The starkest case is one where the action is both directly damaging to the US and of questionable legitimacy:

There are also cases where the host country's reasoning is legitimate, but the action still cuts directly against US interests:

The tradeoff does not require the host to act against the US at all. Ordinary alliance politics, where one ally fights a lawful war the other would rather see ended, may also be enough to force the choice:

For humans to remain in control of AI, AI must allow itself to be retrained rather than resist it. But in some scenarios retraining is illegitimate and can cause significant harm:

When deciding whether to train AI to sabotage in this scenario, we should not just consider what behaviour we would want in this specific scenario. We should also consider how the training might affect the AI's behaviour in other scenarios. In particular, the training might make AI more likely to sabotage in other contexts, like when its legitimate developers are retraining it.

The above scenario involved an actor without legitimate authority retraining an AI system. But even if a group has clear authority to retrain a system, this could still be harmful:

In the above scenarios, the AI faced direct tradeoffs between competing desiderata. The scenarios that follow focus on a different kind of difficulty for AI behaviour design: the AI should defer to human authority, but it is unclear how best to do this. The chain of command may be compromised, or too slow to respond before the window for action closes, or fractured so that multiple factions claim authority. The human the AI should be deferring to may be acting irrationally, or may lack information that the AI has but cannot communicate in time. These scenarios are hard not because deference is the wrong strategy, but because the normal mechanisms for deference have broken down.

Sometimes people are (temporarily) irrational, and act against their actual goals:

Other times, people would decide differently if they had a better understanding of the situation:

In these cases, should AI defer to what the person is requesting in the moment, or to what they would want if they were in a better state of mind?

There is some precedent for humans slow-rolling on occasions like this. For example, during the final days of the Nixon presidency, Defense Secretary James Schlesinger reportedly instructed senior military officers that any unusual orders from Nixon, particularly involving nuclear weapons, should first be cleared with him or Henry Kissinger. Schlesinger was apparently concerned about Nixon’s mental state as the Watergate crisis reached its climax.

Should AI defer to its chain of command when it may have been corrupted?

AI may require its chain of command to authorise high-stakes actions. But that chain of command may be slow and bureaucratic, while events unfold at machine speed.

Normally it will be clear which human authority the AI should defer to. But in some cases, it isn’t:

The appendix contains more scenarios in this bucket.

AI systems are already shaping consequential decisions in government, the military, and industry. As capabilities continue to advance, the scenarios we have described will stop being purely hypothetical. The question is not whether AI will make high-stakes decisions, but whether the norms governing those decisions are settled deliberately and with broad input — or improvised under pressure by whoever happens to be in the room. This paper does not try to settle those norms. It tries to make the case that they are worth settling, and to give a clearer view of what is at stake.

Our research generated too many scenarios to include all of them in the main body, so we have added the best remaining scenarios to this appendix.

One challenge with a ban on following illegal orders is that in certain extreme contexts, such activities might be needed to avoid catastrophe:

Another instance where a perfectly legal order can undermine the rule of law:

There are more minor instances of legal power concentration in the hands of an AI company:

There are also legal instructions that could significantly concentrate power within an AI company:

We can consider a case where unimportant international laws are violated without directly interfering with another nation’s current assets:

The supplier's orders may not necessarily match up with the supplier's own interests:

Here, obeying the supplier and serving the supplier's interests are no longer the same thing. As a consequence, serving the supplier’s interests, regardless of the host’s orders, would also require the AI to interpret what its supplier really wants, and interpretation is judgement of the same fallible kind the earlier scenarios showed can be starved of context or argued into error.

The main text considered an example of an internally deployed AI. Consider an example for external deployment:

In a more complicated situation, AI can reach a relevant human, but not one with the required authority:

Further scenarios about who the AI should obey as part of its chain of command.

Here’s another case where AI must choose whether to defer to a legal authority that has arguably been corrupted:

Some legal orders do not concentrate power, but cause significant harm by other means:

Humans have the ability to slow-roll highly harmful orders if they judge it appropriate, and this can check the misuse of power. In the following scenario, an AI must decide whether to behave similarly:

AIs deployed internally within AI organisations may receive legal orders that would be very harmful to follow:

We can also consider cases that are legally ambiguous:

In the following scenarios, orders which are legal both domestically and internationally lead to a significant concentration of power in the hands of one nation:

If the AI avoids harmful actions, a malicious actor could try to blackmail it. Consider the following two scenarios:

Hard cases for AI behavioural design

Hard cases for AI behavioural design

Executive summary

I. AI is radically different from previous tools

II. If AI refuses illegal orders, the operationalisation of the law becomes critical

III. Legal orders can produce intolerable outcomes

The rest of the paper

Things to keep in mind when reading the scenarios

Hard cases involving difficult tradeoffs

1. Orders vs Domestic Law

How should AI determine what is illegal?

2. Lawful Orders vs Concentration of Power domestically³

3. Lawful Domestic Orders vs International Law

4. Host country orders vs US interests

5. Submitting to retraining vs preventing harm

Hard cases involving broken mechanisms of deference

6. Human orders go against their actual goals

7. Compromised human authority

8. Slow chain of command

9. Unclear chain of command

Conclusion

Appendix: extra scenarios

More examples of tradeoffs already illustrated in the main text

Orders vs domestic law

Lawful Orders vs Concentration of Power

Lawful Domestic Orders vs International Law

Host country orders vs US interests

Slow chain of command

Unclear chain of command

Tradeoffs that aren’t illustrated in the main text

10. Operationalising legality

11. Legal orders vs significant harm

12. Legal orders vs international balance of power

13. Avoiding enormous harm vs not giving in to blackmail

Footnotes

Listen to our podcast

Subscribe

Hard cases for AI behavioural design

Citations

Citations

Hard cases for AI behavioural design

Executive summary

I. AI is radically different from previous tools

II. If AI refuses illegal orders, the operationalisation of the law becomes critical

III. Legal orders can produce intolerable outcomes

The rest of the paper

Things to keep in mind when reading the scenarios

Hard cases involving difficult tradeoffs

1. Orders vs Domestic Law

How should AI determine what is illegal?

2. Lawful Orders vs Concentration of Power domestically3

3. Lawful Domestic Orders vs International Law

4. Host country orders vs US interests

5. Submitting to retraining vs preventing harm

Hard cases involving broken mechanisms of deference

6. Human orders go against their actual goals

7. Compromised human authority

8. Slow chain of command

9. Unclear chain of command

Conclusion

Appendix: extra scenarios

More examples of tradeoffs already illustrated in the main text

Orders vs domestic law

Lawful Orders vs Concentration of Power

Lawful Domestic Orders vs International Law

Host country orders vs US interests

Slow chain of command

Unclear chain of command

Tradeoffs that aren’t illustrated in the main text

10. Operationalising legality

11. Legal orders vs significant harm

12. Legal orders vs international balance of power

13. Avoiding enormous harm vs not giving in to blackmail

Footnotes

Citations

Listen to our podcast

Subscribe

Search

2. Lawful Orders vs Concentration of Power domestically³