Design sketches: defense-favoured coordination tech

Intro

• Fast facilitation — Groups quickly surface key points of consensus views and disagreement, and make decisions everyone can live with.
• Automated negotiation — Complicated bargains are discovered quickly via automated negotiation on behalf of each party, mediated by trusted neutral systems which can find agreements.
• Arbitrarily easy arbitration — Disputes are resolved cheaply and quickly by verifiably neutral AI adjudicators.
• Background networking — People who should know each other get connected (perhaps even before they know to go looking), enabling mutually beneficial trade, coalition building, and more.
• Structured transparency for democratic oversight — Citizens hold their institutions to account in a fine-grained way, without compromising sensitive information.
• Confidential monitoring and verification — Deals can be monitored and verified, even when this requires sharing highly sensitive information, by using trusted AI intermediaries which can’t disclose the information to counterparties. 

• AI delegates and preference elicitation —  AI delegates can faithfully represent and act for a human principal, perhaps supported by customisable off-the-shelf agentic platforms that integrate across many kinds of tech.
• Charter tech — The technologies above, or other coordination technologies, are applied to making governance dynamics more transparent, making it easier to anticipate how governance decisions will influence future coordination, and design institutions with this in mind.

Why coordination tech matters

• Improving economic productivity across the board
• Helping nations avoid wars and other destructive conflicts
• Enabling larger groups to coordinate to avoid exploitation by a small few
• Making democratic governance much more transparent, while protecting sensitive information

• AI companies to collude with each other against the interests of the rest of society2

• A small group of actors to plot a coup
• More selfishness and criminality, as social mechanisms of coordination are replaced by automated ones which don’t incentivise prosociality to the same extent

• The shape and impact of coordination tech is an important part of how things will unfold in the near term, and it’s good for people to be paying more attention to this.
• We’re going to need some kinds of coordination tech to safely navigate the AI transition.
• The devil is in the details. There are ways of advancing coordination tech which are positive in expectation, and ways of doing so which are harmful.

Why ‘defense-favoured’ coordination tech

Fast facilitation

Design sketch

Image

• Gathers written context on the setting and decision
• Holds brief, private conversations with each participant to understand their perspective
• Builds a map of the issue at hand, involving key considerations and points of (dis)agreement

  • Performs and integrates background research where relevant
• Identifies which people are most likely to have input that changes the picture
• Distils down a shareable summary of the map, and seeks feedback from key parties
• Proposes consensus statements or next steps for approval, iterating quickly to find versions that have as broad a backing as possible

Feasibility 

• Ensuring that it’s more efficient and a better user experience for moving towards consensus than other, less AI-based approaches.
• Remaining robust against abusive user behaviour (e.g. you don’t want individuals to get their way via prompt injection or blatantly lying).

Possible starting points // concrete projects

• Build a baby version. This could help us notice obstacles or opportunities that would have been hard to predict in advance. You could focus on the UI or the tech side here, or try to help run pilots at specific organisations or in specific settings.
• Design ways to evaluate fast facilitation tools. This makes it easier to assess and improve on performance. For example, you could create games/test environments with clear “win” and “failure” modes.
• Build subcomponents. For example: 

  • Bots that surface anonymous info.
  • Tools that try to surface areas of consensus or common knowledge as efficiently as possible, while remaining hard to game.
• Make a meeting prep system. Focus first on getting good at meeting prep — creating an agenda and considerations that need live discussion — to reduce possible unease about outsourcing decision-making to AI systems.
• Make a bot to facilitate discussions. This could be used in online community fora, or to survey experts.
• Design ways to create live “maps” of discussions. Fast facilitation is fast because it parallelises communication. This makes it more important to have good tools for maintaining shared context.

Automated negotiation

• You communicate your desires openly with a negotiation delegate who is on your side, asking questions only when needed to build a deeper model about your preferences.
• The delegate goes away, and comes back with a proposal that looks pretty good, along with a strategic analysis explaining the tradeoffs / difficulties in getting more.

Design sketch

Image

• Delegate systems:

  • Read over context documents and query principals about key points of uncertainty to build initial models of preferences.
  • Model the negotiation dynamics and choose strategic approaches to maximise value for their principal.
  • Go back to the principal with further detailed queries when something comes up that crosses an importance threshold and where they are insufficiently confident about being able to model the principal’s views faithfully.
  • Are ultimately trained to get good results by the principal’s lights.
• Neutral mediator system:

  • Is run by a trusted third-party (or in higher stakes situations, perhaps is cryptographically secure with transparent code).
  • Discusses with all parties (either AI delegates, or their principals)

    • Can hear private information without leaking that information to the other party

      • Impossibility theorems mean that it will sometimes be strategically optimal for parties to misrepresent their position to the mediator (unless we give up on the ability to make many actually-good deals); however, we can seek a setup such that it is rarely a good idea to strategically misrepresent information, or that it doesn’t help very much, or that it is hard to identify the circumstances in which it’s better to misrepresent
  • Searches for deals that will be thought well of by all parties, and proposes those to the delegates.
  • Is ultimately trained to help all parties reach fair and desired outcomes, while minimising incentives-to-misrepresent for the parties.

Feasibility

• The kind of security needed for high-stakes applications isn’t possible today.
• Getting systems to be deeply aligned with a principal’s best interests, rather than e.g. pursuing the principal’s short-term gratification via sycophancy, is an unsolved problem.

Possible starting points // concrete projects

• Build an AI delegate for yourself or your friends. See if you can get it to usefully negotiate on your behalf with your friends or colleagues. Or failing that, if it can support you to think through your own negotiation position before you need to communicate with others about it.
• Build a negotiation app with good UI. Building on existing LLMs, build an app which helps people think through their negotiation position in a structured way. Focus on great UI.

  • This could be non-interactive at first, and just involve communication between a human and the app, rather than between any AI systems.
  • But it builds the muscles of a) designing good UI for AI negotiation, and b) people actually using AI to help them with negotiation.
• Run a pilot in an org or community you’re part of. 

  • You could start with fairly low-stakes negotiations, like what temperature to set the office thermostat to or which discussion topics to discuss in a given meeting slot.
  • Experimenting with different styles of negotiation (in terms of how high the stakes are, how complex the structure is, and what the domain is) could be very valuable.

Arbitrarily easy arbitration

Design sketch

Image

1. Agreement ingestion

   • Formal or natural language contracts are parsed and key terms extracted, with parties confirming the system's interpretation before proceeding.
   • The system could also suggest pre-dispute modifications to make agreements clearer, flag potentially unenforceable terms, and maintain public precedent databases that help parties understand likely outcomes before committing.
2. Automated discovery

   • When disputes arise, an automated discovery process gathers relevant documentation, transaction logs, and communications from integrated platforms.
   • The system offers interviews and the chance to submit further evidence to each party.
3. Deep consideration

   • The system builds models of what different viewpoints (e.g. standard legal precedent; commonsense morality; each of the relevant parties) have to say on the situation and possible resolutions, to ensure that it is in touch with all major perspectives.
   • Where there are disagreements, the system simulates debate between reasonable perspectives.
   • It makes an overall judgement as to what is fairest.
4. Transparent reasoning

   • The system produces detailed explanations of its conclusions, with precedent citations and counterfactual analysis where appropriate.
5. (Optional) Smart escrow integration

   • Judgements automatically execute through cryptocurrency escrows or traditional payment rails, with graduated penalties for non-compliance.
   • In cases where the system detects evidence that is highly likely to be fraudulent, or other attempts to manipulate the system, it automatically adds a small sanction to the judgement, in order to disincentivise this behaviour.
6. Opportunities for appeal

   • Either party can pay a small fee to submit further evidence and have the situation re-considered in more depth by an automated system.
   • For larger fees they can have human auditors involved; in the limit they can bring things to the courts.

Feasibility

• Judgement: Systems may not currently have good enough judgement to do 1, 3, 4 in high-stakes contexts (and until recently, they clearly didn’t).
• Real-world evidence assessment: Systems don’t currently know how to handle conflicting evidence provided digitally about what happened in the real world.
• Verifiable fairness/neutrality: The full version of this technology would require a level of fairness and neutrality which isn’t attainable today.

• Starting in lower-stakes or easier contexts (for example, digital-only spaces avoid the challenge of establishing provenance for real-world evidence).
• Creating evals, test environments and other infrastructure that helps us improve performance.

• Trust: As above, some amount of technical work is needed to make systems verifiably fair/neutral. But even if it becomes true that the systems are neutral, people need to build quite a high level of confidence that the system is genuinely impartial before they'll bind themselves to its decisions for meaningful stakes.
• Legal integration: This tech is only useful to the extent that its arbitration decisions are recognised and enforced as legitimate by the traditional legal system, or are enshrined directly via contract in a self-enforcing way.

  • (We are unsure how large a challenge this will be; perhaps you can write contracts today that are taken by the courts as robust. But it may be hard for parties to have large trust in them before they have been tested.)

Possible starting points // concrete projects

• Work with an arbitration firm. Work with (or buy) a firm already offering arbitration services to start automating parts of their central work, and scale up from there.
• Work with an online platform that handles arbitration. Use AI to improve their processes, and scale from there.
• Create a bot to settle informal disputes. Build an arbitration-as-a-service bot that people can use to settle informal disputes.
• Trial a system on internal disputes. This could be at your own organisation, another organisation, or a coalition of early adopter organisations.
• Run a pilot in parallel to regular arbitration. Run a pilot where an automated arbitration system is given access to all the relevant information to resolve disputes, and reaches its own conclusions — in parallel to the regular arbitration process, which forms the basis of the actual decision. You could partner with an arbitration firm, or potentially do this through a coalition of early adopter organisations, perhaps in combination with philanthropic funding.

Background networking

Design sketch

Image

• Interoperable, secure ‘wish profiling’ systems which identify what different participants want.

  • People connect their profiles on existing services (social media, chatbot logs, email, etc).
  • LLM-driven synthesis (perhaps combined with other forms of machine learning) curates a private profile of user desires.
  • Optionally, chatbot-style assistance can interview users on the points of biggest uncertainty, to build a more accurate profile.
• A searchable ‘wish registry’ which organises large collections of wants and offers, while maintaining semi-privacy.

  • Each user’s interests can run searches, finding potential matches and surfacing only enough information about them to know whether they are worth exploring further.

Feasibility

• If sensitive data is too broadly available, it can be used for surveillance, harassment, or exploitation; including by big corporations or states.
• If sensitive data is completely private, it opens up the possibility of collusion, for example among criminals.

• Technically, background networking tools already seem within reach using current systems. Large-scale deployments would require indexing and registry, but it seems possible to get started on these using current systems.

  • One note is that it seems possible to implement background networking in either a centralised or a decentralised way. It’s not clear which is best, though decentralised implementations will be more portable.
• Adoption also seems likely to work, because there are incentives for people to pay to discover trade and cooperation opportunities they would otherwise have missed, analogous to exchange or brokerage fees. Though there are some trickier parts, we expect them to ultimately be surmountable (though timing may be more up for grabs than absolute questions of adoption):

  • In the early stages when not many people are using it, the value of background networking will be more limited. Possible responses include targeting smaller niches initially, and proactively seeking out additional network beneficiaries.
  • It’s harder to incentivise people to pay for speculative things like uncovering groups they’d love that don’t yet exist. You could get around this using entrepreneurial or philanthropic speculation (compare the dominant assurance contract model and related payment incentivisation schemes).

Possible starting points // concrete projects

• Work with existing matchmakers to improve their offering. Find groups that are already doing matchmaking and are eager for better systems — perhaps among community organisers, businesses, recruiters or investors. Work with them to understand the pain points in their current networking, and what automated offerings would be most appealing. Then build those tools and systems. 
• Build a networking tool for a specific community. Build a custom networking system for a particular group or subculture. For example, this could look like a networking app or a plug-in to an existing online forum. This could start delivering value fairly quickly, and provide a good opportunity for iteration. 

Structured transparency for democratic oversight

• The baseline is highly opaque.
• Freedom of information systems help, but can be evaded by non-cooperating institutions.
• Public inquiries can be reasonably thorough, but are expensive and slow.
• Full transparency has many costs and is typically highly resisted.

Design sketch

Image

• Secure data repositories, such that institutions routinely share operational data with a sandboxed environment operated by or on behalf of the oversight body, without any regular human access to the data.
• Continuous ingestion and indexing of institutional public outputs (press releases, regulatory filings, budget documents, etc.) into a searchable database.
• Automated cross-referencing between public claims and internal records.
• Highlighting of potential issues (mismatches between public statements and private information, as well as decisions made in violation of normal procedures).
• Further automated investigation of potential issues, leading to flags to humans in cases with sufficiently large issues flagged with sufficient confidence.
• Importantly, the sandbox outputs its findings but not the underlying data; if there is need for transparency on that, this is a separate oversight question.

Feasibility

• Adversarial dynamics: the technical bar to verify claims against actively adversarial institutions (who are manipulating deposited data, potentially via AI) is substantially higher.

  • This is the bar that we’d need to reach for confidential monitoring below.
• Defamation risk: the downsides of false positives, where your system reports someone misrepresenting things when they were not, could be significant (although can perhaps be mediated by giving people a right-of-rebuttal where they give further data to the AI systems which monitor the confidential data streams).
• Avoiding abuse: designing the systems so that they do not expose the confidential data, and cannot be weaponised to ruin the reputation of a department with very normal levels of error.

Possible starting points // concrete projects

• Retrospective validation on historical cases. Apply consistency-checking tools to document sets from well-understood historical cases where the relevant internal documents have subsequently been released (e.g. Enron emails). This builds the technical foundation, and demonstrates the concept without requiring any current institutional access.
• Institutional public statement reliability tracker. Build a tool tracking whether agencies' public claims about performance, spending, or policy outcomes are consistent with publicly available data — statistical releases, budget documents, prior statements. Start with a single policy domain. This requires no institutional partnerships and builds a public constituency for structured transparency. This is a version of reliability tracking, applied specifically to institutional accountability.
• Pilot a FOIA exemption assessment tool. Partner with an Inspector General office to build a tool that reviews withheld documents and assesses whether claimed exemptions (national security, personal privacy, deliberative process) are applied appropriately. The IG already has legal access under the Inspector General Act; the tool helps them do their existing job faster and builds the working relationship needed for more ambitious deployments. This is also a natural testbed for the sandboxed architecture in miniature — the tool operates within the IG's secure environment, producing exemption-appropriateness findings without the documents themselves leaving the system.

Confidential monitoring and verification

Design sketch

• Its conclusions about the specified questions.
• The degree to which it is satisfied that it had good data access, that it didn’t run into attempts to distort its conclusions, etc.

Image

• Building a Bayesian knowledge graph, establishing hypotheses, and understanding what evidence suggests about those hypotheses.
• Agentic investigatory probes into the confidential data, in order to form grounded assessments on the specified questions.

Feasibility

• Legal audits — for example, claims that the documents not disclosed during a discovery process are only those which are protected by privilege.
• Financial audits — e.g. for the purpose of proving viability to investors without disclosing detailed accounts.
• Supply chain verification — e.g. demonstrating that products were ethically sourced without exposing the suppliers.

Possible starting points // concrete projects

• Start building prototypes. Build a system which can try to detect whether it’s a real or counterfeited environment, and measure its success.
• Work with a law or financial auditing firm. Work with (or buy) a firm that does this kind of work, and experiment with how to robustly automate while retaining very high levels of trustworthiness.
• Explore the viability of complementary technology. For example, you could investigate the feasibility of demonstrating exactly what code is running on a particular physical computer that is in the room with both parties.

Cross-cutting thoughts

Some cross-cutting technologies

AI delegates and preference elicitation

• Direct preference elicitation: develop efficient and appealing interview-style elicitation of values, wishes, preferences and asks.
• Passive data ingestion: build a tool that (consensually) ingests and distils all the available online content about a person — social media, browsing history, email, etc — and extracts principles from it (cf inverse constitutional AI).

• You could have different delegates for different applications.
• Some delegates might represent groups or coalitions.
• Some delegates could be short-lived, and spun up for some particular time-bounded purpose.

Charter tech

1. Through directly improving the governance dynamics in question, helping to avoid capture, conflict, and lock-in.
2. Through compounding effects on future coordination, which will unfold in the context of whatever governance structures are in place.

• A “governance dynamics analyser” that ingests descriptions of constitutions, charters, policies or community norms, builds models of power, incentives, and information flow, and then (a) forecasts likely equilibria and failure modes, (b) red-teams for strategic abuse,5

  and (c) proposes safer rule variants that preserve the framers’ intent.6

• While this tool can be called actively if needed, there is also a classifier running quietly in the background of organisational docs/emails, and when it detects a situation where power dynamics and governance rules are relevant, it runs an assessment — promoting this to user attention just in cases where the proposed rules are likely to be problematic.

• Work with communities that iterate frequently on governance (DAOs, open-source projects) to test analyses against what actually happens when rules change.
• Compile a pattern library of governance failures and successes, documented in enough detail to inform automated analysis.
• Build simulation environments where proposed rules can be stress-tested against populations of agents with varying goals, including adversarial ones.
• Partner with mechanism design researchers to identify which aspects of their formal analysis can be automated and applied to less formal real-world documents.

Adoption pathways

• Adoption by governments and broader society. Many of the most important benefits of coordination tech for society will come from government and broad social adoption, but these groups will be less impacted by commercial incentives. This bites particularly hard for technologies that could be quite expensive in terms of inference compute, like fast facilitation, arbitration and negotiation. By default, these technologies might differentially help wealthy actors, leaving complex societal-level coordination behind. We think that the big levers on this set of challenges are:

  • Building trust and legitimacy earlier, by getting started sooner, building transparently, and investing in evals and other infrastructure to demonstrate performance.
  • Targeting important niches that might be slower to adopt by default. More research would be good here, but two niches that seem potentially important are:

    • Coordination among and between very large groups, like whole societies. This might be both strategically important and lag behind by default.
    • International diplomacy. Probably coordination tech will get adopted more slowly in diplomacy than in business, but there might be very high stakes applications there. 
• Adoption of confidential monitoring and structured transparency. These technologies are less accessible with current models and may require large upfront investments, while many of the benefits are broadly distributed. 

  • This makes it less likely that commercial incentives alone will be enough, and makes philanthropic and government funding more desirable.

Other challenges

• Some actors could get earlier and/or better access to coordination tech than others.7

• Actors that face particular barriers to coordination today could be asymmetrically unblocked by coordination tech.
• Individuals and small groups could become more powerful relative to the coordination mechanisms we already have, like organisations, ideologies, and nation states.

• Value judgements about which actors should hold power.
• How contingent power dynamics play out.
• Big questions like whether ideologies or states are better or worse than the alternatives.
• Predictions about how social dynamics will equilibrate in an AI era that looks very different to our world.

• Large corporations to collude with each other against the interests of the rest of society.8

• A small group of actors to plot a coup.
• More selfishness and criminality, as social mechanisms of coordination are replaced by automated ones which don’t incentivise prosociality to the same extent.

• The upsides are very large. Coordination tech might be close to necessary for safely navigating challenges like the development of AGI, and could empower actors to coordinate against the kinds of misuse listed above.
• The counterfactual is that coordination tech is developed anyway, but with less consideration of the risks and less broad deployment. We think that this set of technologies is going to be sufficiently useful that it’s close to inevitable that they get developed at some point. By engaging early with this space, we can have a bigger impact on a) which versions of the technology are developed, b) how seriously the downsides are taken by default, c) how soon these systems are deployed broadly. 
• Some applications seem robustly good. For example, the potential for misuse is low for technologies like transparent facilitation or widely deployed charter tech. More generally, we expect that projects that are thoughtfully and sensitively run will be able to choose directions which are robustly beneficial.