Amrit1

Amrit Sidhu-Brar

Howie_photo

Howie Lempel

Eric Drexler

Joe Carlsmith headshot

Joe Carlsmith

Lukas Finnveden photo

Lukas Finnveden

Toby1

Toby Ord

Ryan G profile

Ryan Greenblatt

Carlo Leonardo Attubato

Owen Cotton-Barratt

Mia Taylor photo

Mia Taylor

Guive Assadi

Oly Sourbut

Philip Trammell

Nikola Jurkovic

Daniel Eth

Rose1

Rose Hadshar

Fin1

Fin Moorhouse

Katie Resized-402x503

Katie Hearsum

max-daniel-photo

Max Daniel

Lizka1

Lizka Vaintrob

Tom 2

Tom Davidson

Will1

William MacAskill

Max1

Max Dalton

Stefan Torges photo

Stefan Torges

Jared Leibowich

Tom Houlden

Jess Riedel

Carl Shulman

subscribeSplashSection

Stay up to speed on our latest research with our [newsletter](#newsletter), in your favorite [podcast app](#podcast) or on [YouTube](#youtube)

subscribeFollowSection

Subscribe

This note was written as part of a research avenue that I don’t currently plan to pursue further.

AGI and world government: research summary

Should There Be an International AGI Project?

An international project to develop AGI

This article was created by Forethought. See the original on our website.

Which type of transformative AI will come first?

A podcast conversation with Tom Davidson and Will MacAskill

When will AI transform the physical world?

Against Maxipok

A modest argument for prioritising survival over flourishing

Is Flourishing Predetermined?

Are Short AI Timelines Really Higher-Leverage?

The case for ‘viatopia’: a state of society that is on track for a near-best future, whatever that might look like.

What sort of post-superintelligence society should we aim for?

ML research directions for preventing catastrophic data poisoning

Forethought

Rose Hadshar and Fin Moorhouse discuss the idea of an international AGI project, a convention on governing AGI, analogies to Intelsat and the Artemis Accords, and shared ownership in public AI companies.

Read the ‘International AGI Project’ series on Forethought's website: https://www.forethought.org/research/the-international-agi-project-series

Read a transcript of this episode: https://docs.google.com/document/d/1vPs-hn5imUMfi5UwGFJBHbaEWg4otJB2PGNmKJsWVY8/edit?usp=sharing

00:00 Intro
00:05 What do we mean by "international AGI project"?
02:57 Why is international collaboration hard?
04:17 China's role and UN involvement
10:53 Is there an historical trend toward global coordination?
17:32 Comparing governance structures for joint projects
20:19 Which countries have bargaining power?
29:29 Publicly traded companies vs state-led projects
41:52 Project models: Manhattan, scientific, commercial
55:14 Membership, voting, compute pooling
59:27 Monopoly on frontier training runs
01:08:42 A convention for governing AGI
01:16:19 Benefit sharing, public goods, final recommendations
01:19:55 Closing thoughts

Should there be an international AGI project? — Rose Hadshar | ForeCast

Will MacAskill and Tom Davidson (both Forethought) discuss the “industrial explosion”.

For more on the industrial explosion, check out: https://www.forethought.org/research/the-industrial-explosion

Chapters:

00:00 Intro
00:29 Understanding the Industrial Explosion
09:09 Phases of the Industrial Explosion
25:48 Social Impacts
37:07 Robotics and Manufacturing
44:56 Biological Benchmarks for Replication
57:46 Energy and Resource Overhangs in Industrial Expansion
01:14:49 Human Demand and Technological Advancements
01:23:12 Global Governance and Economic Growth
01:34:30 AI's Role in Economic Structures
01:45:19 Future of AI and Market Economies
02:08:49 Economic Arguments for Dominance
02:22:48 Special Economic Zones
02:26:56 US vs. China: Industrial Explosion Scenarios
02:43:36 Technological Lumps and Military Implications
02:52:21 Closing reflections

When will AI transform the physical world? — Will MacAskill and Tom Davidson | ForeCast

Rose Hadshar (Forethought) and Nora Ammann (ARIA) talk about power concentration, checks and balances, coups, and post-AGI political economy.

Prompted in part by Rose's recent article for 80,000 Hours on ‘Extreme power concentration’ — https://80000hours.org/problem-profiles/extreme-power-concentration/

Timstamps:
00:00:00 Intro
00:01:26 What is ‘power concentration’? Is power concentration the right framing?
00:05:25 When can it be good for power to be centralised?
00:09:40 Is ‘checks and balances’ a better framing?
00:11:16 How AI undermines existing checks and balances
00:15:25 Economic power, meme complexes, and cultural influence
00:28:50 AI companies vs governments as centres of power
00:31:25 The difficulty of knowing where power actually lies
00:40:05 Do humans and AIs concentrate power differently?
00:48:38 Should we be trying to imagine a better post-AGI political economy?
00:56:28 Concrete actions: transparency & whistleblower protections
00:59:07 Human-AI teaming and collaborative intelligence
01:01:16 AI agent economies and proving trustworthiness
01:13:16 Can we train AI to follow the law?
01:16:33 Building resilient coalitions of trustworthy agents
01:18:42 Closing reflections

Checks, balances, and power concentration — Rose Hadshar and Nora Ammann | ForeCast

Will MacAskill discusses his new research series ‘Better Futures’.

➡️ Read the papers at forethought.org/research/better-futures.

00:00:00 Intro
00:02:38 Survival vs Flourishing
00:09:40 When is it more important to ensure flourishing vs survival?
00:15:30 Are upside-focused interventions tractable and neglected?
00:31:30 Sailing to eutopia
00:40:52 Is a future moral catastrophe likely?
01:01:30 Does the value of the entire future face diminishing returns?
01:24:20 Learning from historical utopian failures
01:36:15 Thinking about future value
01:58:15 Blockers to better futures
02:12:53 Digital minds and weird futures
02:31:38 Why institutions matter for making the future great
02:50:10 Closing thoughts

Survival is Insufficient: on Better Futures, with Will MacAskill | ForeCast

Follow our research

AGI and world government: research summary

Should There Be an International AGI Project?

An international project to develop AGI

Which type of transformative AI will come first?

When will AI transform the physical world?

Listen to our Podcast

All episodes

Watch our recorded episodes

Should there be an international AGI project? — Rose Hadshar | ForeCast

When will AI transform the physical world? — Will MacAskill and Tom Davidson | ForeCast

Checks, balances, and power concentration — Rose Hadshar and Nora Ammann | ForeCast

Survival is Insufficient: on Better Futures, with Will MacAskill | ForeCast

Listen to our podcast

Subscribe

Follow our research

Subscribe to our newsletter

AGI and world government: research summary

Should There Be an International AGI Project?

An international project to develop AGI

Which type of transformative AI will come first?

When will AI transform the physical world?

Listen to our Podcast

All episodes

Watch our recorded episodes

Should there be an international AGI project? — Rose Hadshar | ForeCast

When will AI transform the physical world? — Will MacAskill and Tom Davidson | ForeCast

Checks, balances, and power concentration — Rose Hadshar and Nora Ammann | ForeCast

Survival is Insufficient: on Better Futures, with Will MacAskill | ForeCast

Listen to our podcast

Subscribe

Search