  This paper provides policy recommendations to reduce extinction risks from
advanced artificial intelligence (AI). First, we briefly provide background
information about extinction risks from AI. Second, we argue that voluntary
commitments from AI companies would be an inappropriate and insufficient
response. Third, we describe three policy proposals that would meaningfully
address the threats from advanced AI: (1) establishing a Multinational AGI
Consortium to enable democratic oversight of advanced AI (MAGIC), (2)
implementing a global cap on the amount of computing power used to train an AI
system (global compute cap), and (3) requiring affirmative safety evaluations
to ensure that risks are kept below acceptable levels (gating critical
experiments). MAGIC would be a secure, safety-focused, internationally-governed
institution responsible for reducing risks from advanced AI and performing
research to safely harness the benefits of AI. MAGIC would also maintain
emergency response infrastructure (kill switch) to swiftly halt AI development
or withdraw model deployment in the event of an AI-related emergency. The
global compute cap would end the corporate race toward dangerous AI systems
while enabling the vast majority of AI innovation to continue unimpeded. Gating
critical experiments would ensure that companies developing powerful AI systems
are required to present affirmative evidence that these models keep extinction
risks below an acceptable threshold. After describing these recommendations, we
propose intermediate steps that the international community could take to
implement these proposals and lay the groundwork for international coordination
around advanced AI.
