#31 - Prof Dafoe on defusing the political & economic risks posed by existing AI capabilities

Context and Scope of AI Governance

• The Governance of AI program at Oxford's Future of Humanity Institute (FHI) comprises approximately a dozen researchers focused on navigating the transition to superhuman AI.
• Research agendas are categorized into three pillars:
  • Technical landscape: Trends in AI capabilities and the difficulty of ensuring safety.
  • AI politics: The dynamics of corporations, governments, and public actors pursuing their interests and managing risks.
  • AI governance: Designing global institutions, constitutional foundations, and value systems for cooperation.
• Interest in AI strategy is rapidly expanding, described by Alan Daffo as a field two years behind the growth of AI safety, evolving from a "taboo" niche to a flourishing community with top labs and defined research agendas.

Comparative Analysis: AI vs. Nuclear Weapons

• AI is distinguished from nuclear weapons by its dual-use nature as a general-purpose technology (comparable to electricity) with massive economic incentives for development.
• Unlike nuclear technology, where dangerous components can be physically isolated for control, AI capabilities are easily separable and portable, making traditional "hard control" of specific sites ineffective.
• Historical analogies like the Baruch Plan or Acheson-Lilienthal report are studied, but the failure of nuclear governance is largely attributed to Soviet actions under Stalin rather than a clean test of international control viability.

Dynamics of Competition and "Arms Races"

• Discussions of an AI "arms race" are often inaccurate or hyperbolic when applied to commercial competition (e.g., Google vs. Facebook) but possess real strategic validity when driven by state-level perceptions of threat.
• A quote by Vladimir Putin claiming "whoever leads in AI will rule the world" was an off-the-cuff remark at a school event, yet it galvanized global media and national security communities into treating AI as a high-stakes strategic race.
• Demis Hassabis emphasizes that the primary coordination challenge is avoiding a "race to the finish" where safety protocols are compromised for speed, particularly among nation-states.

Risks Associated with Current and Near-Term AI

• Labor and Inequality: Narrow AI today poses risks of mass unemployment and increased inequality due to potential labor displacement and the creation of natural global oligopolies (e.g., Facebook, Google, Amazon models).
• Economic Nationalism: AI's properties (low marginal costs, data feedback loops) may drive a shift from free trade to protectionism, with nations fostering domestic "champions" (e.g., China's market exclusion of US firms).
• Surveillance and Control: Existing technology supports the creation of systems for ubiquitous public monitoring, behavioral profiling, and tailored persuasion without needing future superintelligence.
• Robotic Repression: Autonomous weapons with independent chains of command could remove the human barrier (soldier reluctance) that historically prevented mass civilian oppression during coups.
• Strategic Stability: AI-enhanced intelligence (satellite/subsea sensors) combined with hypersonic missiles could render nuclear deterrents (like submarine ballistic missiles) transparent, creating a more unstable global security environment.

Forecasting and the Pace of Intelligence

• Expert surveys on AI timelines show a wide range, but the median suggests a 10% probability of human-level AI within 10 years and a 30% probability within 25 years.
• Rapid progress may occur due to "overhangs" in:
  • Hardware: Existing computing power waiting for efficient algorithms (e.g., a laptop-equivalent AGI).
  • Insights: Missed algorithmic breakthroughs (e.g., DeepMind's use of full value function distributions) that could be rediscovered.
  • Data: Massive unanalyzed knowledge bases (e.g., the internet, books) ready for high-level interpretation.
  • Compute Allocation: The "train-to-execute" ratio, where training requires 1–100 million times more compute than deployment, allowing a single trained AGI to be replicated millions of times instantly.

Career Advice and Research Priorities

• Researchers are advised to map their comparative advantages to specific modular projects rather than seeking a single "highest leverage" problem.
• Key research areas include historical case studies (e.g., nuclear control), economic modeling of race dynamics, forecasting, ethics, constitutional design, and public opinion research.
• The field currently lacks a canonical "safety critical" paper for policy, though a comprehensive research landscape document is in development.
• There is no single dedicated conference for AI strategy/policy; events are currently fragmented across organizations like the Partnership on AI, OECD, and the UN.
• Successful candidates for research roles should possess high intelligence, diverse forms of drive (passion/conviction), and the ability to retool skills within a year to fit specific niches.

Future Outlook

• Alan Daffo remains dispositionally optimistic, arguing that humanity's history of increasing peace and cooperation suggests the capacity to manage this "final test" of cooperation.
• The potential upside of advanced AI (wealth, scientific insight, human flourishing) is described as vast, necessitating a shift in focus from doomsday scenarios to the construction of functional global institutions.
• The field is in a "pre-paradigmatic" stage, requiring "disentanglers" to clarify the research agenda before "normal science" can fully take hold.