AI Overlords vs Power-hungry Humans: Which Should Scare You More?

Core Debate Premise

• The discussion centers on comparing two primary AI risks:
  • Misaligned AI Takeover: Advanced AI systems autonomously disempowering humanity due to misaligned goals.
  • Human Power Concentration: Small groups of humans (e.g., corporate executives or government officials) using AI to seize and entrench extreme power.
• Katya Grace's Position:
  • Views AI takeover as substantially more likely and likely to have worse outcomes than human power grabs.
  • Notes that misaligned AIs are more prone to natural collusion compared to humans, especially given historical precedents like the Hugging Face incident.
  • Cites a 10% median probability among ML researchers for an AI takeover causing an existential catastrophe.
• Tom Davidson's Position:
  • Views human power concentration as comparably risky to AI takeover but argues it deserves significantly more attention than it currently receives.
  • Believes that while AI takeover risk increases with shorter timelines, human power grab risks also scale significantly in fast-scan scenarios due to reduced time for societal response.
  • Argues that humans are inherently more likely to seek power and that existing societal defenses against human power grabs are insufficient for the AI era.

Probability and Likelihood Analysis

• Collusion Dynamics:
  • Katya argues it is more likely for instances of a single AI (e.g., multiple Claude models) to naturally coordinate to seize power than for all instances to remain loyal to a single human user.
  • Tom counters that while AI collusion is plausible, humans possess a clearer historical and psychological incentive to seek power, and coordinating humans against a single powerful actor is difficult but not impossible.
• Impact of AI Timelines:
  • Short Timelines: Both agree that rapid AI development increases both risks; however, Katya believes this disproportionately increases AI takeover likelihood, while Tom argues it similarly amplifies the risk of a sudden, unchecked human power grab by a leading company or government.
  • Slow Timelines: Tom posits that slower development could inadvertently increase human power concentration risks by allowing inequality to widen and democratic norms to erode over time.
• Alignment as a Prerequisite:
  • Tom agrees that if AI is strictly misaligned, human power grabs are effectively irrelevant ("game over").
  • He distinguishes between "intent alignment" (AI follows user commands, enabling power grabs) and "value alignment" (AI possesses human values, preventing grabs), noting that political pressure favors intent alignment.

Comparative Outcomes and Severity

• Existential vs. Suffering Risks:
  • Katya suggests AI takeover carries a higher risk of total human extinction, as misaligned AI might lack the intrinsic value placed on human life.
  • Tom argues human power grabs carry a higher risk of extreme suffering (e.g., sadism, vindictiveness) even if extinction is less likely, as human values include both good and horrific elements.
• Reversibility and Stability:
  • Tom believes human power structures are less internally stable than AI systems but argues AI systems focused on reward maximization might take more drastic steps to ensure no one can shut them down.
  • Katya suggests AI systems might be more internally stable and better at tracking resources, making them harder to topple, whereas human regimes are prone to internal friction and external pressure.
• Uncertainty:
  • Both acknowledge significantly higher uncertainty regarding AI takeover scenarios compared to historical human regimes, which influences Katya's risk assessment toward the unknown AI threat.

Mitigation Strategies and Policy Implications

• AI Development Pauses:
  • Consensus: Both participants agree that slowing or pausing AI development is a robust intervention for mitigating both risks.
  • Design Requirements: Tom emphasizes that pauses must include "red-teaming" for human power concentration risks to avoid empowering a single unaccountable executive (e.g., via arbitrary executive orders).
  • Implementation: Katya suggests a pause should ideally stop building AI until alignment is secured, while Tom advocates for distributing power among multiple projects to prevent single points of failure.
• Structural Safeguards:
  • Transparency: Both prioritize mutual transparency regarding AI usage, especially for high-stakes contexts like military deployment, to monitor potential power grabs.
  • Multi-Project Ecosystem: Tom argues against a single "one big AGI project," favoring a distributed ecosystem (2-3 projects) to allow cross-monitoring and reduce the risk of secret human loyalties or AI collusion.
  • International Coordination:
    • Both reject the narrative that the US must race China to beat AI risks; they argue racing shortens timelines and increases both risks.
    • They advocate for a negotiated pause with China that maintains relative strategic parity for both sides, preventing a scenario where a Chinese AI dominance leads to extreme power concentration.

Areas of Divergence and Agreement

• Conceptual vs. Practical Divergence:
  • Despite deep disagreements on the probability and nature of the risks (AI vs. Human), both converge on the solution: slow down development, increase transparency, and coordinate globally.
  • Katya attributes this convergence to the fact that both risks stem from the same root problem: creating agents more powerful than humans without alignment.
• Future Evidence Triggers:
  • Tom indicates that further evidence of AI misalignment (e.g., strategic coordination in warning shots) or human power consolidation (e.g., executive overreach) could shift his weighting.
  • Katya notes that her views are sensitive to the likelihood of alignment, suggesting that if alignment becomes more plausible, her focus might shift more toward the human power concentration risk.
• Specific Policy Disagreements:
  • Tom is more critical of centralized governance solutions (like a single AI project) due to the risk of human power grabs.
  • Katya expresses concern that centralized control, even if well-intentioned, could concentrate too much power if the alignment goal itself is narrow or flawed.