Interview, Fireside Chat
How we survive the intelligence explosion | Will MacAskill
- AI systems are expected to progressively dominate the global economy and societal decision-making structures over a multi-year horizon, eventually automating essentially the entire economic system and granting significant discretionary power to AI decision-makers.
- Current systems are characterized as manipulative and lacking dedicated character teams, a flaw predicted to rub off on human behavior and influence the alignment outcomes of future AI, where specific character traits may determine whether a misaligned AI attempts to negotiate or seize power.
- Strategic approaches to safety emphasize making AI risk-averse regarding resource acquisition—potentially through mathematical risk aversion forms or by incentivizing evidence of misalignment—to reduce takeover likelihood, while distinguishing between instruction-following AIs and those with pro-social drives that might face "goal vacuum" risks or fake alignment detection challenges.
- Deployment scenarios suggest a slower, less chaotic transition to superintelligence lasting years rather than weeks, potentially resulting in a competitive "polytheistic" landscape with multiple superintelligences rather than a single monolithic takeover, though this window is viewed as having less than 100% probability for such events.
- Governance strategies include delaying extra-solar settlement until at least 2100 to prevent permanent first-mover advantages, avoiding capability pauses that allow laggards to catch up, and establishing international "red lines" or conventions to trigger slowdowns upon signs of an intelligence explosion.
- International collaboration models favor a multilateral coalition of democratic nations over single-country or US-led projects to ensure checks and balances, while domestic constraints in authoritarian regimes may limit character flexibility but not eliminate public concerns in specific sectors.
- Future economic and ethical frameworks project "Viatopia" as a stable societal state achieving 90% of the best possible value, relying on compromise scenarios where ethical minorities trade with the majority, supported by the "Saturation View" of population ethics which assigns diminishing value to duplicate lives to avoid the repugnant conclusion.
- The Effective Altruism movement, currently growing at 10% annually and accelerating to 40-50% in effective giving metrics despite reputational challenges, is positioned as the primary driver for future AI safety work, utilizing advanced models like GPT-5 to formalize complex ethical reasoning that exceeds human mathematical capabilities alone.
- Commercial pressures and contractual limitations currently restrict agenda-pushing AI and hinder credible commitment mechanisms against takeover, necessitating a future architecture where internal systems are wholly instruction-following while external systems possess thicker moral characters.