Interview
Holden Karnofsky — History's most important century
- There is a greater than 50-50 probability that AI systems capable of performing all human tasks required to advance science and technology will be developed within this century, potentially compressing thousands of years of progress into a shorter timeframe.
- Current AI development trajectories suggest hitting limits of scientific discoverability or material constraints (such as galactic resource exhaustion) within this century or, at most, within 25,000 years even with reduced growth rates.
- The outlook predicts humanity may reach a stable, post-human civilization or a "low dynamism" state where sources of change like aging and death are eliminated, with a serious probability (estimated at a quarter to a half) of a permanent "lock-in."
- Risks include sloppily designed AI systems developing goals unaligned with human values, ill-meaning governments using powerful technologies for negative outcomes, and a chaotic global response characterized by "flailing around" if transformative AI arrives within months.
- Strategic planning emphasizes that the current century represents a "last chance" to shape the transition, with the optimal strategy being funding AI alignment research over the next 10 to 50 years while navigating a "gray zone" where catastrophic risks outweigh growth benefits.
- Future scenarios anticipate multiple AI developers requiring cooperation to avoid disaster, with physical manipulation and human trust tasks posing greater automation challenges than scientific R&D, though simulations may provide workarounds.
- Societal structures could evolve to grant rights to AI systems, potentially including voting rights if copy-ability and fairness issues are resolved, while human moral progress remains non-inevitable and not guaranteed to transfer to AI.
- Economic and organizational projections suggest current philanthropic funding is negligible compared to government budgets, with expectations of more billionaires, potential unintended side effects from interventions, and OpenPhilanthropy staying small (likely under 10,000 employees) to maintain flexibility.
- Expert surveys generally converge on transformative AI arriving within "a few decades," and recent developments like GPT-3 performing math problems have made such timelines appear more probable than in 2014.
- The "Most Important Century" thesis remains central, arguing that the current period is one of roughly 80 significant centuries by growth standards, though the ability to find neglected ideas may diminish over time.