The most important century for humanity | Holden Karnofsky

Open Philanthropy Organizational Shifts

• Leadership Restructuring: Open Philanthropy has promoted Alexander Berger to Co-CEO, creating a two-headed organization to manage a growing distinction between two frameworks of doing good.
  • Global Health and Well-being: Led by Berger, this track focuses on traditional philanthropy metrics (e.g., disability-adjusted life years, poverty reduction) and requires scaling operations to manage a large volume of grantees and capital.
  • Long-termism: Led by Holden Karnofsky, this track focuses on maximizing the probability of a positive long-run future, measuring impact by the expected goodness of the whole future rather than immediate human welfare metrics.
• Strategic Rationale: The split allows Karnofsky to dedicate his full attention to the long-termist framework, which is currently viewed as more of a small, nimble, experimental operation rather than a capital-intensive grantmaking engine.
  • Karnofsky believes the primary bottleneck for long-termism is not capital, but a lack of shared worldview and "headspace" among potential grantees regarding the importance of the coming century.

The "Most Important Century" Thesis

• Core Hypothesis: Humans are likely living in the most important century in history because we are the first intelligent life to potentially transition from a biological, single-planet civilization to a stable, digital, galaxy-spanning civilization.
  • Evidence from Economic History: Current economic growth (approx. 2% annually) is unsustainable for more than a few thousand years at this rate; extrapolating current trends leads to physical impossibilities (e.g., an economy larger than the mass of the galaxy per atom).
  • The "PASTA" Concept: A "Process for Automating Scientific and Technological Advancement" could replace human-driven innovation, compressing 100,000 years of potential progress into 10 or 100 years.
• Timeline Estimates:
  • Compute Thresholds: Projections (e.g., Ajay Akottra's "Biological Anchors" report) suggest the computational power required to match a human brain will become affordable within this century.
  • Explosive Growth: Once AI can automate scientific discovery, economic growth is expected to shift from constant exponential growth to accelerating growth, potentially leading to a singularity event before 2100.
• Digital People: A pivotal outcome of this timeline is the ability to simulate human minds ("digital people").
  • Properties: Digital entities can be copied, run at different speeds, and placed in custom virtual environments, removing biological constraints like aging, disease, and physical resource limits.
  • Moral Status: Karnofsky argues digital people should be considered moral agents; consciousness is likely substrate-independent (algorithmic rather than biological), meaning a digital simulation of a person would share the same moral value as a biological person.
• Galactic Civilization: A digital civilization could expand across the galaxy by building computers on asteroids or other planets, using solar energy, and maintaining stability through error correction and code replication.

AI Forecasting and Expert Uncertainty

• Forecasting Challenges: Karnofsky notes that AI experts often disagree on timelines and capabilities because they focus on specific technical bottlenecks rather than macro-forecasting.
  • Lack of Consensus: Unlike climatology, there is no mature field of "AI-ology" with a strong consensus, making it difficult for outsiders to gauge risk.
  • Priors and Skepticism: Karnofsky admits he was initially skeptical of long-termism but shifted his view after realizing the "burden of proof" argument (that the century must be weird) applies to us now just as much as to historical periods of rapid change.
• Evaluation Intuition: There is a disconnect between the public perception of AI (incremental progress) and the potential for sudden capability jumps, similar to the transition from pre-boiling water to boiling water where surface changes are invisible until a phase change occurs.

Critiques, Weaknesses, and Actionable Steps

• Potential Failure Modes: The thesis could be wrong if:
  • Biological brains are far more complex than current compute estimates suggest.
  • Physical limits on chip manufacturing prevent the accumulation of necessary compute.
  • Societal regulation or fear halts AI development entirely.
  • The "soft" problems of AI alignment (getting AI to do what we want) prove intractable.
• Current Open Philanthropy Strategy:
  • Funding AI Alignment: While the timeline is uncertain, Open Philanthropy continues to fund technical AI alignment research because preventing misaligned outcomes is a high-impact, necessary step regardless of the exact timeline.
  • Avoiding Racing Narratives: Karnofsky warns against "racing" to build AI for geopolitical reasons, suggesting this approach increases the risk of losing control before values are aligned.
  • The "Long Reflection": Karnofsky advocates for a period of reflection and global negotiation before deploying powerful AI, allowing humanity to agree on the values and rules governing the future digital civilization.
• Philosophical Stance on "Weirdness": Karnofsky argues that rejecting ideas solely because they seem "weird" or counter-intuitive is a flawed heuristic for high-stakes decision-making; history shows that reality often defies common sense (e.g., quantum mechanics, historical economic acceleration).

Personal Context and Disclosure

• Open Philanthropy Funding: Open Philanthropy is a major funder of 80,000 Hours.
• Karnofsky's Background: Karnofsky co-founded GiveWell in 2007 and Open Philanthropy in 2014; he initially approached the meeting with founders Dustin Moskovitz and Cari Turner (the "cover story" being a romantic date) as a low-priority event, despite their intent to deploy billions.
• Career Pivot: Karnofsky moved to focus exclusively on long-termism after concluding that the global health side was sufficiently staffed to scale, while the long-term side required his specific attention to worldview and experimental grantmaking.