#40 - Katja Grace on forecasting future technology & how much we should trust expert predictions

AI Impacts Overview

• AI Impacts is a research organization led by Katja Grace, focused on forecasting AI timelines and impacts.
• The organization operates with approximately two full-time equivalent staff and has been running for about three years.
• Its primary methodology involves answering high-level questions through lower-level empirical research, such as analyzing hardware trajectories and comparing brain computation to supercomputing power.

Key Findings from the "When Will AI Exceed Human Performance?" Survey (2017)

• The survey surveyed 1,600 machine learning researchers (published at NIPS or ICML in 2015), yielding 352 responses (21% rate).
• Timeline Estimates: Experts gave a median estimate of 45 years for "high-level machine intelligence" (machines doing every task better/cheaper than humans).
• Framing Bias: When asked about "fully automatable occupations," the median estimate shifted to 120 years, suggesting significant sensitivity to question framing.
• Probability vs. Year Direction: Asking experts to assign a probability to a fixed year yielded earlier timelines than asking them to assign a year to a fixed probability.
• Expert Consensus: There is a wide variance in expert opinions; while 20% of the median group believed a 50% chance of high-level AI existed within 15 years, others saw near-zero chance even in 100 years.
• Risk Perception: The median expert estimated a 5% chance that progressive machine learning would result in human extinction.
• Safety Priorities: Just under half of respondents favored increasing AI safety efforts beyond current levels, with 35% viewing AI alignment as at least as valuable as other AI research problems.
• Geographic Disparity: Researchers based in Asia predicted significantly faster AI progress timelines compared to those in North America and Europe.
• Intelligence Explosion: Only 12% of experts assigned greater than an 80% probability to the "intelligence explosion" hypothesis (rapid self-improvement after human-level AI).

Hardware and Compute Trajectories

• Compute Growth: The computational cost for training headline AI models has doubled every 3.5 months since 2012, representing a ~300,000x increase over six years.
• Brain Equivalency: Recent analysis estimates the human brain performs between $1.8 \times 10^{13}$ and $6.4 \times 10^{14}$ "traversed edges" per second, a workload comparable to current top-tier supercomputers.
• Cost Parity: Running human-brain-equivalent hardware is currently estimated to cost between $5,000 and $200,000, whereas running a supercomputer costs $2,000–$40,000 per hour; a cost parity with human labor is projected within a few decades.
• Uncertainty: Estimates for the hardware required to simulate a human brain vary by 10 orders of magnitude in older literature, driving a shift toward measuring communication bandwidth rather than raw calculations.

Discontinuity and Takeoff Speed

• Historical Precedent: AI Impacts identified only four major historical technological discontinuities, including nuclear weapons (6,000 years of prior progress compressed into years) and high-temperature superconductors.
• Expert Views on Takeoff: Most surveyed researchers do not anticipate a sudden, explosive takeoff, preferring models where progress remains gradual despite feedback loops.
• Hardware vs. Software: It remains an open question whether future AI progress will be driven primarily by hardware scaling (which is predictable) or software breakthroughs (which may cause discontinuities).
• Current Consensus: While many anticipate a "discontinuity," few have a concrete argument for why AI specifically would differ from other technologies that typically show incremental progress.

Career Advice and Research Gaps

• Neglect: AI forecasting is considered highly neglected compared to technical AI safety, with very few full-time researchers dedicated to the field.
• Entry Barriers: The organization recommends individuals start with modular, low-level projects (e.g., historical analysis of tech progress) rather than immediately pursuing a PhD.
• Skill Requirements: Success in this field is attributed more to general research skills, curiosity, and the ability to synthesize disparate data than to specific domain expertise.
• Hiring Priorities: AI Impacts seeks candidates comfortable jumping between diverse topics (e.g., hardware, primate brains, history) and resilient to open-ended, unstructured research questions.

Strategic Outlook on AI Safety

• Scenario Preferences: Katja Grace favors gradual "seeding of influence" scenarios over sudden "god-like" takeovers, citing the historical rarity of sudden technological discontinuities.
• Actionability: She argues that early intervention in AI strategy is valuable even without precise timelines, as current understanding is akin to "walking down a dark tunnel."
• Community Perception: A primary source of friction between AI researchers and safety advocates is the perception that current narrow AI is too limited to warrant long-term general AI concerns, a view Grace contests as underestimating future scaling potential.
• Funding Need: AI Impacts is actively seeking funding to expand its team and conduct more rigorous research into AI timelines and impacts.