Interview
#40 - Katja Grace on forecasting future technology & how much we should trust expert predictions
- 80,000 Hours plans to release the first of an "advanced career guide" series within the next year, with the initial article already available, targeting regular podcast listeners and aiming to reach new audiences capable of action through sharing.
- Katja Grace anticipates that AI will significantly alter the world likely sooner than the next century or two, potentially making current human-centric improvements in gifted education obsolete and rendering humans unemployed in long-term thinking work.
- Extinction risks from AI are expected to be the most plausible and imminent compared to other risks, with scenarios involving powerful AI lacking human values leading to a permanent loss of human control over resources and decision-making regardless of whether progress is slow or rapid.
- While discontinuities in technology are considered rare, AI progress is expected to be driven by increased compute spending and hardware efficiency gains like half-precision operations, with the cost of human-level compute predicted to match human costs within a couple of decades.
- A survey of machine learning researchers provides only small evidence regarding timelines, as expert answers are viewed as largely uninformative and widely dispersed, suggesting linear extrapolations and simple models are currently superior forecasting tools.
- The "intelligence explosion" argument is considered lacking due to existing world feedback loops, with a higher expectation of gradual AI influence seeded across many systems rather than a sudden godlike takeover by a single entity.
- Rob Wiblin expects the current survey and episode to help listeners understand AI concerns and baseline forecasts, potentially convincing them to take risks more seriously or increase funding for safety work, with results verifiable in 10 years regarding task automation.
- Katja Grace identifies AI impact as a highly neglected field with tractable projects, noting that general research skills are better suited for forecasting than deep single-field expertise, and that early attention could yield an outsized impact even if the timeline is distant.
- AI Impacts work is expected to direct efforts toward AI safety, policy, and governance to address real-world implications, while the organization itself is positioned to use additional funding to hire personnel and pursue promising research into AI timelines.
- Efforts to shape future technology 20 years in advance are deemed worth pursuing despite a lack of track record, as the importance of the outcome justifies the attempt, and the field of AI safety is viewed as increasingly mainstream.