Interview, Podcast
What superforecasters and experts think about existential risks | Ezra Karger
- AI risk concern groups project a 40% probability of bad outcomes, including extinction or a 50%+ population drop, within the next 1,000 years, whereas skeptics assign a 30% probability to such clusters of bad outcomes.
- Both groups anticipate human well-being will be high by 2100 absent extinction, though skeptics expect this level to be lower than that of the concern group.
- Regarding the development of powerful AI, both the concern group (88%) and skeptics (90%) expect systems exceeding human cognitive performance in 95% of economically relevant domains to emerge by 2100.
- The concern group estimates a 4% chance that powerful AI will be developed but not widely deployed due to coordinated human decision-making, while skeptics estimate this probability at 20%.
- The concern group expects significant belief updates by 2030 resulting from alignment researchers changing their minds, major powers going to war, or an independent body concluding AI can autonomously replicate, acquire resources, and evade deactivation.
- Skeptics expect the largest belief updates by 2030 to result from superforecasters changing their views on existential risk, alongside significant updates driven by progress in lethal technologies or AI systems influencing democratic elections.
- The Forecasting Research Institute predicts a "muddy picture" regarding whether higher accuracy correlates with greater risk concern, though it expects reference classes to reduce risk forecasts significantly in low-probability domains.
- Forecasting Research Institute plans to produce better answers for low-probability domains within one to two years and anticipate a dynamic benchmark updating daily or weekly by summer to track AI progress.
- The Forecasting Research Institute expects short-term forecast resolutions to be muddled by year's end but aims to gather evidence on the accuracy of different groups.
- Forecasts indicate superforecasters will outperform on some questions while domain experts will be more accurate on others, particularly AI-related questions where conditions are moving quickly.
- Access to reference classes across the probability space is expected to improve forecasting ability in low-probability domains, and recruiting a sample of 2,000 defined experts is planned to improve study representativeness.
- Asking experts and superforecasters to predict what other groups will think is expected to elicit more accurate responses than unincentivized beliefs, potentially leading to convergence between groups if an independent body concludes AI can evade deactivation.
- The Forecasting Research Institute expects LLMs to currently be noticeably worse than humans on average but notes they may approach human performance on specific questions and be superior where human uncertainty is near 50%.
- LLMs are expected to be less accurate than humans on questions in the lower probability space but can improve human forecasts when used as assistants.
- Forecasting on a menu of policies and their causal effects is expected to help decision-makers make better decisions regarding nuclear risk.