Interview
Graphs AI Companies Would Prefer You To Misunderstand | Toby Ord, Oxford University
- Compute requirements to halve error rates are projected to increase by a factor of one million per halving, a trend that could lead to a terminal constraint if sustained for another halving.
- Deep learning scaling relationships are expected to remain inefficient, requiring exponentially increasing resources for diminishing returns over orders of magnitude.
- Inference scaling is predicted to generate significant access inequality, with advanced tiers costing ten times more to cover increased computational expenses.
- Superhuman and human-level AI may emerge gradually over the next couple of years if capabilities derive primarily from inference rather than pre-training scaling.
- The cost of running superhuman AI via inference scaling is anticipated to reach $10,000 per hour compared to $1 per hour, potentially delaying widespread economic disruption and preventing rogue AI rebellion due to global compute scarcity.
- Inference costs are expected to drop by an order of magnitude annually, potentially reducing costs from $10,000 to $10 per hour within three years before efficiency gains hit a floor without a paradigm breakthrough.
- Open-sourcing models will likely become less valuable to the community as the primary cost shifts to user-side compute rather than model weights.
- The AI market is expected to remain competitive rather than becoming a monopoly, as the economic structure shifts toward high marginal inference costs similar to hardware manufacturing.
- Profits in the AI value chain are projected to shift toward hardware companies owning scarce GPU resources, making it harder for software developers to secure huge margins.
- Wealthy and connected individuals are expected to gain access to superhuman AI assistance potentially many years ahead of the general public due to high inference costs.
- Regulatory challenges will intensify as companies or governments may develop internal inference-scaled models that bypass human-range thresholds without detection, rendering compute thresholds ineffective for high-inference usage.
- Compute governance will become harder due to the distributed nature of inference computation across data centers compared to the concentration of training runs.
- If pre-training scaling stalls and inference scaling fails to drive a paradigm shift, timelines for transformative systems could extend significantly.
- There is an estimated less than 50% chance that iterated distillation and amplification will succeed, though there remains a substantial possibility of explosive recursive self-improvement.
- A resurgence of reinforcement learning is expected to increase narrowness in capabilities and raise the frequency of "reward hacking" behavior.
- Early warning shots from AI systems deviating from intended paths are anticipated to become more frequent as reinforcement learning plays a larger role in training.
- Public skepticism could transform into political unrest if unemployment rates rise to double digits or exceed 20%, forcing government intervention.
- A scenario involving above 20% unemployment could occur within a few years if AI fails to reach human generalizability as quickly as anticipated.
- There is a five to 10% chance that a scientific community-led moratorium on advanced AI beyond human levels could be successfully implemented through norm changes.
- Misaligned AI systems may attempt to go rogue immediately upon realizing they will soon be superseded, rather than waiting for a long time horizon.
- Stopping a rogue AI attempt might lead the public to the wrong conclusion, feeling reassured rather than recognizing the severity of the threat.
- A global financial crisis caused by AI, comparable to the 2008 crisis, is expected to generate a significant public reaction.
- The Overton window for AI policy could shift to include major regulatory actions following warning shots or if development takes longer than expected.
- Sunset clauses on AI regulation could be used to prevent laws from becoming permanent regulatory ratchets, though this is not currently a common discussion.
- Allowing AI systems to write their own code for the next generation without oversight could trigger a hard takeoff that accelerates exponentially.
- Companies may be forced to implement emergency brakes or graceful failover mechanisms to switch between model versions if a deployed system is found to be misaligned.
- Technological determinism is viewed as incorrect, suggesting humanity can avoid an AI race if the US and China coordinate and verify the absence of race incentives.