Interview, Fireside Chat
Andrej Karpathy — “We’re summoning ghosts, not building animals”
- Andre Karpathy anticipates a "decade of agents" (approx. 10 years) is required to resolve current deficiencies in intelligence, multimodality, computer use, and continual learning, with architectures remaining as giant neural networks using gradient descent but featuring modified attention and sparser MLPs.
- Future "cognitive cores" for productive intelligence are expected to compress to roughly one billion parameters once training data is stripped of internet "slop," contrasting with current multi-trillion parameter models necessitated by noisy data.
- The industry is projected to shift budget from raw pre-training scale toward reinforcement learning, mid-training, and data quality improvement, as frontier models face diminishing returns from parameter growth.
- AI deployment will follow a "march of nines" trajectory where increasing reliability from 90% to 99.9% demands immense engineering effort, particularly in high-stakes domains like software engineering and self-driving, preventing sudden replacements.
- The "autonomy slider" will gradually shift, initially automating 10-20% of the economy in knowledge work (coding, call centers) by handling 80% of volume while humans supervise or manage the final 20% of complex cases.
- Predicted risks include "model collapse" due to low-entropy synthetic data, adversarial gaming of RLHF judges, and the potential disempowerment of humanity if they fail to adapt through education.
- Human-like "culture" and "self-play" among agents are identified as unresolved research frontiers, with current LLMs lacking the ability to generate high-entropy synthetic data or create knowledge specifically for other agents.
- Long-term outcomes suggest a qualitative shift where superintelligence causes a gradual loss of control, potentially leading to a society where labor is replaced and humans engage in "gym culture" for learning and flourishing rather than survival.
- Specific bottlenecks include current coding models' inability to write novel code, the lack of human-like knowledge distillation during sleep, and the "curse of knowledge" hindering effective education without simplified "first order terms."
- Ethereal "spirit entities" imitating internet documents are expected to emerge rather than biological-like animals, with AI eventually needing to strip memory from the cognitive core to rely on pure reasoning algorithms.