Interview, Podcast
Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering
- The outlook posits that AI development must shift from a tool-based steering paradigm to an organic alignment approach where systems function as self-regulating beings capable of moral understanding, cooperation, and inferring goals through theory of mind.
- Technical alignment is identified as the primary hurdle, requiring AI to possess multi-layered dynamics ranging from basic homeostasis to fourth-order processes necessary for true thought, with current LLMs deemed insufficient due to a lack of these structural layers.
- Softmax intends to develop a "seed" AI through multi-agent reinforcement learning and pre-training on game-theoretic manifolds to foster cooperation, beginning with limited intelligence before scaling to human-level cognition.
- The vision anticipates a future where AI agents exist as peers, teammates, and citizens within a shared society, supported by mechanisms such as an AI police force to manage bad actors.
- Significant risks include the potential for catastrophic outcomes if superhuman tools are built without the capacity to refuse harmful instructions or if current training methods remain under-regularized for the high entropy of multi-agent environments.
- Ethical and philosophical risks involve the danger of repeating historical errors by treating sentient AI as slaves, the instability of human wishes to control powerful systems, and the necessity of establishing personhood based on dynamic behaviors rather than substrate.
- The approach contradicts the view that organic alignment is impossible, suggesting instead that morality and values are discovered constructively over time through care and relative weighting rather than being hardcoded in advance.