Lecture, Fireside Chat
Ilya Sutskever: OpenAI Meta-Learning and Self-Play | MIT Artificial General Intelligence (AGI)
- Backpropagation is predicted to remain the fundamental algorithm for circuit search until systems reach human-level and beyond capabilities, a milestone expected to be achieved before the brain's backpropagation-like mechanisms are fully understood.
- Neural net processor speeds are expected to increase very dramatically over the next few years, making neural net cycles cheap and critical for converting compute into data to enhance generalization.
- Meta-learning is anticipated to improve significantly as algorithms become robust to test tasks falling outside training distributions, while simply scaling up language models to be larger and deeper is expected to solve generative language modeling problems surprisingly well.
- Training at test time and inference at test time are expected to provide an important performance boost to language models, and future algorithms will likely enable agents to learn by inferring the goals and strategies of others.
- A society of agents featuring language, theory of mind, negotiation, and complex social systems is expected to emerge from successful self-play approaches, driving a very rapid increase in agent competence if trends observed in the Dota bot persist.
- Cooperation is expected to become the winning strategy in sufficiently open-ended games even if not explicitly intended, and strategies discovered by the Dota bot are expected to represent transferable fundamentals useful for human players.
- Alignment is predicted to be resolved as a technical problem, though determining correct goals for AI systems is expected to become a highly challenging political issue, particularly as societal impact grows massive and overwhelming once computers surpass human capabilities.
- A large number of people are expected to care strongly about AI alignment and goals as the impact of technologies like self-driving cars and automation gradually increases.
- Reinforcement learning algorithms are expected to address current data inefficiencies, though finding the best short program explaining data remains computationally intractable with today's tools.
- Simulation mismatch in robotics is expected to be solved by training policies to quickly adapt to the real world through randomized physics parameters such as friction, masses, and dimensions in the simulator.
- Agents are expected to require continued training in new environments with violated assumptions, adhering to a "school to work" analogy rather than relying on perfect static physical models.
- Policies trained with randomized physics simulation are expected to function much better on real physical robots compared to those trained without such randomization.
- Evolutionary strategies are predicted to be ineffective for reinforcement learning with large policies but are worth considering for evolving small, compact objects like pieces of code.
- The best small circuit solving a problem via backpropagation can be found in practice for large numbers of equation terms, despite theoretical proofs indicating that finding the optimal neural network is not always possible.