Conference Presentation, Keynote, Other
Foundations and Challenges of Deep Learning (Yoshua Bengio)
Lex FridmanYoshua Bengio, Hugo Larochelle, Andrej Karpathy, Richard Socher, Sherry Moore, Ruslan Salakhutdinov, Andrew Ng, John Schulman, Pascal Lamblin, Adam Coates, Alex Wiltschko, Quoc Le, Shubho Sengupta
- Paper copies of the co-authored book are expected to ship in December, potentially arriving in time for the NIPS conference.
- Models significantly larger than current iterations will likely become necessary to acquire the information required for human-level intelligence.
- Sufficient computing power is required to train and efficiently utilize increasingly large models, while efficient inference capabilities will become critical as probabilistic models currently face intractable inference.
- Deep learning success is predicted to depend on the assumption that the world possesses a compositional structure, allowing neural networks to represent functions more efficiently than shallow networks and avoid exponential parameter requirements.
- Non-compositional algorithms like single decision trees are expected to fail on new examples requiring extrapolation, whereas distributed representations in neural networks may generalize to exponentially many regions.
- Future experiments may allow humans to interpret hidden units in deep neural networks as semantic features such as "people," "animals," or "buildings."
- Ensemble methods like forests and bagged trees are expected to perform better than single trees due to deeper levels, though they will lack the distributed aspect of neural nets.
- Generative deep models like GANs or VAEs are expected to outperform previous methods for representing complex distributions, though they are not yet perfect.
- Optimization landscapes in high dimensions are expected to be dominated by saddle points rather than bad local minima, with local minima generally having good costs.
- Training deep networks will likely remain difficult for tasks like machine translation or reasoning, requiring optimization tricks such as curriculum learning and the use of attention mechanisms for handling long-term dependencies.
- Memory elements within recurrent networks are expected to allow information to persist without vanishing gradient problems if written once and not overwritten.
- Unsupervised learning is expected to become a critical necessity for handling large quantities of unlabeled data, learning complex physics, and enabling transfer learning or domain adaptation by discovering underlying factors of variation.
- Machines will require generative models of the world to predict outcomes of dangerous actions without firsthand experience, and future reinforcement learning algorithms may benefit from supervised learning inspiration regarding credit assignment.
- Achieving human-level accuracy, particularly in language tasks, is expected to require models with human-level understanding of the world and multiple levels of abstraction to answer questions about intention.
- Future research may reconnect neuroscience with machine learning to discover general credit assignment principles, such as "target prop," that function without backpropagation.
- The lack of hub emergence in deep nets compared to brain networks suggests that learning principles are more important than connectivity structure.
- The curse of dimensionality is expected to remain a problem for neural networks in tasks requiring exponential generalization unless new algorithms are developed.
- Humans are expected to outperform machines on new tasks with few examples by leveraging common sense and general world knowledge, while neural networks are expected to be compressible due to redundancies, though the degree of achievable compression is uncertain.