Foundations and Challenges of Deep Learning (Yoshua Bengio)

Five Essential Ingredients for Human-Level Machine Learning:

• Sufficient data to represent complex world knowledge.
• Algorithms capable of representing complicated functions (e.g., neural networks).
• Adequate computing power for training and inference.
• Efficient models that avoid the "curse of dimensionality" by utilizing compositional representations.
• Computational efficiency during model usage (inference), which probabilistic models sometimes fail to provide due to intractability.

Mechanisms for Beating the Curse of Dimensionality:

• Deep learning bypasses the exponential growth of data configurations by composing small units into larger functions.
• Two distinct types of composition are utilized:
  • Parallel Composition: Distributed representations where units in a single layer combine to represent exponentially many configurations (e.g., word embeddings).
  • Sequential Composition: Hierarchical depth where output from one level serves as input for the next, building increasingly abstract features.
• Neural networks achieve exponential growth in representable regions with only a linear growth in parameters, unlike non-parametric methods (e.g., decision trees, SVMs) which require parameters to grow linearly with regions, failing at generalization for unseen configurations.

Empirical Evidence of Feature Discovery:

• Experiments in Antonio Torralba's lab at MIT demonstrated that convolutional networks trained to classify scene types autonomously learn semantic features (e.g., detecting people, animals, buildings, lighting) without explicit instruction.
• Theoretical results indicate that deep ReLU networks generate a piecewise linear function where the number of pieces grows exponentially with network depth.
• While single-layer networks can theoretically represent any function, doing so often requires an exponential number of units; depth allows for efficient representation of specific functions found in the real world.

Optimization Landscape of High-Dimensional Neural Networks:

• Contrary to 1990s theories about exponentially many poor local minima, high-dimensional optimization landscapes are dominated by saddle points rather than local minima.
• As network size increases, the distribution of costs at local minima converges, meaning most local minima found are of high quality and close to the global minimum.
• The probability of a true local minimum (where curvature is positive in all directions) becomes exponentially small in high dimensions, whereas saddle points (curving up in some directions, down in others) are prevalent.
• Stochastic Gradient Descent (SGD) generally avoids getting stuck at saddle points but may struggle with ill-conditioning or "narrow" regions where curvature varies significantly.

Challenges in Training and Long-Term Dependencies:

• Current systems still struggle with long-term dependencies, though solutions include skip connections, multiple time scales, and attention mechanisms.
• Attention mechanisms function as an internal state mechanism, allowing networks to read/write from memory elements that persist without vanishing gradient issues.
• Complex reasoning and structured output tasks (e.g., machine translation) require combining deep learning with reinforcement learning and planning.

The Critical Role of Unsupervised Learning:

• Unsupervised learning enables the acquisition of knowledge from vast amounts of unlabeled data, mimicking how humans (e.g., toddlers) learn physics and object properties without explicit instruction.
• Capturing the joint distribution of data via unsupervised methods allows models to answer multiple types of queries (e.g., image-to-text, text-to-image) rather than just predicting a single label $P(Y|X)$.
• Unsupervised pre-training acts as a regularizer for supervised tasks by discovering explanatory factors of the input data.
• For safety-critical applications like self-driving cars, generative models learned via unsupervised methods are essential to simulate dangerous scenarios that cannot be experienced in reality.

Disentangled Factors and Abstraction:

• A key goal for future intelligence is disentangling independent factors of variation (e.g., separating identity from pose in images) to allow generalization to new tasks.
• Multi-level abstraction (from pixels to words to meaning) is necessary because higher levels of representation make complex queries and actions more efficient to compute.
• Current neural networks often "cheat" by using spurious cues; true intelligence requires understanding the underlying generative processes of the world.

Bridging Machine Learning and Neuroscience:

• Backpropagation, while effective for machines, is not biologically plausible; research is exploring alternatives like "target propagation" and gradient estimation methods resembling Spike Timing-Dependent Plasticity (STDP).
• Future progress may depend on integrating credit assignment principles from supervised learning into reinforcement learning frameworks.

Current Limitations and Future Directions:

• The "curse of dimensionality" remains a problem when data requirements exceed current computational capacities, necessitating better algorithmic inductive biases.
• Deep learning currently requires orders of magnitude more data than humans to achieve comparable performance, highlighting the need for better unsupervised learning of world models.
• Compressing large networks is possible due to redundancy, but current parameter redundancy (e.g., unit permutations) offers limited compression gains compared to structural optimization.

Administrative and Event Details:

• The speaker (Yoshua Bengio) and co-authors Ian Goodfellow and Aaron Courville have a book available for pre-sale by MIT Press, with expected shipping in December.
• The speaker acknowledges Microsoft bugs during the presentation that were resolved.
• A dinner for women attendees is scheduled immediately following the workshop session.