Foundations of Deep Learning (Hugo Larochelle, Twitter)

Course Scope & Prerequisites

• Presentation covers foundational feedforward neural networks, training mechanics (loss, backprop, SGD), and modern deep learning techniques (dropout, batch normalization).
• Speaker notes that while the lecture covers basics, detailed derivations are available in his online lectures (searchable via "Hugo Lavachel," noting he is distinct from the skateboarder).

Feedforward Neural Network Architecture & Notation

• A multilayer feedforward network maps an input vector $x$ to an output $f(x)$, typically for classification where output units correspond to classes (e.g., 10 units for digits 0–9).
• Forward Propagation Formula:
  • Layer $k$ pre-activation $a^{(k)}$ is a linear transformation of the previous layer's activation $h^{(k-1)}$: $a^{(k)} = W^{(k)}h^{(k-1)} + b^{(k)}$.
  • Hidden layer activation $h^{(k)}$ is computed via a non-linear function $g$: $h^{(k)} = g(a^{(k)})$.
• Activation Function Choices:
  • Sigmoid: Squashes pre-activation to range $(0, 1)$; prone to saturation (gradients vanish) when inputs are very large or small.
  • Tanh: Squashes pre-activation to range $(-1, 1)$; also saturates at extreme values, causing gradient issues.
  • ReLU (Rectified Linear Unit): Outputs $0$ for negative inputs and the input itself for positive values; unbounded above but bounded below.
    • Partial derivative is $1$ if $a > 0$ and $0$ otherwise; prevents vanishing gradients better than sigmoid/tanh for positive activations but can "die" if units remain negative.
  • Softmax (Output Layer): Converts pre-activations to a probability distribution over $C$ classes by exponentiating and normalizing (summing to 1).

Universal Approximation Capability

• A single hidden layer neural network with sufficient units and non-linear activations (e.g., sigmoid, tanh) can approximate any continuous function arbitrarily well.
• This theoretical result does not provide a method for finding the optimal weights/biases, necessitating an optimization-based training approach.

Training Framework: Empirical Risk Minimization

• Objective: Minimize the average loss over the training set plus a regularization term: $\min_{\theta} \frac{1}{T}\sum L(f(x), y) + \lambda R(\theta)$.
• Loss Function: Typically Cross-Entropy (Negative Log-Likelihood) for classification, measuring the divergence between the empirical distribution (one-hot label) and the model's predicted distribution.
• Regularization: L2 regularization (weight decay) is common; gradients are typically not computed for bias terms.
• Optimization Algorithm: Stochastic Gradient Descent (SGD) is the standard method.
  • Parameters are initialized randomly (weights) and to zero (biases).
  • Weights cannot be initialized to zero or identical values to prevent symmetry and gradient stagnation.
  • Xavier/Glorot initialization (e.g., uniform distribution scaled by layer size) is recommended for tanh-based networks to keep units unsaturated.

Backpropagation Mechanics

• Gradients are computed via the chain rule, propagating from the output layer back to the input.
• Key Gradient Formulas:
  • Gradient w.r.t. weights: $\nabla_W = \nabla_a \cdot (h^{(k-1)})^T$.
  • Gradient w.r.t. bias: $\nabla_b = \nabla_a$.
  • Gradient propagation to previous layer pre-activation: $\nabla_{a^{(k-1)}} = (W^{(k)})^T \cdot \nabla_a \odot g'(a^{(k-1)})$.
• Vanishing Gradients: Occur when activation function derivatives ($g'$) approach zero (saturation in sigmoid/tanh) or are zero (ReLu below 0), halting learning in lower layers.
• Implementation: Modern libraries (PyTorch, TensorFlow, Theano) automate this via computational graphs (flow graphs), executing a "backward" pass to compute gradients automatically.

Hyperparameter Tuning Strategies

• Grid Search: Tests all combinations of hyperparameter values; computationally expensive and prone to "holes" if specific values cause complete failure (e.g., high learning rates).
• Random Search: Samples values from specified distributions (often log-uniform for learning rates); generally more efficient than grid search for high-dimensional spaces.
• Early Stopping: Monitors validation set performance; halts training if validation error stops improving, preventing overfitting and saving compute.
• Learning Rate Decay: Reduces the step size $\alpha$ when validation performance plateaus to refine convergence.
• Mini-batches: Compute gradients on subsets (e.g., 64 or 128 samples) rather than single examples to leverage vectorized matrix-matrix operations for speed.

Advanced Optimization Techniques

• Momentum: Accumulates previous update directions to accelerate convergence in consistent directions: $v_t = \beta v_{t-1} + \nabla L$.
• Adaptive Learning Rates:
  • AdaGrad: Scales learning rate per parameter by the cumulative sum of squared gradients.
  • RMSprop: Uses an exponential moving average of squared gradients instead of cumulative sum.
  • Adam: Combines momentum with RMSprop-like adaptive learning rates; widely used in practice.

Debugging & Validation Practices

• Gradient Checking: Compare analytical gradients from backprop against finite difference approximations to verify implementation correctness.
• Overfitting a Tiny Dataset: Run the network on a very small subset (e.g., 50 examples) to ensure it can achieve near-zero training loss; failure indicates bugs in initialization, gradients, or learning rate.

Motivations for Deep Networks (Multi-Layer)

• Biological Plausibility: Mimics the visual cortex hierarchy (edges $\to$ parts $\to$ objects).
• Theoretical Efficiency: Deep networks can represent specific Boolean functions and complex patterns with exponentially fewer units than shallow networks.
• Empirical Success: Driven breakthroughs in speech recognition and visual object recognition.

Challenges in Training Deep Nets & Solutions

• Underfitting: Often caused by vanishing gradients; mitigated by better optimization (GPU acceleration, adaptive rates), longer training, or better initialization.
• Overfitting: Caused by excessive model capacity relative to data; addressed via regularization.

Dropout

• Mechanism: Stochastically sets hidden unit activations to zero during training (typically with $p=0.5$) to prevent co-adaptation of features.
• Training Impact: Requires resampling the dropout mask for every example; typically doubles the number of epochs needed to converge.
• Inference: Masks are replaced by their probabilities (e.g., scaling activations by 0.5) to approximate an ensemble of many thinned networks.
• Effect: Acts as a regularizer; often renders standard weight decay less critical in conjunction with batch normalization.

Batch Normalization

• Mechanism: Normalizes pre-activations within a mini-batch to zero mean and unit variance, followed by learnable affine transformation ($\gamma x + \beta$).
• Training Phase: Computes mean/variance using only the current mini-batch to avoid expensive full-dataset passes.
• Test Phase: Uses global running averages of mean and variance computed during training.
• Benefits:
  • Stabilizes training and allows higher learning rates.
  • Reduces dependence on careful initialization.
  • Often reduces the need for dropout (dual effect of regularization and optimization).

Q&A Highlights

• ReLU Sparsity: ReLU induces sparsity because it outputs exactly zero for all negative inputs; this is theoretically linked to sparse coding optimization models where linear transformations followed by ReLU-like thresholds yield sparse representations.