Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU)

• Current State of Unsupervised Learning: While research is active, unsupervised learning has not yet reached the maturity level of supervised learning techniques like convolutional networks.
• Data Motivation: Most available data today (images, speech, social networks, scientific data) is unlabeled, driving the need for models that discover structure without supervision.
• Hierarchical Representation: Deep learning frameworks aim to learn hierarchical data representations in an unsupervised manner to facilitate cross-domain applications.

Non-Probabilistic Models

• Sparse Coding:
  • Originated in 1996 to model early visual processing in the brain, functioning as an edge detector.
  • Learns a dictionary of bases ($\Phi$) where every data point is represented as a sparse linear combination.
  • Optimization involves alternating between solving a Lasso problem for activations (fixing bases) and a quadratic programming problem for bases (fixing activations).
  • Produces features useful for classification, achieving substantial improvements over PCA and logistic regression on datasets like Caltech-101.
  • Can be interpreted as finding an over-complete representation with an implicit non-linear encoding function and an explicit linear decoding function.
• Autoencoders:
  • Consist of an encoder (bottom-up pass) and a decoder (top-down reconstruction) designed to minimize reconstruction error.
  • Functions as a non-linear extension of Principal Component Analysis (PCA) when linear layers are used; otherwise, it learns richer features.
  • Constraints (e.g., bottlenecks, sparsity, binary features) are required to prevent the model from learning a trivial identity mapping.
  • Stacked Autoencoders: Can be trained greedily layer-wise to learn low-level to high-level features, often serving as a pre-training mechanism for limited labeled data.
  • Dimensionality Reduction: Demonstrated by compressing high-dimensional face data into 30-dimensional codes, effectively regularizing noise (e.g., removing glasses or mustaches) more sharply than PCA.
  • Semantic Hashing: Compresses data into binary representations (e.g., 20-bit codes) to enable efficient memory lookups and retrieval in sub-millisecond times for millions of images.
  • Predictive Sparse Decomposition: Combines sparse coding constraints with an encoder that approximates the latent code, optimizing both components.

Probabilistic Models

• Tractable Models:
  • Includes fully observed belief networks and neural autoregressive density estimators.
  • Recent success with Pixel Recurrent Neural Networks (Pixel CNNs) capable of generating high-resolution images by modeling pixel conditional probabilities.
  • Limitation: It remains unclear how well these models learn representations useful for downstream tasks like classification.
• Intractable Models:
  • Restricted Boltzmann Machines (RBMs):
    • Bipartite graphical models with visible (data) and hidden (feature detector) units.
    • Learning maximizes log-likelihood by matching data-driven correlations with model-driven correlations.
    • Training requires approximating the intractable partition function, typically using Contrastive Divergence (CD-1) to run the Markov Chain for only one step.
    • Capable of modeling binary, real-valued (Gaussian), and count data (Softmax), often discovering edges or topics similar to sparse coding.
    • Learn distributed representations where regions grow exponentially with parameters, outperforming mixture-based models (like LDA).
    • Deep Boltzmann Machines: Extend RBMs with multiple hidden layers and dependencies between them, though inference and learning become significantly more difficult.
  • Helmholtz Machines & Variational Autoencoders (VAEs):
    • Helmholtz Machines (1995): Comprise a stochastic generative network and a separate recognition network for approximate inference.
    • Wake-Sleep Algorithm: The original training method for Helmholtz machines that failed in practice; modern VAEs solve this via optimization.
    • VAE Mechanics: Define a generative process via cascaded stochastic layers and use variational inference to maximize a lower bound on the log-likelihood.
    • Reparameterization Trick: Introduced in 2014, this allows gradients to pass through stochastic layers by separating noise (auxiliary variables) from deterministic parameters, enabling efficient backpropagation.
    • Stochasticity Necessity: Required for modeling distributions of outcomes (e.g., generating diverse images for a single text caption like "a flying stop sign" where different samples show different cars or clouds).

Generative Adversarial Networks (GANs)

• Core Concept: A game-theoretic framework with two players: a Generator ($G$) and a Discriminator ($D$).
  • The Discriminator attempts to distinguish between real data and generated samples.
  • The Generator attempts to produce samples that fool the Discriminator into classifying them as real.
• Objective Function: A minimax game where the discriminator maximizes the probability of correct classification, and the generator minimizes the probability of the discriminator being correct (or maximizes the probability of being fooled).
• Advantages: Unlike VAEs or Boltzmann machines, GANs do not require explicit density specification or maximum likelihood estimation.
• Performance: Produce significantly sharper and more realistic images (e.g., L-SUN bedrooms, CIFAR, ImageNet) compared to VAEs, which often yield blurry results due to L2 loss penalties on pixel misalignment.
• Limitations: Evaluation metrics are not fully resolved; models may memorize training data or struggle to assign probability to new, diverse samples (e.g., a "new horse" class).

Application Examples & Results

• Multimodal Modeling: Hierarchical models successfully link images and noisy text tags (e.g., 1 million unlabeled images), utilizing pre-training on unlabeled data to improve performance on labeled subsets.
• Semantic Operations: Vector arithmetic on image representations allows for operations like "kitten + box" (kitten in a box) or replacing "day" with "night."
• Text Generation: Recurrent generative models trained on romance novels can generate syntactically correct but semantically inconsistent text.
• One-Shot Learning: Generative models trained on single examples can produce sketches or characters that are indistinguishable from human drawings in 50/50 voting tests.

Q&A Highlights

• GAN vs. VAE Clarity: GANs produce sharper images because they do not penalize specific edge placements (avoiding L2 loss blurring), focusing instead on fooling the discriminator. VAEs are prone to blurring but offer more globally coherent scenes.
• Pre-processing: No mandatory need for PCA pre-processing in VAEs; ZCA whitening or mean subtraction is sometimes used but often unnecessary if learning from raw pixels.
• Partition Function Estimation: Estimating the partition function for binary RBMs is NP-hard without specific structural constraints (unlike ferromagnetic Ising models), necessitating approximation methods like Contrastive Divergence.
• Text Representation: Semantic hashing for text can utilize word2vec, GloVe, or bidirectional GRUs to map sequences into differentiable binary spaces for efficient retrieval.