Lecture, Panel
Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU)
Lex FridmanRuslan Salakhutdinov, Hugo Larochelle, Andrej Karpathy, Richard Socher, Sherry Moore, Andrew Ng, John Schulman, Pascal Lamblin, Adam Coates, Alex Wiltschko, Quoc Le, Yoshua Bengio, Shubho Sengupta
- Unsupervised learning is not yet equivalent to supervised learning, with the goal of automatically finding the right representations from vast amounts of unlabeled data remaining unachieved.
- The data space for images, speech, social networks, and scientific data is continuously expanding, with the majority of current data being unlabeled.
- The presentation covers sparse coding, autoencoders, clustering, and tractable probabilistic models like neural auto-aggressive density estimators and pixel recurrent neural networks.
- Intractable models including Boltzmann machines and variational autoencoders (VAEs) are addressed, noting significant recent community development in these areas.
- Recent model advancements such as generative adversarial networks (GANs) and moment matching networks are highlighted for their ability to operate without specifying density.
- Generative models, including restricted Boltzmann machines and deep Boltzmann machines, are introduced as tools capable of modeling complicated data.
- Specific attention is given to recent developments in variational autoencoders and the introduction of GANs as a final class of models.
- Expectations are lowered regarding current results, acknowledging that despite extensive work over the last decade, the field requires more progress.
- Online and efficient optimization for sparse coding has seen significant improvements over the last 10 years.
- Autoencoders are noted for their ability to compress data into 20-dimensional binary codes, enabling memory storage and sub-millisecond retrieval from millions of images.
- Semantic hashing functions mapping data to binary presentations are identified as useful for large database searches, with bidirectional GRUs currently being the method of choice for text representation.
- Recent advances in the last couple of years have improved conditional generation and image in-painting capabilities.
- It remains to be seen how effective current representations are for downstream tasks like image classification.
- Implementing the Contrastive Divergence algorithm is described as requiring approximately 10 lines of MATLAB code.
- Learning Boltzmann machines is expected to be difficult due to the high-dimensional nature of computing the second term, with deep Boltzmann machines becoming increasingly hard as layers and dependencies increase.
- Pre-training on a million unlabeled data points can lead to performance improvements even when only 25,000 labeled images are available.
- Hierarchical models are expected to allow information to flow stably between images and text, addressing confusion caused by noisy tags and the scarcity of specific animal images.
- Historical models like Helmholtz machines and the 1995 wake-sleep algorithm are discussed, noting they were previously considered ineffective until recent work solved their implementation issues.
- The VAE is described as a Helmholtz machine using multilayer nonlinear neural network conditionals, where the 2014 reparameterization trick significantly reduces training variance and improves efficiency.
- VAEs utilizing Gaussian loss functions tend to produce diffused or less clear images compared to GANs, which implicitly handle edge accuracy and avoid such blur.
- GANs are anticipated to introduce specific artifacts, whereas VAEs do not exhibit these issues, leading to hybrid approaches that use VAEs for coherent scenes and GANs for sharpening.
- PCA preprocessing is not necessary for VAEs, though it may sometimes offer benefits.
- Variational inference and variational methods are expected to underperform compared to contrastive divergence approximations for binary RBMs.
- FPRAS for estimating log partition functions in RBMs is predicted to fail in practice as the problem becomes NP-hard without ferromagnetic structure assumptions.
- Despite progress made in the last decade, unsupervised learning is characterized as a field where substantial further progress is still needed.