Interview, Podcast
Ishan Misra: Self-Supervised Deep Learning in Computer Vision | Lex Fridman Podcast #206
- Self-supervised learning (SSL) is defined as a paradigm where the data itself serves as the supervision signal, eliminating the need for explicit human annotation of concepts.
- The term "self-supervised" is preferred over "unsupervised" to explicitly denote that the algorithm extracts specific supervision signals from the data structure rather than learning in a vacuum.
- Supervised learning relies on humans labeling data (e.g., drawing bounding boxes around objects), which has proven non-scalable; the ImageNet dataset required 22 human-years to annotate only 14 million images across 22,000 concepts.
- Semi-supervised learning combines a small set of labeled data with vast amounts of unlabeled data, using the unlabeled portion to increase model confidence and consistency.
- Core mechanisms for unlocking supervision involve specific "tricks" that impose constraints on the data to force the model to learn world physics or semantics.
- In Natural Language Processing (NLP): The dominant technique is masking, where words in a sentence are hidden (e.g., "The [MASK] jumped over the fence"), and the model predicts the missing word based on context, leveraging the distributional hypothesis (words in similar contexts have similar meanings).
- In Computer Vision: The most effective technique is contrastive learning via data augmentation, where two different crops or perturbations of the same image are treated as "positive pairs" that must have similar feature representations.
- Common augmentations include changing colors, contrast, brightness, and cropping, though these are currently "human-designed" biases rather than learned processes.
- Video prediction: Models are trained to predict the next frame or sequence of a video, forcing them to learn object permanence, gravity, and motion dynamics without human labels.
- Architectural and methodological developments have significantly advanced the field.
- Transformers and Self-Attention allow models to pay attention to all elements in a sequence or image simultaneously, enabling better context understanding than local-window-based ConvNets.
- Contrastive Learning: Involves creating an embedding space where "positive" pairs (related data points) are pulled together and "negative" pairs (unrelated data points) are pushed apart.
- Non-contrastive methods (e.g., clustering, self-distillation) avoid the need for explicit negative samples, which are often noisy or insufficient in large-scale settings.
- SwAV is a non-contrastive method that performs online clustering of image crops, ensuring equal partitioning across a fixed number of clusters to prevent representation collapse (where all inputs map to the same feature).
- SEER is a system that trained large convolutional models on 1 billion uncurated internet images, demonstrating that SSL can work effectively "in the wild" without the bias of curated datasets like ImageNet.
- Multimodal learning leverages the correlation between different data modalities (e.g., audio and video) to learn richer representations.
- In Audiovisual Instance Discrimination, models train separate networks for video and audio streams, forcing their feature embeddings to be similar when the modalities align (e.g., a video of a guitar playing paired with its sound).
- This approach successfully learns to localize sound sources in video (e.g., identifying where a guitar is being strummed) without explicit audio-visual labels.
- Scaling and infrastructure challenges require specialized hardware and data strategies.
- Data Augmentation vs. Data Quantity: Current vision SSL paradigms rely so heavily on data augmentation that improving the quality of augmentation is more impactful than simply increasing the volume of raw data.
- Distributed Training: Training billion-parameter models on billion-image datasets requires minimizing communication costs between compute nodes, often favoring synchronized training approaches.
- RegNets are efficient network architectures optimized for both FLOPs and memory usage, allowing massive models to fit on single GPUs.
- Limitations and future directions for deep learning and SSL include:
- Data Efficiency: Deep learning models struggle to generalize from very few examples (few-shot learning) compared to humans who use inductive bias and domain knowledge.
- Continual Learning: Neural networks suffer from "catastrophic forgetting," where learning a new task erases knowledge of previous tasks, unlike humans who retain accumulated knowledge.
- Reasoning vs. Recognition: Current networks excel at snap-judgment recognition but lack the compositional ability to reason about novel, complex scenarios (e.g., physics puzzles) they haven't seen.
- Explainability: The "black box" nature of self-supervised models makes it difficult to trace how they arrive at specific decisions, posing challenges for safety-critical applications like autonomous driving.
- Autonomous Driving applications:
- Self-supervised learning is applicable through predictive modeling, where cars learn to predict future states or driver actions based on video streams.
- Active Learning is utilized via Tesla's "data engine," where cars collect edge cases (near-miss or failure scenarios) to retrain and improve models, focusing on the "long tail" of rare driving events.
- Ishan Mizra estimates fully solved autonomous driving (safe enough for public liability) is 5 to 10+ years away, dependent on sensor improvements and the ability to scale data collection for edge cases.
- Philosophical and broader AI implications:
- Objectness and Concepts: SSL allows complex concepts like object boundaries, permanence, and symmetry to emerge naturally from pixels without explicit definition.
- Simulation: Mizra is skeptical that simulators alone can solve complex AI problems (like autonomous driving) due to the difficulty of simulating human behavior and the "reality gap."
- Embodiment and Emotion: True general intelligence may require physical interaction with the world and the capacity for emotion, which arises from the mismatch between prediction and observation.
- Consciousness: To foster deep human-AI interaction, systems may need to display elements of consciousness or the capacity to suffer, as this facilitates anthropomorphism and connection.
- The Meaning of Life: The conversation suggests the meaning of life is an endless quest; AI's rigid objective functions differ from human evolution, where diverse and evolving objectives make us interesting.
- Advice for researchers and students:
- Problem Selection: Choose problems that are exciting and solvable within a realistic timeframe (e.g., a PhD duration).
- Writing Strategy: Focus on articulating a single, simple idea deeply rather than cramming multiple concepts; start writing early in the research process to refine the core hypothesis.
- Technical Practice: "Get your hands dirty" by struggling with implementation and debugging rather than relying immediately on libraries or tutorials.
- Programming: Python is the dominant language; PyTorch is preferred in research for its imperative design and ease of debugging compared to TensorFlow's declarative model.
- Mindset: Cultivate hunger for the work, resilience against frequent failure, and the belief that failure is a necessary step toward discovery.