Deep Learning State of the Art (2019)

Context and Scope

• The presentation defines 2019 as a pivotal moment: the culmination of 2017–2018 deep learning breakthroughs while marking the beginning of the next phase of development.
• The lecture focuses on fundamental ideas and theoretical developments rather than specific benchmark results or GitHub algorithms.
• 2018 is identified as the "year of natural language processing" (NLP), comparable to the 2012 "ImageNet moment" for computer vision.
• The speaker explicitly excludes certain application domains, such as medical applications and protein folding, from this specific summary of trends.

NLP Architectural Evolution

• Encoder-Decoder RNNs: Early architectures encoded input sequences into fixed-size vectors using LSTM/GRU units to generate output sequences of varying lengths.
• Attention Mechanisms: Introduced to allow the decoder to selectively focus on specific parts of the input sequence rather than relying on a single fixed vector.
• Self-Attention: Enabled encoders to weigh different parts of the input sequence against each other to form richer representations of word context.
• Transformer Model: Utilizes self-attention in the encoder and attention in the decoder to capture rich context, replacing the sequential processing of RNNs.
• BERT (Bidirectional Encoder Representations from Transformers): Achieved a massive performance leap by masking 15% of tokens and training the model to predict them bidirectionally.
  • Enables rich contextual embeddings, sentence classification, sentence pair classification, and question answering.
  • The speaker identifies BERT as the single most significant breakthrough of the period.

Applied Deep Learning and Robotics

• Tesla Autopilot Hardware:
  • Hardware Version 1 used a fixed, non-learning Intel Mobileye system.
  • Hardware Version 2 implements an NVIDIA Drive PX2 system with eight cameras and an Inception network variant for drivable area segmentation and object detection.
  • Over 1 billion miles have been driven in Autopilot, with neural networks continuously learning via weekly updates.
• Neural Architecture Search (AutoML):
  • Systems like Google AutoML use reinforcement learning and RNNs to automatically determine architecture, hyperparameters, and modules.
  • Adanet: An 2018 advancement that uses reinforcement learning to build ensembles of multiple architectures rather than single models.

Data Strategies and Augmentation

• AutoAugment: Uses reinforcement learning to discover optimal data augmentation policies (e.g., rotation, color manipulation) for specific datasets.
  • Demonstrated transferability of augmentation policies across different datasets (e.g., from ImageNet to smaller datasets).
• Synthetic Data: NVIDIA and others use realistic and intentionally unrealistic synthetic scenes (varying lighting, rotation, objects) to train models.
  • Networks trained on synthetic data can achieve state-of-the-art performance on real-world tasks when fine-tuned with small amounts of real data.
• Annotation Automation:
  • Polygon RNN: Uses RNNs to suggest polygon outlines for segmentation tasks after a user provides a bounding box, reducing manual labeling costs.

Accessibility and Training Efficiency

• Dawn Bench: A Stanford-led competition measuring training speed and cost for achieving specific ImageNet/CIFAR-10 accuracy thresholds.
• FastAI Breakthrough: Achieved 93% ImageNet accuracy for $25 and 94% CIFAR-10 accuracy for 26 cents by jointly manipulating the learning rate and momentum parameters.
  • Democratizes deep learning for academic and independent researchers without massive computational resources.

Generative Models (GANs)

• BigGAN (Google DeepMind): Achieved high-resolution image generation through scaling model capacity and batch size rather than architectural novelty.
• Video-to-Video Synthesis (NVIDIA): Optimized for temporal consistency to eliminate "jumpiness" in generated video.
  • Applications include face edge detection, body pose to image generation, and object style transfer.

Perception and Segmentation

• Object Detection: Evolved through region-based methods (R-CNN variants) and single-shot methods, though the speaker notes a lack of recent breakthrough ideas compared to architecture changes.
• Semantic Segmentation:
  • DeepLabV3+: Current state-of-the-art for open-source systems.
  • Key Innovation: Utilizes dilated (atrous) convolutions with increasing rates to achieve multi-scale processing without increasing parameter count, effectively expanding the model's field of view.

Deep Reinforcement Learning

• AlphaGo Zero: Beat the original AlphaGo and defeated the human world champion in Go using only self-play and zero human supervision.
• AlphaZero: Applied to chess and Shogi, beating top engines (Stockfish) in four hours of distributed training.
  • Demonstrates "human-like" intuition by estimating board value with limited look-ahead, contrasting with the massive tree-search of engines like Deep Blue.
• OpenAI Dota 2:
  • 2017: 1v1 bot defeated top professional players.
  • 2018: OpenAI5 team lost two games in a 5v5 match at The International, highlighting the difficulty of imperfect information, teamwork, and long time horizons.
• Poker: AI has solved Heads-Up No-Limit Hold'em but has not yet solved general team Texas No-Limit Hold'em.

Frameworks and Future Outlook

• Framework Maturity: TensorFlow 1.0 and PyTorch 1.0 standardized the field; TensorFlow 2.0 (2019) introduced eager execution to improve accessibility.
• Theoretical Critique: Geoffrey Hinton suggested that backpropagation is "broken" and needs to be replaced by fundamentally new ideas.
• Future Direction: Current state-of-the-art relies heavily on 60s/80s ideas (stochastic gradient descent, backpropagation), suggesting the next major leap requires a revolution in optimization theory.