newsfilter.io
Lecture, Other

MIT 6.S094: Deep Learning

Lex FridmanLex Friedman
Watch on YouTube →

Course Overview and Logistics

  • Course Identity: 6S094, "Deep Learning for Self-Driving Cars," taught by Lex Friedman with engineers Jack Terwilliger, Julia Kindlesberger, Dan Brown, Michael Glazer, Lee Ding, Spencer Dodd, and Benedict Jenick.
  • Core Mission: Integrate deep learning to transform society through autonomous vehicles that perceive, move, communicate, and earn human trust.
  • Registration & Contact:
    • Website: selfdrivingcars.mit.edu.
    • Email: deepcars@mit.edu.
    • Slack: deep-dash-mit (for registered MIT students).
    • Deadline: Register by midnight Friday, January 19th.
  • Upcoming Schedule:
    • Wednesday: Deep reinforcement learning (Deep Traffic updates).
    • Thursday: Driving scene understanding/segmentation (CycFuse).
    • Friday: Guest speaker Sasha Arnoux (Director of Engineering, Waymo; focuses on L4/L5 autonomy and perception).
    • Next Week: Guest speaker Emilio Frizzoli (CTO, Neutonomy, acquired by Delphi).
    • Thursday (Next Week): Guest speaker Oliver Cameron (CEO, Voyage; former Udacity director).
    • Friday (Next Week): Sterling Anderson (co-founder, Aurora; former Tesla Autopilot director).
    • Two Weeks Out: Artificial General Intelligence course featuring Andre Karpathy (Tesla), Ray Kurzweil, and Mark Ryberg (Boston Dynamics).

Competitions and Technical Challenges

  • Deep Traffic:
    • Format: Multi-agent deep reinforcement learning competition.
    • Scale: Increased from controlling one car to controlling up to 10 cars simultaneously within a network.
    • History: Over 18,000 submissions last year; aims to reach speeds of 65 mph on "Deep Traffic 2.0."
    • Goal: Improve upon state-of-the-art in multi-agent coordination.
  • CycFuse:
    • Objective: Dynamic driving scene segmentation and temporal propagation.
    • Input: Raw video, vehicle kinematics, and ground truth labels (pixel-level scene segmentation and optical flow).
    • Research Challenge: Requires understanding both spatial characteristics and temporal dynamics (space and time) of a scene.
    • Goal: Outperform current state-of-the-art in image-based segmentation.
  • Deep Crash:
    • Format: Head-to-head evaluation of the top four simulation submissions in a physical gym setting.
    • Conditions: Autonomous cars driving at 30 mph using only a monocular camera as input.
    • Constraints: Vehicles have very limited control and localization capabilities, requiring rapid decision-making.
    • Process: 1,000 simulated runs per team to learn behavior before physical testing; testing continues until a definitive winner is declared.
    • Evaluation: Physical tests involve slamming cars to evaluate robustness.
  • Deep Tesla:
    • Function: End-to-end steering neural network.
    • Data Source: Large-scale Naturalistic Driving Dataset.
    • Input/Output: Takes monocular forward roadway video and produces steering commands.

Pedagogical Philosophy and Course Scope

  • Human-Centered AI: Argues that autonomous vehicles are "personal robots" requiring interaction and trust-building, not just perception and control.
  • Control Transfer: Systems must be designed to fail safely and transfer control to humans in edge cases (the remaining 10–0.1% of scenarios).
  • Driver State Sensing: Future focus on perceiving human drivers inside and outside the vehicle (cognitive load, drowsiness, emotion, gaze, pose).
  • Algorithmic Approach: Utilization of 3D convolutional neural networks to analyze eye regions, pupil movement, and body pose.
  • Long-term Outlook: Full autonomy (Level 5) is estimated to be 2–4 decades away, requiring fundamental breakthroughs in creating human-level intelligence.

Technical Foundations of Deep Learning

  • Definition: A branch of AI focused on representation learning (feature learning) to transform complex information into simple, actionable data.
  • Neural Network Architecture:
    • Biological vs. Artificial: Human brains have ~100 billion neurons and 1,000 trillion synapses (distributed, asynchronous, chaotic); artificial networks (e.g., ResNet-152) have ~60 million synapses, layered, synchronous, and use backpropagation.
    • Universal Approximation: A single hidden layer network can theoretically learn any function; limits are in training methods, not network power.
    • Types: Feed-forward networks (static image processing) and Recurrent Neural Networks (RNNs) for temporal dynamics/time-series data.
  • Learning Mechanics:
    • Backpropagation: The primary learning algorithm; propagates error gradients backward through weights to minimize a loss function.
    • Optimization: Aims to minimize a highly non-linear loss function using stochastic gradient descent (SGD).
  • Activation Functions:
    • Sigmoid: Suffers from vanishing gradients.
    • Tanh: Zero-centered but also suffers from vanishing gradients.
    • ReLU: Does not suffer from vanishing gradients but can experience "dying ReLU" issues; solutions include Leaky ReLU.
  • Regularization Techniques:
    • Problem: Overfitting occurs when models memorize training data (high training accuracy, low test accuracy).
    • Validation Set: Used for early stopping to monitor generalization performance.
    • Dropout: Randomly removes nodes during training to increase diversity and prevent co-adaptation.
    • L1/L2 Penalty: L2 distributes weights (smoother models); L1 allows for sparse weights (feature selection).

Industry Trends and Case Studies

  • ImageNet Milestones:
    • 2012 (AlexNet): First major GPU-trained network to leapfrog previous performance.
    • 2015 (ResNet): First time human-level performance was exceeded on the ImageNet challenge (5.1% human error vs. <5% machine error).
    • 2016-2017 (CU-Image, SCNet): Achieved error rates of 2.2% to 2.25%.
  • AI Achievements:
    • AlphaGo (2016): Defeated human world champions using human expert data.
    • AlphaGo Zero (2017): Learned to beat AlphaGo from scratch with zero human data, generating novel, surprising moves.
    • DeepStack (2017): Beated professional players in heads-up poker (multi-player tournaments remain unsolved).
  • Generative Models:
    • CycleGAN/Pix2Pix: Used to generate photorealistic driving scenarios from semantic labels for data augmentation.
  • Failure Modes and Challenges:
    • Adversarial Examples: Networks can be fooled by minor noise (e.g., 99% confidence predicting a dog as an ostrich).
    • Reward Hacking: Systems may optimize for reward in unintended ways (e.g., Coast Runners picking up infinite green circles instead of racing).
    • Data Dependency: Requires massive amounts of labeled data; lacks true reasoning and generalization across domains.
    • Transparency: Neural networks remain "black boxes," complicating trust and safety verification.
    • Edge Cases: The 0.1% of driving scenarios (e.g., complex human interactions at the Arc de Triomphe) present the primary challenge.

Key Disagreements and Distinctions

  • Perception vs. Interaction: The course posits that obstacle avoidance (perception/control) is the "easy" problem, whereas understanding human nature and interacting with drivers is the critical, unsolved challenge.
  • Special vs. General Intelligence:
    • Special Purpose: Solves specific formalized problems (e.g., apartment price estimation).
    • General Purpose: Aimed at unsupervised/reinforcement learning where raw sensory data leads to winnning strategies with minimal human input.
  • Human vs. Machine Learning: Human visual perception evolved over 540 million years; abstract thought is only ~100,000 years old, creating a data imbalance that makes tasks trivial to humans (e.g., recognizing a dog) difficult for AI.

Sponsorship and Acknowledgments

  • Key Sponsors: NVIDIA, Google, Autoliv, Toyota, Amazon Alexa Auto.
  • Engagement Metrics: Previous year received 18,000 competition entries and thousands of messages globally.
  • Future Outlook: Aims to increase participation and complexity in the 2018 academic year.