MIT 6.S094: Convolutional Neural Networks for End-to-End Learning of the Driving Task

Course Focus and Objectives

• The lecture transitions from general neural networks to Convolutional Neural Networks (CNNs) applied to images, specifically for autonomous driving.
• A key illustrative case study is a three-class classification problem: detecting traffic light colors (red, yellow, green).
• Two primary software tools are introduced: a browser-based JavaScript implementation (ConvNet.js) for immediate, accessible training, and TensorFlow for robust, GPU-accelerated offline training.

Computer Vision Fundamentals

• Images are treated as collections of discrete numerical pixels (0–255), allowing standard machine learning methods (supervised, clustering, semi-supervised, reinforcement) to apply to visual data.
• Problem Difficulty: Computer vision faces significant challenges including viewpoint variation, scale changes, occlusions, deformations, background clutter, inter-class variation, and illumination changes.
• Task Types:
  • Regression: Mapping images to continuous outputs (e.g., steering wheel angle).
  • Classification: Mapping images to discrete labels via probability outputs and thresholds.

Performance Benchmarks and Datasets

• Human Baseline: Humans achieve approximately 94% accuracy on the CIFAR-10 dataset.
• Algorithm Performance:
  • Random guessing yields ~10% accuracy on CIFAR-10.
  • K-Nearest Neighbors (KNN) using pixel-wise L1 distance achieves ~30% accuracy (with optimal K=7).
  • State-of-the-art CNNs have surpassed human performance, reaching 95.4% accuracy on CIFAR-10.
• Key Datasets:
  • MNIST: Handwritten digits (0–9).
  • ImageNet: Massive hierarchy of labeled images.
  • CIFAR-10/CIFAR-100: Small 32x32 pixel images for efficient algorithm validation (10 or 100 categories).
  • Places: Natural scene categorization.

Convolutional Neural Network (CNN) Architecture

• Core Mechanism: CNNs exploit spatial consistency by using filters with shared weights that slide across the input volume (height, width, depth).
• Spatial Invariance: Weight sharing allows the network to detect features (e.g., edges) regardless of their position in the image.
• Layer Types:
  • Convolutional Layers: Extract features using filters; output is a 3D volume.
  • Pooling Layers: Reduce dimensionality (e.g., Max Pooling) to lower computational load and focus on feature presence rather than precise location.
  • Fully Connected Layers: Used at the end to map extracted features to class scores (e.g., 10 neurons for digit classification).
• Advanced Applications:
  • Semantic Segmentation: Mapping image to image to identify object boundaries pixel-by-pixel.
  • Object Detection: Identifying candidate regions and classifying them.
  • End-to-End Learning: Replacing hand-engineered feature extraction with raw input-to-control mapping (e.g., stereo video to trajectory).

Autonomous Driving Context and Data

• Safety Statistics: Human-driven vehicles have ~30,000 fatalities/year (1 in 90 million miles); Tesla Autopilot (300M+ miles driven) has recorded only 1 fatality as of Dec 2016.
• Data Collection: MIT instruments Tesla vehicles with standard webcams (Logitech C920) to collect forward roadway video and driver state data.
• Sensor Integration:
  • LiDAR: Provides high-resolution 3D point clouds but struggles in adverse weather (rain/snow).
  • Computer Vision: Robust to weather changes; uses lane markings and vehicle detection for localization.
  • Radar: Reliable for distance measurement and collision avoidance.
• Driver State Monitoring: CNNs analyze driver video to detect gaze direction, head pose, pupil dilation, and micro-saccades to assess distraction or emotional states (e.g., frustration vs. satisfaction).

End-to-End Driving Projects

• DeepTesla (ConvNet.js): A browser-based project allowing users to train a CNN on real-world Tesla video to predict steering angles (-20 to +20 degrees).
  • Data: 10 video clips (5 autopilot, 5 human-driven) with synchronized CAN bus steering data.
  • Visualization: Features a barcode overlay to synchronize video and control streams; displays loss curves and network feature maps.
• TensorFlow Implementation: A more powerful offline version capable of handling larger datasets and complex architectures.
  • Output Comparison: Visualizes agreement (green) and disagreement (red) between predicted and actual steering wheels.
• Tutorial Recommendation: The course suggests starting with a TensorFlow implementation of traffic light classification to understand feature extraction before attempting full steering control.

Challenges and Limitations

• Data Volume: Neural networks require vast amounts of data; driving is challenging because the critical "edge cases" (crashes) are rare compared to normal driving.
• Generalization: Models excel at common scenarios but struggle to generalize to outliers or unseen complex environments.
• Accuracy Requirements: Driving demands near-100% accuracy, unlike classification tasks (e.g., cat vs. dog) where minor errors are acceptable.
• Training Constraints: Browser-based tools (ConvNet.js) lack GPU acceleration, limiting the size and depth of networks compared to TensorFlow.