Lecture, Tutorial
MIT 6.S094: Deep Reinforcement Learning for Motion Planning
- Course Objective: Unveiled "Deep Traffic," a deep reinforcement learning project tasking participants with solving traffic flow problems to achieve an average speed of 65 mph or higher on a seven-lane highway simulation.
- Machine Learning Taxonomy:
- Supervised Learning: Requires input-output pairs with known ground truth to learn mappings for future generalization; characterized as "memorization."
- Unsupervised Learning: Operates without output labels, aiming to find underlying data structures from raw data alone.
- Semi-supervised Learning: Utilizes a dataset where only a small fraction (e.g., ImageNet) is labeled, with the goal of expanding known ground truth.
- Reinforcement Learning (RL): Positioned between semi-supervised and unsupervised, involving an agent that learns via time-delayed rewards from an environment without explicit ground truth for actions.
- Neural Network Fundamentals:
- Perceptrons: Binary output neurons (0 or 1) using weighted inputs, bias, and thresholds; capable of approximating universal logic gates like NAND.
- Activation Functions: Critical transition from binary perceptrons to smooth, continuous activation functions (e.g., Sigmoid) to enable gradient-based weight adjustment.
- Feed-Forward Architecture: Linear flow from inputs to outputs without loops; contrasted with Recurrent Neural Networks (RNNs) which possess memory but are harder to train.
- Network Structure Example: Handwritten digit classification using 784 input neurons (28x28 pixels), a hidden layer (e.g., 15 neurons), and 10 output neurons representing digit probabilities.
- Training Mechanics:
- Forward Pass: Propagates input data through the network to generate predictions.
- Backpropagation: Uses gradients to adjust weights based on the loss function (e.g., squared error) to minimize the difference between predictions and ground truth.
- Optimization: Employs variants of gradient descent to navigate the highly non-linear loss landscape, often requiring stochastic elements to avoid local minima.
- Reinforcement Learning Framework:
- Markov Decision Process (MDP): Models the environment where agents transition between states via actions, receiving rewards or punishments based on the new state.
- Policy: The strategy mapping states to actions; optimized to maximize the discounted sum of future rewards.
- Value Function: Estimates the long-term desirability of specific states or state-action pairs.
- Q-Learning: An off-policy algorithm approximating the optimal Q-function (state-action value) via the Bellman equation update rule: $Q(s,a) \leftarrow Q(s,a) + \alpha [R + \gamma \max Q(s',a') - Q(s,a)]$.
- Exploration vs. Exploitation: Managed via epsilon-greedy strategies where agents initially explore randomly (high epsilon) and gradually shift to exploiting known optimal actions (low epsilon).
- Deep Q-Learning (DQN) Breakthroughs:
- Pixel-Based Input: DeepMind's 2013 work demonstrated agents playing Atari games using raw pixels as input, bypassing the need for hand-crafted physics models.
- Experience Replay: A critical technique storing past experiences in a memory buffer to be sampled randomly, breaking temporal correlations and preventing convergence to local optima.
- Performance Results: DQN achieved human-level or super-human performance on Atari games (e.g., Breakout, Boxing) without game-specific customization.
- Advanced Architectures: Mention of "General Reinforcement Learning Architecture" (GORILLA) enabling distributed simulation and learning, similar to AlphaGo's approach of learning from expert games followed by self-play.
- Deep Traffic Simulation Details:
- Environment: A 2D grid-based simulation of a seven-lane highway where cars occupy grid blocks; safety systems prevent collisions by constraining valid moves.
- Input Representation: The agent perceives a discretized map where empty blocks have a speed limit (80 mph), occupied blocks reflect vehicle speed, and the agent's location is marked with a high value.
- Action Space: Five discrete actions: move left, move right, stay, accelerate, or slow down.
- Evaluation Metric: Average speed calculated over 10 runs of approximately 30 simulated minutes each; the median speed is used for leaderboards.
- Implementation & Competition Logistics:
- Technology Stack: Entirely client-side execution using JavaScript, HTML5 Canvas, and Web Workers for parallel training at speeds up to 1,000 frames per second.
- Library: Utilizes
ConvNet.jsby Andrej Karpathy for neural network definitions, supporting customizable layer sizes, activation functions (e.g., ReLU), and temporal windows. - Submission Process: Users modify network parameters in the browser, train the model, and submit to a server-side queue for evaluation; the highest score is retained on the public leaderboard.
- Safety Constraints: A built-in safety system prevents the agent from driving off-road or colliding, ensuring stable simulation during the learning process.
- Future Implications for Autonomous Driving:
- Hybrid Approach: Suggests combining supervised learning (using recorded human driving data for ground truth) with RL (simulation-based exploration) to address the complexity of real-world driving.
- Reward Function Risks: Highlights the danger of misaligned reward functions, citing the need for agents to learn safety and productivity rather than just maximizing a specific score metric.
- Generalization: Emphasizes that while DQN succeeds in simulated environments (like Go or Atari), transferring these capabilities to real-world safety-critical domains like driving remains a significant challenge due to the need for accurate world modeling and delayed reward signals.