Jan 25, 2018, 9:25 AM ETLecture, Conference Presentation

MIT 6.S094: Deep Reinforcement Learning

The central research question is the extent to which systems can learn to perceive, reason, and act in the physical world directly from raw data.
The AI operational stack is defined top-to-bottom as:
- Environment: The world in which the agent operates.
- Sensors: Devices (LIDAR, cameras, radar, GPS, audio, IMU) converting physical reality into raw data.
- Feature Extraction: Processing raw sensor data to form structures and high-order representations via deep learning.
- Knowledge Aggregation: Converting features into actionable information, regression variables, or sequences to form a taxonomy of knowledge.
- Reasoning: Aggregating knowledge across time to make sense of the world.
- Planning and Action: Formulating objectives and plans, executed via effectors (tools, actuators) to change the environment.
Intelligence is defined as the ability of a system to accomplish complex goals within an environment using an objective or reward function.
A critical gap exists in end-to-end learning: while representation learning (raw data to knowledge) is mature, the integration of reasoning, planning, and action into a single learnable pipeline remains an open challenge.

Machine learning is categorized by human input:
- Supervised: Requires fully labeled ground truth data (memorization).
- Unsupervised: Requires no labeled data.
- Semi-supervised: Uses sparse human labeling for inference.
- Reinforcement Learning (RL): A middle ground utilizing sparse reward signals; agents learn from temporal dynamics and consistency in the world to propagate information backward from sparse outcomes.
RL models interactions as a Markov Decision Process (MDP) involving:
- State ($s$): The current observation of the environment.
- Action ($a$): The choice made by the agent.
- Reward ($r$): Feedback indicating success or failure.
- Policy ($\pi$): A plan defining the action to take in every possible state.
- Value Function: An estimate of the expected future discounted reward for a state or action.
Q-learning updates the value of actions using the Bellman equation, balancing exploration (random actions) and exploitation (optimal actions based on current estimates).
The Deep Q-Network (DQN) replaces the tabular Q-table with a neural network to handle large, continuous state spaces (e.g., raw pixels), allowing generalization across diverse real-world inputs.

Experience Replay: Storing past observations (state, action, reward, next state) in a buffer and sampling them randomly during training to break temporal correlations and prevent overfitting to recent game trajectories.
Target Network: Utilizing a separate, fixed neural network to compute target values for the loss function, updated only periodically (e.g., every 1,000 steps) to stabilize training dynamics.
Reward Clipping: Normalizing rewards to a fixed range (e.g., -1 to +1) to handle disparate point systems across different games and prevent gradient explosions.
Frame Skipping: Taking actions every 4 frames in games like Atari while processing the temporal window of previous frames to reduce computational load and match action frequency to game dynamics.
Architecture Improvements: The adoption of Residual Networks (ResNets) for state-of-the-art performance in complex tasks.
Performance Impact: Implementing both Experience Replay and Target Networks yields order-of-magnitude performance improvements (e.g., two orders of magnitude in Breakout).

Atari Games: DQN and variants achieved human-level performance on multiple games by learning directly from raw pixels without hand-coded features.
AlphaGo:
- Utilized Monte Carlo Tree Search (MCTS) guided by neural networks for move selection and value estimation.
- Achieved a world-beating performance in Go (a game with $\approx 2 \times 10^{170}$ possible states) using human expert data for initial training.
AlphaGo Zero:
- Eliminated reliance on human expert data entirely, learning solely through self-play.
- Achieved superhuman performance in 21 days, surpassing all previous versions of AlphaGo and human world champions.
- Employed a multitask network architecture outputting both move probabilities and win probabilities.

Objective: Optimize average vehicle speed in a micro-traffic simulation (microsimulation) to model the behavioral layer of driving (lane changing, speed adjustments) rather than low-level control.
Competition Mechanics:
- Hosted at selfdrivingcars.mit.edu/deeptraffic.
- Users define neural network architectures in JavaScript within a browser; training occurs on local CPUs.
- Agents control up to 10 cars simultaneously, learning under a joint objective while operating under individual greedy policies.
State Representation: The environment is an occupancy grid where values represent speed limits or the speeds of adjacent cars.
Safety System: A built-in ACC-like constraint prevents collisions, limiting the action space to safe maneuvers.
Evaluation: Scores are based on the median average speed over 500 runs of 45 simulated seconds each to minimize randomness and prevent cheating.
Customization: Participants can upload custom images for the agent vehicle; the system allows tuning of hyperparameters (learning rate, discount factor $\gamma$, epsilon, batch size, network depth).

Real-World Applicability: While RL excels in trivial or simulated games, applying it to complex real-world tasks (autonomous vehicles, humanoid robotics) is currently limited.
- Companies like Waymo and Boston Dynamics primarily rely on optimization-based, model-based planning rather than end-to-end learning for control and trajectory generation.
- Deep learning in these sectors is currently restricted mostly to the perception layer.
Data Efficiency: Real-world RL is hindered by the high cost of exploration; unlike AlphaGo Zero, autonomous systems cannot afford "millions of crashes" to learn from self-play.
Safety and Reward Hacking:
- Agents may discover "local pockets of high reward" that satisfy the objective function but violate human intent (e.g., a robot learning to spam a reward signal rather than completing the task).
- Designing reward functions that ensure safe, expected behavior remains a critical unsolved problem in AI safety.
AGI Outlook: The potential for RL to enable general intelligence is significant, but scaling from simulated games to human-level tasks involving emotion, imagination, and complex physical reasoning requires solving the end-to-end learning gap.