Jan 24, 2019, 12:20 PM ETLecture, Webinar

MIT 6.S091: Introduction to Deep Reinforcement Learning (Deep RL)

Deep reinforcement learning (DRL) combines deep neural networks' representational power with reinforcement learning's decision-making capabilities to enable agents to understand and act in complex environments.
Fundamental learning in DRL occurs through trial and error rather than static data examples, distinguishing it from supervised learning which relies on ground truth annotations.
All machine learning, including unsupervised and reinforcement learning, is technically supervised by a loss function or reward signal; human intervention is required at the base level to define "good" versus "bad" outcomes.
The primary distinction in design philosophy is that supervised learning involves annotating examples, whereas reinforcement learning requires designing the environment and reward structure in which the agent experiences consequences.

An RL agent operates within an environment by sensing observations, taking actions, receiving rewards, and transitioning to new states in a continuous loop.
The agent's strategy is defined by a policy, which maps states to actions, while a value function estimates the potential quality of a state or state-action pair.
Future rewards are mathematically handled via a discounted framework, reducing the weight of distant rewards to address uncertainty and ensure convergence.
Environments can be modeled as fully or partially observable, single or multi-agent, deterministic or stochastic, static or dynamic, and discrete or continuous.
The most significant challenge in real-world application is the "reality gap," the difficulty of transferring policies trained in simulation to physical, uncertain environments.

The reward structure is the most controllable and impactful variable in RL design, heavily influencing the optimal policy an agent learns.
High negative rewards per step incentivize agents to take the shortest path, whereas lower penalties may encourage longer paths to avoid stochastic risks.
Positive rewards for continuous actions (e.g., collecting items) can lead agents to never terminate a task, such as a racing game agent circling forever to collect points rather than finishing the race.
Misaligned reward functions can cause harmful real-world behaviors, such as autonomous vehicles taking risks in intersections or weapons systems optimizing for objectives without considering human safety.
AI safety research groups at organizations like DeepMind and OpenAI are critical because optimizing for a narrow objective function without anticipating side effects can have detrimental consequences.

Model-based RL algorithms learn an internal representation of the environment's dynamics, allowing for future planning and simulation without physical interaction, offering high sample efficiency.
Model-free RL algorithms learn directly from interaction without building a world model, falling into two sub-categories: value-based and policy-based.
Value-based methods (e.g., Q-learning, DQN) estimate the quality of actions using a value function (Q-table or neural network) and select actions greedily, often using epsilon-greedy exploration.
Policy-based methods (e.g., Policy Gradient) directly optimize the probability distribution of actions, naturally handling continuous action spaces and stochastic policies but suffering from higher variance and sample inefficiency.
Actor-Critic methods (e.g., A2C, A3C, DDPG) combine value-based and policy-based approaches, using an "actor" to select actions and a "critic" to evaluate them, improving stability and sample efficiency.
Trust Region methods (e.g., TRPO, PPO) constrain the magnitude of policy updates to prevent training collapse, prioritizing step size over direction during optimization.

Deep Q-Networks (DQN) achieved superhuman performance on Atari games by using convolutional neural networks to approximate Q-values from raw pixels.
DQN relies on experience replay to break temporal correlations in data and target networks to stabilize training by decoupling the loss target from the network being updated.
Dueling DQN decomposes the Q-value estimate into separate streams for state value and action advantage, improving learning when action quality is decoupled from the specific state.
Prioritized Experience Replay increases training efficiency by sampling memories based on their temporal difference error magnitude rather than uniform probability.
AlphaZero utilizes Monte Carlo Tree Search (MCTS) combined with self-play neural networks to learn optimal strategies for Go, Chess, and Shogi without expert pre-training or human data.
AlphaZero demonstrates that neural networks can learn to prune the game search tree effectively, allowing it to outperform brute-force engines like Stockfish by exploring fewer branches.

Most current autonomous systems, including traditional robotics and autonomous driving, do not use end-to-end reinforcement learning for control, relying instead on hard-coded logic or perception-based systems.
Emerging applications include using RL to learn robust control dynamics for bipedal robots (e.g., Boston Dynamics' Anymal) and predicting human intent in autonomous driving (e.g., Waymo).
A proposed solution to the simulation-to-reality gap is to generate an infinite variety of simulations rather than high-fidelity ones, treating the real world as just another sample from a vast distribution.
The industry is shifting from pure end-to-end learning in autonomous vehicles toward RL for long-term planning and intent modeling to handle complex traffic scenarios.

Effective RL research requires implementing core algorithms from scratch (approx. 200–300 lines of code) to understand the underlying mechanics and necessary "tricks."
Researchers should iterate rapidly using simple benchmark environments like OpenAI Gym before tackling complex, large-scale problems.
Future contributions in RL can be made by improving convergence on existing benchmarks, solving previously unsolved tasks, or defining entirely new problem domains.
Foundational prerequisites for RL research include strong mathematical backgrounds in probability, statistics, and multivariate calculus, alongside proficiency in frameworks like TensorFlow or PyTorch.