Lecture, Conference Presentation
Deep Reinforcement Learning (John Schulman, OpenAI)
Lex FridmanJohn Schulman, Hugo Larochelle, Andrej Karpathy, Richard Socher, Sherry Moore, Ruslan Salakhutdinov, Andrew Ng, Pascal Lamblin, Adam Coates, Alex Wiltschko, Quoc Le, Yoshua Bengio, Shubho Sengupta
- Deep reinforcement learning is predicted to be suitable for tasks where an agent interacts with an unknown environment to maximize a reward function, such as robotics, inventory management, and machine translation, though it may be overkill for problems with few parameters or where other methods suffice.
- Core techniques discussed include policy gradient methods, Q-learning, and Sarsa, with predictions that policy gradient methods are easier to get working out of the box, while Q-function methods offer higher sample efficiency when they work but are harder to debug and less general.
- Significant challenges identified include the stateful nature of reinforcement learning, which complicates decision-making due to delayed effects and non-differentiable reward functions, requiring agents to look ahead rather than act greedily.
- The outlook highlights that current methods often require extensive tuning, with discount factors typically set to 0.99 or 0.95 to account for exponential reward decay, and that performance degrades if the time between action and reward exceeds roughly 1/(1-gamma) steps without hierarchical approaches.
- Optimization risks involve high variance in basic estimators, local minima in the behavior space, and the potential for large policy updates to disrupt learning; Trust Region Policy Optimization (TRPO) and baseline functions are proposed to mitigate step size and variance issues respectively.
- Applications demonstrated include learning to walk for simulated humanoids without prior knowledge, real-time manipulation tasks, and solving complex games like Atari and Go, though real-world deployment often requires weeks of training time and patience.
- Future expectations suggest active research into model-based learning for improved sample efficiency, handling non-stationary environments, and developing hierarchical RL for long time horizons, with specific contests and real-world experiments anticipated soon.
- Theoretical foundations cover Markov Decision Processes defined by state and action spaces, where the goal is maximizing expected cumulative reward, with specific considerations for episodic settings, deterministic versus stochastic policies, and the use of feature engineering or neural networks for function approximation.
- Practical implementations note that while Q-learning works without knowing system dynamics, methods like A3C provide asynchronous policy gradient implementations, and model-based approaches are still being validated for compelling speed and efficiency gains.
- Risks include the difficulty of credit assignment with sparse rewards, the impact of time-step discretization on exploration and delay, and the fact that changing dynamics parameters slightly usually maintains policy robustness only if the underlying stationary assumption holds.