Lecture, Tutorial
MIT 6.S094: Deep Reinforcement Learning for Motion Planning
- Plans to unveil the "deep traffic" tutorial and project where participants solve a traffic problem using deep reinforcement learning, with a specific requirement to build a network achieving 65 miles per hour or higher.
- Predicts a "very special prize" for the competition winner, the details of which remain unannounced, though entries will be submitted via a browser-based interface where the simulation and training run on separate threads in Web Workers.
- Expects the competition scoring to involve 10 runs of approximately 30 simulated minutes each, utilizing the median speed of these runs, with results evaluated server-side to prevent cheating and only the highest score counted upon resubmission.
- Anticipates that the competition environment utilizes ComNetJS (developed by Andrej Karpathy) and the V8 engine to enable fast neural network training within the browser.
- Describes reinforcement learning as a process akin to being human, involving learning from delayed rewards and expanding the known portion of data to generalize beyond specific training sets.
- Warns that while neural networks can learn arbitrary logical functions, stacking many perceptrons can result in non-smooth outputs that are difficult to control, and hand-crafting functions for complex worlds may be infeasible due to sparsity.
- Notes that deep Q-learning can approximate optimal Q-functions through experience without preconceived notions, but requires significant world simulation and exploration to learn tasks like playing Atari games or navigating complex states.
- Predicts that deep Q-learning agents can learn to outperform humans in games without prior knowledge, yet acknowledges that the reality involves extensive data pre-processing and that the "truth is messier than the hope."
- Highlights that for complex problems, the number of states increases, requiring longer durations before epsilon decreases from 1 to 0 to transition from exploration to greedy behavior.
- Warns that incorrect reward functions can have huge negative impacts, particularly in safety-critical applications like driving where initial reward estimates may not be safe or productive for society.
- Foresees a future where reinforcement learning systems are deployed in arbitrarily defined worlds and driven by data from recorded video of human drivers before running agents in simulation.
- Expects feed-forward neural networks to be discussed as an alternative to recurrent networks, which are described as "really hard to train," and notes that using 10 outputs may perform better than the minimum 4 needed for digit classification.
- Warns participants that if the network performs poorly, the count of cars passed may become negative, and advises saving work frequently while noting the public leaderboard will grow as entries queue.
- Notes that links to the tutorial and competition (cars.mit.edu and deep traffic) may be temporarily removed from the website due to unexpected high traffic from Hacker News.
- Identifies the AlphaGo victory over the Go world champion as the most amazing accomplishment of deep reinforcement learning to date.