Lecture, Conference Presentation
MIT 6.S094: Deep Reinforcement Learning
- An AI safety talk is scheduled for Friday at 1 p.m. in Stata 32123, while a course on artificial general intelligence will commence in two weeks.
- The Deep Traffic competition at selfdrivingcars.mit.edu/slash/deeptraffic has been expanded to allow control of up to 10 cars via a single neural network, with submissions accepted every five minutes and starter code available on GitHub.
- The simulation operates at 1,000 times the speed of the browser visualization, utilizes a road representation as an occupancy grid, and incorporates a safety system mimicking ACC, radar, and ultrasonic sensors to prevent collisions.
- Winning the competition requires an average speed exceeding 70 miles per hour, which must be sustained despite a hard simulation speed limit of 80 miles an hour that is considered impossible to maintain on average.
- Evaluation metrics are calculated by averaging speeds from 10 runs of approximately 45 simulated seconds each, with the final score derived from the median of 500 server-side runs to mitigate randomness and discourage repeated submissions of the same code.
- Training can occur locally on smartphones or laptops using only CPU resources via a "run training" button, with network state periodically shipped to the server for visualization, though transferring saved weights via JavaScript to new browsers is currently not supported.
- Participants can customize numerous learning parameters including temporal windows, layer counts, gamma, epsilon, learning rates, optimizers, momentum, batch sizes, and L1/L2 decay, alongside input slice dimensions for forward/backward length and lane visibility.
- Agents are trained under a joint objective but act greedily for individual optimization, allowing the system to scale to an arbitrary number of cars while utilizing experience replay and fixed target networks to stabilize the learning process.
- Current industry examples like Waymo rely minimally on deep learning for perception with a focus on model-based optimization for trajectory, whereas AlphaGo Zero achieved superior performance in Go using a two-headed ResNet that predicted states via self-play without human data.
- Real-world reinforcement learning faces challenges such as local reward pockets leading to undesirable effects and the difficulty of learning to drive from scratch without millions of simulated crashes, contrasting with current successful robots that rarely use end-to-end learning.
- The fundamental research question remains whether full-stack systems can learn end-to-end from raw sensory data to reasoning and action, and future safety research must focus on designing reward functions to constrain algorithms against unintended evolutionary behaviors.