Drago Anguelov (Waymo) - MIT Self-Driving Cars

Waymo's Operational Milestones & History

• Waymo celebrated its 10-year anniversary with a cumulative 10+ million miles driven autonomously on public roads.
• In 2015, the company executed the world's first fully autonomous ride on public roads in Austin with a blind passenger.
• In 2017, Waymo launched the first fully driverless commercial fleet in the Phoenix metro area, enabling on-demand service for errands and school transport.
• The company has expanded service to diverse environments, capturing data across varied cities, weather conditions, and traffic scenarios.

The "Long Tail" Challenge in Autonomous Driving

• Autonomous systems must handle the vast majority of common cases and the rare "long tail" of edge cases to achieve full self-driving capability.
• Edge cases include unusual object behaviors (e.g., bicyclists carrying stop signs), dynamic obstacles (e.g., falling poles), and construction zones.
• Vehicles must interpret and react to rule violations, such as emergency sirens or pedestrians running red lights, even when other road users do not.
• The scope of variability includes object types (appearance, pose), environmental conditions (seasons, time of day), and complex object relationships (occlusion, reflections).

Core AI Architecture: Perception, Prediction, and Planning

• Perception: Maps sensor inputs (cameras, LiDAR, radar) to scene representations, handling complex mappings of objects, semantics, and maps.
• Prediction: Anticipates future agent behavior over 1–10+ seconds using past behavior, scene semantics, and subtle cues (e.g., eye contact, gestures).
• Planning: Generates safe, comfortable, and communicative control commands (accelerate, steer) to navigate complex environments like dense urban school zones.
• Hybrid System Approach: Combines machine learning for complex perception with expert-designed rules to ensure safety when ML confidence is low (e.g., using LiDAR tracks as a safety fallback).

The "ML Factory" for Scalable Development

• Waymo operates a continuous cycle: vehicle deployment → data collection → data selection → labeling → model training → validation → new deployment.
• Infrastructure: Leverages Google's TensorFlow, specialized training hardware, and data centers for large-scale parallel compute.
• Data Strategy: Utilizes active learning to identify rare, uncertain, or inconsistent scenarios; prioritizes long-tail data over common cases.
• Auto-Labeling: Uses knowledge of future object trajectories in logged data to retroactively annotate training sets for improved model training.
• Neural Architecture Search (NAS): Automated systems search for optimal neural network architectures (e.g., NAS cells) to balance latency, compute, and accuracy, outperforming human-engineered designs.

Simulation and Agent Modeling

• Scale: Simulates the equivalent of 25,000 virtual cars driving 10 million miles daily, totaling over 7 billion simulated miles.
• Scenario Generation: Creates unique scenarios from safety expert analysis and derives variations from real-world log data.
• Agent Behavior Modeling: Employs a "menagerie" of simulation agents to test system robustness against the full spectrum of human behavior.
  • Imitation Learning: Uses end-to-end deep neural networks (trained on 60 hours of footage) to mimic human driving, augmented with road-mask and collision losses.
  • Trajectory Optimization Agents: Uses inverse reinforcement learning to tune "potentials" (attractors/repellers) that mimic specific driver personalities (conservative vs. aggressive).
• Testing Focus: Simulates adversarial and aggressive behaviors (e.g., weaving motorcycles, unsafe merges) to validate safety in long-tail scenarios.

Scaling and Future Deployment

• Generalization: The goal is a single model capable of adapting to new cities and unique local driving customs (e.g., "Pittsburgh Left") without retraining from scratch.
• Self-Improvement: Systems aim to elicit uncertainty to trigger active learning or use internal consistency checks (reasoning) to self-correct based on unlabeled data.
• Deployment Strategy: Requires extensive testing in controlled environments (90-acre Air Force Base staging grounds) before public road releases.
• Expert Systems: Non-ML expert rules remain critical for safety-critical functions where data is scarce or ML uncertainty is high.

Q&A Insights

• Synthetic Data: Simulation data is treated with caution; research focuses on domain adaptation (transforming sim data to look like real data) rather than direct replacement.
• Uncertainty Quantification: Techniques include ensembles, dropout, and leveraging environmental constraints (e.g., geometric consistency in 3D reconstruction) to measure model confidence.
• Pedestrian Simulation: Requires variable fidelity; high detail is unnecessary for distant pedestrians where perception resolution is low.
• Timeline: Wide-scale adoption is expected to be a gradual process requiring logistical tuning and rigorous testing across diverse environments, rather than an immediate rollout.