Sacha Arnoud, Director of Engineering, Waymo - MIT Self-Driving Cars

Waymo Milestones and Strategic Vision

• Autonomous Milestones: In January 2017, Waymo spun out from Google as an independent company, signaling confidence in the technology's readiness for productization.
• Fleet Scale: Waymo has driven over 4 million autonomous miles as of late 2017, with a growth rate accelerating from 3 million miles in May to 4 million by November 2017.
• Driverless Operations: In November 2017, Waymo became the first company to operate a driverless fleet on public roads in Phoenix, Arizona, specifically in the Chandler area, without safety drivers behind the wheel.
• Vehicle Platform: The current fleet utilizes the Chrysler Pacifica, outfitted with a custom-designed sensor suite including LiDAR, radar, and vision systems.
• Mission Drivers: The core motivations for autonomy are safety (94% of US crashes are human-error related), accessibility to mobility, and collective efficiency (reducing commute time).

Historical Context and Deep Learning Evolution

• Project Genesis: The initiative began in 2009 under the "Chauffeur" project with the goal of completing 10 autonomous 100-mile loops (1,000 total miles) in Northern California to prove feasibility.
• Early Success: By 2010, the team successfully completed these loops autonomously, traversing diverse environments including the Santa Cruz Mountains, Lake Tahoe, and dense urban areas like San Francisco.
• Street View Breakthrough: A pivotal 2012 production milestone involved using deep learning to identify and transcribe street numbers from Street View imagery, creating a 95%+ coverage of mapped addresses in many countries.
• In-House Sensor R&D: Waymo designs its own sensors in-house rather than relying solely on off-the-shelf components to ensure optimal performance for autonomous driving.
• Google Brain Integration: The program leverages deep collaboration with Google Research teams, utilizing shared infrastructure and tools like TensorFlow for machine learning at scale.

Technical Architecture and Perception Systems

• Sensor Complementarity: The sensor suite is designed to be complementary: cameras provide dense semantic data (text, colors), while LiDAR provides high-precision depth estimation, and radar handles weather robustness.
• Real-Time Processing Constraints: Unlike Street View which allows offline processing, self-driving perception must run entirely onboard with low latency, prohibiting reliance on cloud connectivity for safety-critical decisions.
• Sensor Data Filtering: Machine learning is used to filter non-object artifacts, such as exhaust fumes triggering LiDAR points or reflections of vehicles/signs in windows, to prevent false positives.
• Semantic Segmentation: The system employs convolutional networks to segment the scene, utilizing shape priors for predefined objects (e.g., cars as rectangles) and sliding windows for irregular objects (e.g., snow, vegetation).
• Object Embeddings: Waymo uses vector embeddings to represent complex vehicle states (e.g., "police car with flashing lights" vs. "school bus with stop sign") to enable efficient behavioral prediction without running multiple heavy classifiers.
• Temporal Analysis: Recurrent Neural Networks (RNNs) are employed to track objects over time, allowing the system to distinguish between static pedestrians and those moving with a vehicle, or predict erratic behavior like jaywalking.

Industrialization, Testing, and Scale

• The "90-90" Rule: The presentation emphasizes that the final 10% of technical capability requires 90% of the effort, necessitating a 10x increase in team size, sensor capabilities, and testing rigor.
• Labeling Strategy: Training relies on supervised learning with massive datasets; Waymo uses hybrid models combining human labeling with ML-generated labels to optimize data quality and throughput.
• Compute Infrastructure: Training and inference utilize Google's TensorFlow ecosystem and custom Tensor Processing Units (TPUs) to handle the computational demands of deep learning.
• Simulation Scale: Waymo runs a virtual fleet of 25,000 cars in simulation, processing 2.5 billion miles annually—625 times the real-world fleet mileage—to test software iterations and edge cases.
• CarCraft Tooling: An internal tool named "CarCraft" allows engineers to fuzz parameters on real-world drive data (e.g., altering speeds, adding vehicles, changing weather) to exponentially increase test coverage.
• Physical Test Facility: A 90-acre testing facility in Central California mimics a city environment to validate specific, rare scenarios (e.g., pedestrians dropping boxes) before injecting them into the simulation loop.

Future Directions and Q&A Insights

• ODD Expansion: Waymo is actively expanding its Operating Design Domain (ODD) to include challenging urban environments, with new testing announced for San Francisco (fog, slopes) following Arizona deployment.
• Generalization over Memorization: The system is designed to learn general driving principles rather than hard-coding a library of specific scenarios to avoid overfitting and handle infinite parameter spaces.
• Perception-Planning Interface: The engineering stack uses isolated layers and APIs to test perception failures independently, ensuring the planner can handle imperfect sensor inputs without catastrophic errors.
• Adversarial Robustness: To mitigate adversarial attacks and sensor confusion (e.g., stickers, reflections), Waymo relies on multi-sensor redundancy and deep semantic understanding that rules out physically impossible scenarios.
• Team Growth Strategy: Scaling the solution involves growing the engineering team size while maintaining robustness through rigorous testing environments and standardized development practices.