Aug 5, 2024, 10:00 AM ETInterview, Fireside Chat

How Waymo Is Using GenAI to Build a Better Driver

Dimitri Dolgov first entered autonomous vehicle development during the 2007 DARPA Urban Challenge at Stanford, characterizing the event as a "pivotal" foundational moment for the industry.
Dolgov co-founded the Google self-driving project in 2009 alongside a dozen colleagues from the DARPA challenges, with support from Larry Page and Sergey Brin.
The Google self-driving project officially spun out as Waymo in 2016, marking the transition from research project to standalone commercial entity.
Current vehicle testing includes over 15 million miles driven in full autonomy (rider-only mode) and tens of billions of simulated miles.
Waymo vehicles have demonstrated a 3.5x reduction in injury-causing accidents and a 2x reduction in police-reportable incidents compared to human drivers in operational areas.
A joint study with Swiss Re covering 3.8 million miles showed a 76% reduction in property damage collisions and a 100% reduction in claims related to bodily injury where Waymo was the contributing party.

Early autonomous systems (2007–2009) relied on classical decision trees, hand-engineered features, and kernel-based computer vision.
The 2012 breakthrough in Convolutional Neural Networks (AlexNet/ ImageNet) enabled significant advancements in object detection and classification across cameras, LiDAR, and radar.
The 2017 introduction of Transformers allowed Waymo to apply sequence prediction to behavior modeling, decision-making, and simulation, moving beyond pure perception.
Current development focuses on combining legacy "Waymo AI" (perfection in specific driving tasks) with Vision-Language Models (VLMs) to leverage general world knowledge.
Waymo rejects a binary "rules-based vs. end-to-end" debate, arguing that while end-to-end and foundation models provide significant boosts, they remain insufficient for full autonomy without additional layers of safety and planning.
Scaling laws in autonomous driving require not just data volume but "right kind of data" that specifically trains models on rare, long-tail edge cases.
The company prioritizes training massive foundation models and then distilling them into smaller, onboard systems to meet real-time computational constraints.
End-to-end models still face specific weaknesses regarding hallucinations, explainability, goal-oriented planning, and understanding 3D spatial dynamics.

Simulation is critical for evaluating system readiness, allowing Waymo to generate tens of billions of miles of experience for every 15 million physical miles driven.
Synthetic data in simulation serves three primary functions: generating variations of real-world events (scaling rare scenarios), creating purely synthetic scenarios never seen in the physical world, and stress-testing sensor perception.
Simulation realism is quantified across three dimensions: sensor/perception realism, dynamic actor behavior realism, and macro-level statistical realism (frequency of occurrence).
Waymo employs a "readiness framework" that compares system performance against a model of an "attentive, very good human driver" rather than just average human statistics.
Validation methodologies include continuous public reporting of safety data, partnerships with reinsurers like Swiss Re for objective impact analysis, and iterative model testing against specific high-stakes events.

A primary remaining challenge is scaling to "magical" user experiences, specifically the complexity of pick-up and drop-off logistics in dense urban environments.
Waymo faces significant difficulty in dynamic semantic tasks, such as determining when to block a driveway, reacting to opening garage doors, or navigating complex parking lots safely.
The company operates 24/7 in San Francisco, Phoenix, Los Angeles, and Austin, handling diverse weather conditions including fog, rainstorms, and dust storms.
Hardware strategy relies on a redundant sensor suite combining cameras, LiDAR, and radar, currently deploying fifth-generation sensors and working toward sixth-generation hardware.
Cameras provide high-resolution color data, LiDAR offers direct 3D measurement and operates in total darkness, and radar provides velocity data and redundancy in low-visibility weather.

Waymo's mission is to build a "generalizable driver" deployable across ride-hailing, delivery, trucking, and personal vehicle markets.
The company utilizes a hybrid go-to-market strategy, operating its own app in Phoenix while simultaneously partnering with Uber and Uber Eats to accelerate access.
The industry has seen a "thinning" of competitors compared to the rapid commoditization of LLMs, due to the physical world's noise, safety consequences, and real-time processing requirements.
Barriers to entry include the high cost of safety validation, regulatory burdens, and the impossibility of shortcuts in achieving full autonomy.
Waymo aims to deploy the technology broadly and deliberately, exploring various commercial structures to maximize access while maintaining safety standards.

Dolgov advises job seekers to identify "problems that matter" to the world, emphasizing that meaningful problems are often inherently difficult.
He encourages building persistently without being discouraged by the unknown or negative feedback from others.
The field is characterized by a long, costly journey, but Dolgov views the current ability to offer safe, driverless rides as the most powerful manifestation of AI in the physical world today.