Other
Tesla AI Day Highlights | Lex Fridman
- The presenter identifies Tesla's AI Day presentation as a unique convergence of scale and engineering required to solve the autonomous driving and general robotics perception tasks, which are deemed significantly harder than widely anticipated.
- The core technical innovation is the shift from operating in 2D image space to predicting directly in 3D vector space, a difficult step that aligns machine learning with the physical reality of the world.
- Tesla has implemented multi-scale feature-level fusion of camera sensor data before detection, contrasting with traditional approaches that combine individual sensor decisions only after separate processing.
- The system incorporates temporal context by using a spatial Recurrent Neural Network (RNN) architecture, creating a 2D grid around the vehicle where each cell functions as an RNN to model time and kinematics.
- Planned future improvements aim to move the fusion of space and time earlier in the network pipeline to enable more seamless, full end-to-end driving with multiple modalities.
- Neural networks are now utilized as heuristics for action-space planning to prune search trees, a technique analogous to their use in Monte Carlo tree search for games like Go and Chess, allowing the system to avoid local optima.
- Data annotation strategy involves a custom in-house team labeling vector space directly rather than image space, with subsequent projection, which streamlines the training of the new architecture.
- Tesla employs a self-supervised "fleet auto-labeling" method where clips of video, IMU, GPS, and odometry from multiple vehicles at the same location are registered to jointly annotate static and moving objects.
- Simulation is used to generate labels for ultra-complex edge cases, such as scenes with hundreds of pedestrians, which are difficult to capture and label accurately in the real world.
- The retraining infrastructure currently utilizes approximately 10,000 GPUs, with full end-to-end network retraining occurring every one to two weeks.
- The "Dojo" training supercomputer, built with in-house D1 chips, features single tiles delivering 9 FLOPS and arbitrary scalability, with a demonstrated cluster reaching 1.1 exaflops.
- The operational loop of data collection, auto-labeling, manual edge-case labeling, retraining, and deployment is characterized as an infinite scale process without a defined performance ceiling.
- Dojo is positioned as a potential "AI training as a service" competitor to AWS and Google Cloud, capable of handling any large-scale machine learning problem via a simplified PyTorch deployment model.
- The underlying neural network architecture and data engine are explicitly designed to be transferable beyond automotive applications to domestic, factory, and humanoid robotic forms.
- The presenter views the Tesla Bot as a parallel solution to the autonomous driving problem, potentially resolving perception, movement, and manipulation while human-robot interaction remains an ongoing research frontier.