Fireside Chat, Interview
Fei-Fei Li is Solving the Hardest Problem in Robotics | World Labs with a16z
- World Labs envisions a long-term "multiverse" where the real world functions as a scalable digital infrastructure, enabling people and developers to act across different spaces, with robotics serving as a core application of this spatial intelligence.
- The Cynics team plans to execute a "real to sim to real" pipeline that performs dense reconstruction of appearance, geometry, and dynamics to map real environments into a digital world, aiming to eventually replace real-world data and evaluation needs with scalable, generated digital data.
- The Marble base model is expected to convert image, few-shot image, or text prompts into geometrically consistent 3D worlds, serving as a foundation for efficient environment reconstruction and modeling.
- Future robotics foundation models are anticipated to incorporate "actions as inputs" to function as forward simulators predicting environmental changes, or "actions as outputs" to serve as policy models determining actions required to reach specific goals.
- Generated digital worlds are expected to provide consistency across space, time, viewpoints, and interactions, offering sufficient signals to solve object permanence issues and serve as a model-agnostic, embodiment-agnostic infrastructure for single arms, mobile manipulators, and grippers.
- Simulation is positioned to play a critical role in robotics by enabling counterfactual reasoning, systematic randomization of lighting, friction, geometry, and physical parameters, and speed-ups to "human speed" or faster to train on all dynamics efficiently.
- The technology is expected to allow clients to distinguish between performance checkpoints (e.g., 90% vs. 92%) faster and more safely in simulation, with high alignment ensuring that better simulated performance correlates with better real-world performance.
- World Labs expects to initiate a data flywheel where simulation-driven models feed into robot policy models that execute in the real environment to collect new data, shifting environmental modeling from pure physics toward learning-based approaches as data accumulates.
- Deployment strategy prioritizes semi-structured environments such as warehouses and hotels before advancing to fully unstructured settings like homes, though achieving human-level power efficiency and capabilities will require a long-term integration of hardware, software, and brain details.
- Unlike language models, robotic models are expected to require reliable out-of-the-box performance in real environments without human oversight, driving the need for scalable digital worlds to replace costly and unsafe real-world data collection.
- The business roadmap anticipates that within approximately two years, the combined Cynics and World Labs teams will validate customers in a small number of important vertical use cases, establishing "lighthouse examples" to scale the business.
- Current clients are nearing the deployment stage with practical tasks creating immediate value, with offerings ranging from the "real to sim" pipeline for digitalization and evaluation to the full "real to sim to real" pipeline for running policies on client hardware.
- Cynics plans to maintain a contained tech stack with a thoughtful integration approach, while opening an New York office to attract East Coast talent and remotely test engineering stacks, remaining open for business to robotics companies at any stage from early development to deployment.