Interview
Training General Robots for Any Task: Physical Intelligence’s Karol Hausman and Tobi Springenberg
- Physical Intelligence anticipates its robotic foundation models will soon enable any robot to perform any task, with PyStar 0.6 serving as a demonstration of performance sufficient for real-world deployment, a milestone reached approximately two months ago rather than the previously estimated five-year timeline.
- The company plans to solve the intelligence bottleneck by focusing on general-purpose models capable of controlling diverse robotic form factors and generalizing to new environments without specific retraining, rather than developing specific vertical robot products.
- Generalization is expected to be achieved through diverse data collections, where the shift from a bootstrap phase to a deployment phase will generate data at negative cost as robots perform economically valuable tasks, expanding the range of deployable applications over time.
- While the technology has reached a deployable threshold, Toby notes that significant performance improvements ("hill climbing") are still required for many applications to meet day-to-day business standards, and the aperture of deployable tasks is projected to grow as models mature and accumulate data.
- The current transformer architecture based on VLMs is expected to evolve, with the backbone model potentially changing in five to six years, driven by advances in LLM reasoning and the cross-pollination of capabilities as models scale.
- Future strategies involve relying on autonomous data collection to build a convex hull of tasks for pre-training, with models expected to learn from experience as an initial step toward continual learning.
- Technical improvements include the value function predicting failures 30 to 40 steps in advance, as well as world models and autoregressive approaches targeting counterfactual and credit assignment problems similar to reinforcement learning.
- Toby predicts that general physical intelligence may be no harder than specialized problems like self-driving, citing unexpected generalization capabilities, and warns against pre-baking rules like Newtonian physics into model weights as a limiting factor.
- The speakers foresee robotics providing a new avenue for abstract reasoning in images and trajectories that will benefit the broader LLM world, while anticipating that the intelligence bottleneck must be addressed before widespread deployment in homes and industrial settings occurs.
- Generalization remains an "open challenge," and the company identifies the ability to operate in completely new settings similar to training data as a critical expectation for future success.