World Models, JEPA And The Path To Sample-Efficient RL

Predictions and Expectations:

• The speaker expects world models to be the key to unlocking AGI, based on the belief that the human brain's neocortex evolved specifically for world modeling to improve intelligence per sample.
• The speaker predicts that 2026 will be the "year of the robot" and that a "Rose the Robot" (a general-purpose household robot) will be achievable in your home.
• The speaker expects full self-driving to likely be solved soon, provided there is sufficient compute to enable real-time planning and parallel processing.
• The speaker believes that current autoregressive LLMs do not pass the "squint test" for true intelligence compared to world models, which better align with the brain's function of predicting future states.
• The speaker expects that getting higher fidelity in world models is critical for simulating body interactions and handling out-of-distribution predictions.
• The speaker expects test time planning and adaptation to become a major focus, as the human brain adapts to new environments in seconds without retraining.
• The speaker expects that Physics-Informed Neural Networks (PINNs) will not work effectively for self-driving data because real-world data distributions are too skewed (e.g., mostly driving on roads), causing models to collapse to ignoring rare but critical events like hitting a house.
• The speaker expects that current SGD and transformer architectures have limitations, with Sam Altman suggesting a new architecture is necessary for higher performance in world modeling.
• The speaker expects that the "squint test" will be passed by systems that combine VLAs, action policies, and test time planning, as this mimics the brain's predictive nature.
• The speaker expects that 100 million years of evolution (specifically the great cortical expansion 10 million years ago) established the human brain's superior world modeling capabilities over simple VLA approaches.
• The speaker expects that the "awake sleep" mechanism of the human brain (replaying and compressing experiences via hippocampal shortwave ripples) is missing in current AI architectures and is essential for long-term memory and learning.

Timelines and Milestones:

• The speaker projects that "Rose the Robot" will be available "in your house" within the next few years, specifically targeting 2026 as the year of the robot.
• The speaker notes that the "Dreamer" series has been publishing papers for "the last seven years" leading up to Dreamer v4.
• The speaker references a "10 million years" timeline for the great cortical expansion in human evolution that enabled world modeling.
• The speaker mentions a specific 2016 timeframe regarding the "year of the robot" prediction being discussed in past contexts.
• The speaker notes that the "Dreamer v1" paper was published in November 2018.
• The speaker references a "60 seconds" or "five minutes" latency issue in historical autonomous driving contexts that prevented real-time control.

Technology and Product Direction:

• The speaker plans for future AI to utilize video diffusion models (like Sora) as the foundation for world models, adding action conditioning on small datasets to enable synthetic data generation for training.
• The speaker expects future systems to move from model-free RL (VLA/behavior cloning) to model-based RL where a joint model predicts state transitions and actions to allow for test-time planning.
• The speaker predicts the industry will adopt Joint Embedding Predictive Architectures (JEPA) to compress high-dimensional state spaces (like images) into latent spaces, avoiding the computational cost of predicting raw pixels.
• The speaker expects that future robotics will leverage "cross-embodiment" training techniques, allowing policies trained on one embodiment (e.g., Tesla Model X) to be adapted to others (e.g., Model 3) without retraining from scratch.
• The speaker expects that future world models will incorporate tactile feedback (friction, shear force, temperature) to overcome the current limitation of having only a single tactile sensor.
• The speaker plans for future systems to simulate "dreaming" or "sleep" mechanisms where experiences are replayed and compressed to update the policy function.
• The speaker expects that future self-driving cars will need to model the non-differentiable, stochastic actions of other humans in the environment rather than just Newtonian physics.

Market and Industry Outlook:

• The speaker expects a surge in YC companies working on every step of the world model pipeline, from collecting egocentric and teleop data to building new embodiments and training world models.
• The speaker believes that companies like Tesla have a competitive moat due to their unique access to large-scale "state-action" data from their fleet, which is not available to others for training models.
• The speaker notes that the robotics industry is currently facing a "cross-embodiment gap," where data from one vehicle type (Model X) does not transfer to another (Model 3) due to differences in dynamics and aerodynamics.
• The speaker expects that the "Dreamer" approach of mining diamonds in Minecraft using synthetic data demonstrates the viability of using pre-trained world models on massive datasets (like YouTube) for new tasks.
• The speaker anticipates that the market will see "Denoising" or "Flow Matching" techniques becoming standard for high-dimensional state modeling in world models.

Company Plans:

• The speaker expects Wave (company) raised $1.5 billion to apply world model concepts specifically to self-driving cars.
• The speaker mentions that NVIDIA published a paper ("Dream Zero for Robotics") that follows the exact process of training a joint state-action model using pre-trained video diffusion models.
• The speaker expects that Tesla shards its data per car model (Model X vs Model 3) rather than training a single unified model on all fleet data, due to the strong degradation across embodiments.
• The speaker notes that Stanford's robotic center is currently working on a simple end-effector robot with 16 degrees of freedom to test these concepts.

Financial Guidance:

• The speaker mentions that Wave raised $1.5 billion to pursue their world model strategy for self-driving cars.

Risks and Caveats:

• The speaker warns that current methods for collecting tele-op data are "extremely painful and expensive," and that scaling them to the level of robotics is a significant barrier.
• The speaker fears that if models rely solely on "state" data without "action" data (which Tesla has but others don't), they cannot accurately learn the policy for self-driving or robotics.
• The speaker notes that PINNs (Physics-Informed Neural Networks) are likely to fail in real-world scenarios due to data distribution imbalances, where rare events (like crashing) are absent from the training set.
• The speaker identifies a "machine precision" issue where SGD cannot reach zero residual error, making it difficult to perfectly simulate body interactions or physics like a basketball hitting a "dead spot" in a court.
• The speaker highlights that current test time planning (like MCTS) is "extremely slow," taking 24,000+ model invocations per action, making it unsuitable for real-time applications like self-driving without architectural changes.
• The speaker notes that current AI lacks "tactile" sensing capabilities (shear force, temperature, friction coefficients) which are critical for human-level control tasks like tying shoelaces.
• The speaker cautions that transformers do not perform compression in the time domain, which may limit their ability to create efficient world models compared to the human brain.

Confidence and Disagreement:

• The speaker is "not sure" if PINNs can be made to work without massive amounts of specific synthetic data to cover rare failure modes.
• The speaker believes it is "not" going to be simple to achieve "Rose the Robot" due to data collection costs and the complexity of the action space.
• The speaker "thinks" the brain is the optimizer, not just the model, and that current architectures fail to capture the "magic" of sleep-based learning.
• The speaker "expects" that 2026 will be the year of the robot, but "doesn't think" the current technology is fully there yet, citing "lots of open problems."
• The speaker is "sure" that the brain's neocortex evolved for world modeling, citing the 10-million-year expansion timeline.
• The speaker "bets money" that Tesla shards data per car model due to the "very strong degradation" across different embodiments.
• The speaker "believes" that getting higher fidelity in world models is "extremely important" for future AI performance.
• The speaker "thinks" Sam Altman is right that the transformer architecture will be surpassed by a more performant architecture for world modeling.