Why Robotics Still Isn't Solved - But Could Be Soon | YC Paper Club

Current State of Robotics and Key Challenges

• The speaker notes that the "next year robotics will be solved" prediction has recurred for a decade (citing AlphaGo, Mujoco, and Aloha) without mass deployment, with 2026 currently passing as a promised milestone without commercial availability of robots like "Neo 1x," "Pi," or "Figure."
• Current success is largely limited to "demos" and "work cells," with no mass-market robot services (e.g., "Rosie") available.
• Tele-operations data collection remains "remarkably hard" and "finicky," creating a bottleneck for scaling up robot policies.
• Four primary technical barriers prevent scaling robotics today:
  • Physical Real-World Modeling: Sim-to-real gaps persist; world models (e.g., Dreamer V1-4) often fail to respect physics, causing simulated failures (e.g., driving into a store resulting in "highway" physics) that do not translate to reality.
  • Deformable Objects: Estimating transition functions ($s_t$ to $s_{t+1}$) conditioned on actions is unsolved due to the lack of action representation; it requires massive data that is difficult to collect.
  • Sensory-Motor Issues: Robots lack the "epidermis" (multimodal sensory feedback) humans possess, relying on single FT sensors and wrist cameras instead of detecting normal force, moisture, temperature, or friction.
  • Embodiment Drift: Hardware degradation (dust, corrosion, battery variance) alters actuator behavior over time, causing stale mappings in Vision-Language-Action (VLA) models and necessitating retraining or data collection.

Presentation: MEM (Multi-Scale Embodied Memory)

• Goal: Enable robots to execute long-horizon tasks (e.g., full bedroom cleaning, cooking full meals) that exceed the current two-minute limit of standard policies.
• Problem Identified: Standard policies (e.g., Pi, Groot) lack memory, leading to infinite loops (e.g., washing dishes forever) or errors (e.g., burning food) due to an inability to track time or task progress.
• Proposed Solution: Decompose policies and memory into two levels to balance effectiveness and efficiency:
  • Short-Term Memory (Low-Level Policy): Uses a temporal Vision Transformer (ViT) encoder to compress dense visual frames into a low-context representation for dexterous manipulation; achieves fast inference.
  • Long-Term Memory (High-Level Policy): Utilizes a recurrent text-based "scratchpad" to summarize past events, offering a compressed, debuggable representation that reduces token load compared to raw images.
• Key Outcome: The system demonstrates "in-context adaptation," allowing robots to identify and correct mistakes (e.g., switching sides to open a fridge or adjusting grip on a chopstick) rather than repeating failures.
• Implementation Detail: High-level policies were trained via Supervised Fine-Tuning (SFT) where annotators explicitly defined relevant context; future work proposes Reinforcement Learning (RL) to auto-determine which information to retain.

Presentation: Refine and Bootstrap Embodiment-Specific Chain of Thought (R&B Encore)

• Objective: Solve the "chicken-and-egg" problem of embodied reasoning by bootstrapping the source of reasoning data and the model that consumes it without human annotation oracles.
• Methodology: A self-supervised cycle using variational inference:
  • Reasoning Proposer: Generates diverse reasoning traces (e.g., planning, perceptual lists, gripper position) for given contexts.
  • Reasoning Validator: Scores traces based on three metrics: concision, non-triviality, and action predictiveness.
• Key Findings:
  • Selective Reasoning is Critical: Exhaustive reasoning (e.g., analyzing every object) is often distracting; the method successfully pruned "perceptual reasoning" for manipulation tasks in favor of "move/gripper" reasoning.
  • Embodiment Specificity: Different form factors require different reasoning (e.g., legged locomotion needs structural affordances, while self-driving needs visible objects and meta-actions).
  • Generalization: The approach improved out-of-distribution performance across manipulation, navigation, and driving, working on models ranging from 1B to 30B parameters.
• Latency Management: The team employs "action forcing" to drop reasoning traces during inference, mitigating latency issues while retaining the benefits of training on textual reasoning data.

Presentation: SimTool Reel & Play to Perfect

• Core Achievement: A single, frozen policy controls both a 22-DOF hand and 7-DOF arm to perform dexterous tool use (e.g., brushing, hammering, screwing) zero-shot on 12 unseen tools.
• Methodology:
  • Unified Objective: Treats all tool tasks as "goal reaching," moving objects from a current pose to a desired goal pose, eliminating the need for task-specific labels or reward redesign.
  • Training: Uses massive parallel Reinforcement Learning (RL) in simulation with procedurally generated primitive objects (cubes/cylinders) and random goals.
  • Inference: Goal sequences are provided via human video demonstrations processed by Foundation Pose and SAM, acting as a "task prompt" without retraining.
• Performance:
  • Achieves 60Hz control and handles "in-hand reorientation" which fixed-grasp baselines fail at.
  • Demonstrates strong recovery behaviors (e.g., re-grasping dropped tools) due to domain randomization where random forces are applied during simulation.
• Follow-up (Play to Perfect): Extends the dexterous prior to contact-rich assembly tasks by first training on "task-agnostic play" before fine-tuning on sparse-reward assembly.
• Failure Modes: ~60% of failures stem from pose tracking (occlusion of markers); other failures include dropped objects or excessive arm twisting (limiting of the LSTM policy).

Presentation: The "Robotics Application Company" Strategy

• New Category: "Robotics Application Companies" (or "neo-integrators") are emerging as the most viable business model, focusing on end-to-end ownership of specific physical problems rather than generalist foundation models.
• Operational Pattern:
  • Start with Teleop: Deploy off-the-shelf hardware and use teleoperation to validate the business case before building custom hardware or foundation models.
  • Scrub the Problem: Focus on specific, high-value domains (e.g., data center management, warehouse logistics, small-scale manufacturing, food prep) where labor is scarce or expensive.
  • Data-First Approach: Build proprietary data layers and evaluation metrics from day one; prioritize collecting data that makes the model better, often involving manual annotation and "bad plane" filtering.
• Infrastructure Needs: Standard web data infrastructure (Postgres) fails for robotics; physical data requires multi-modal, multi-rate, episodic storage and querying capabilities (e.g., Rerun).
• Market Thesis: This model avoids the need for billion-dollar seed raises, focusing on pragmatic iteration, fast hardware replacement, and operational support to build a "moat" that pure model companies cannot replicate.

Presentation: General Instinct & World Action Models (WAM)

• Context: World Action Models (WAMs) (e.g., NVIDIA's Dream Zero) predict future frames and kinematics to improve action generation but are currently too slow (requiring 2 GB200s per model) for economic scalability.
• Optimization Strategy: General Instinct builds infrastructure to accelerate WAM inference:
  • Distillation: Compressed VAE encoding, smaller Diffusion Transformers (DIT), and reducing flow matching steps from 50-100 to 1-2 steps.
  • Architecture Shift: Moves from "generative" (decoding full frames) to "latent" WAMs (ImageRAM/Fast1) where the model predicts future latent states without decoding pixels, using cross-attention between video and action transformers.
  • Result: Achieves inference speeds of 60Hz (500ms per chunk) on JSON stores, making real-time deployment feasible.
• Technical Insight: Future kinematics prediction acts as physics supervision; while VLA models predict actions from current states, WAMs learn the correlation between pixel-level physics and action.
• Business Focus: Providing the "VLLM" equivalent for physical AI, focusing on edge deployment infrastructure for real-time, scalable WAMs rather than end-to-end application integration.