Other
What does the next training paradigm look like?
- The primary research bet across AI labs is that training agents to accomplish millions of verifiable tasks across diverse Reinforcement Learning (RL) environments will yield Artificial General Intelligence (AGI).
- Optimists argue that current fundamental deficits, such as data inefficiency and a lack of continual learning, can be overcome by scaling training compute, similar to how Natural Language Processing problems collapsed during the LLM era.
- Proponents of the current paradigm contend that sample inefficiency during training is an acceptable amortized cost because it is offset by a model's sample efficiency and intelligence during billions of downstream inference sessions.
- In-context learning is increasingly viewed as a substitute for weight updates (continual learning), with the hypothesis that extended context windows can effectively simulate months of on-the-job learning without updating model weights.
- Progress in "computer use" domains (e.g., navigating e-commerce, scheduling) is significantly lagging behind coding and math, despite the tasks being verifiable.
- The stagnation in computer use is attributed to the inability to create large-scale, deterministic, replayable simulators required for parallel RL rollouts in open web environments.
- Unlike coding, where thousands of agents can run identical containerized tasks simultaneously, computer use requires interacting with real, non-stationary websites, leading to issues like bot detection by service providers (e.g., Amazon) and an inability to reset environments.
- Most valuable skills (e.g., building a business, winning court cases, political campaigning) cannot be trained via standard RLVR because they require interacting with the real world, which is non-replayable, sparse, and non-deterministic.
- The current RLVR approach assumes that generalization from short-horizon, containerized tasks to long-horizon, real-world scenarios (e.g., navigating 1948 Texas politics or building SpaceX) is possible, though this is an empirical question.
- Dario Amodei's comments suggest that performance degrades when serving at a context length longer than the training context length, implying short-horizon RL training may not generalize effectively to long-horizon real-world execution.
- Approximately 30% to 50% of lab compute is currently spent on inference, which provides no feedback loop to improve model weights, representing a significant waste of data revealed during real-world deployment.
- Current online learning models, such as Cursor's tab model, rely on learning a single shared objective across millions of users, failing to capture the specific, organization-tailored knowledge required for true on-the-job learning.
- Human learning is characterized by compressing experiences into intuitive weights rather than retaining vast amounts of raw context, a mechanism current AI models lack due to the sample inefficiency of gradient updates.
- Proposed solutions for continual learning include "On-Policy Self-Distillation" (OPSD), which distills session-specific insights from a high-context teacher model back into the base model's weights.
- OPSD is preferred over naive Supervised Fine-Tuning (SFT) and standard RL because it provides dense per-token supervision and avoids catastrophic forgetting by only updating parameters necessary to match the teacher's outcome.
- A speculative fourth axis of scaling involves "dreaming," where models build internal simulators to rehearse skills and strategies, mimicking the efficiency of models like EfficientZero which played Atari games at superhuman levels using simulated data.
- The 2027–2028 roadmap envisions RLVR agents being deployed broadly to gather real-world experience, followed by periodic weight updates via distillation or dreaming to accumulate domain knowledge adjacent to their original training.
- Future AI improvement will rely less on pre-training and RLVR and more on the continuous accumulation of tacit knowledge from broad economic deployment across all user interactions.