Interview
John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI
- Projected Capabilities (1–2 Years):
- Models are expected to execute entire coding projects, moving beyond single-step suggestions to writing, testing, and iterating across multiple files based on high-level instructions.
- Long-horizon training (e.g., reinforcement learning for multi-step tasks) is identified as the primary unlock for extended coherent operation, enabling models to function as collaborators rather than search engines.
- Improved error recovery and sample efficiency will allow models to handle edge cases with minimal data, generalizing from pre-training examples of "getting back on track."
- Projected Capabilities (5 Years):
- Models will handle increasingly complex tasks, scaling from five minutes of coherence to tasks requiring hours or days of continuous operation.
- While compute costs for long-horizon tasks will rise, progress may follow phase transitions rather than clean scaling laws, as the same "mental machinery" may apply across different time scales (e.g., one month vs. one year).
- Technical Architecture:
- Pre-training: Optimized to imitate internet content (text, code) and maximize likelihood, resulting in well-calibrated models capable of generating diverse content and assuming various personas.
- Post-training: Targets a narrower, specific persona (helpful assistant) by optimizing for human preference and utility rather than raw content imitation; responsible for the bulk of GPT-4's performance gains over GPT-3.5.
- Generalization Evidence:
- Fine-tuning on English data automatically confers competence in other languages (e.g., Spanish) without explicit training.
- Minimal data (e.g., ~30 examples) regarding a model's own limitations can generalize to a broad range of capabilities the model was never explicitly trained on.
- Memory and Learning: Future systems will likely combine long-context windows with fine-tuning and "active learning," where models introspect on their knowledge gaps to seek information rather than relying solely on static pre-training snapshots.
- AGI and Safety Protocols:
- Timeline: AGI could emerge within 2–3 years if long-horizon RL training unlocks human-level coherence, though other bottlenecks (e.g., handling ambiguity, expert taste) may persist.
- Deployment Strategy:
- If AGI arrives unexpectedly early, OpenAI would pause further training and deployment to assess safety.
- Coordination among major AI entities would be required to prevent a "race to the bottom" on safety standards.
- A preferred pathway involves incremental capability increases with corresponding alignment improvements, allowing for continuous monitoring and detection of "scary" behaviors before broad release.
- Safety Mechanisms: Relies on "defense in depth," including red-teaming, simulated deployments, robust monitoring systems to detect immediate misbehavior, and ensuring models lack instrumental incentives for world domination in specific task contexts.
- Economic and Societal Integration:
- AI as Agents: The trajectory moves toward AI partners that proactively manage projects, suggesting next steps, and acting in the background, shifting the user experience from query-response to collaborative workflow.
- Business Automation: While models may soon be capable of running firms, there is a strategic preference for maintaining human oversight on critical decisions to mitigate tail risks and liability.
- Regulation: If fully autonomous AI firms become significantly more efficient, human-in-the-loop requirements may necessitate international regulatory coordination to prevent market disadvantages.
- UI Evolution: Websites may not require total redesign; vision capabilities will allow AI to interact with standard human-centric UIs, though optimized text-based representations and interaction indicators will likely be adopted for AI efficiency.
- Research and Data Dynamics:
- Compute Allocation: The ratio of compute spent on post-training versus pre-training is expected to increase as models become more capable of "thinking for themselves" rather than merely imitating web data.
- Data Walls: There is no immediate expectation of hitting a hard data wall, though the nature of pre-training will evolve; larger models demonstrate better transfer learning across modalities (e.g., code to reasoning) than smaller models.
- Replicability: The ML field remains relatively healthy due to practical grounding, though there is a noted need for more fundamental scientific inquiry beyond benchmark hill-climbing.
- Data Collection: Post-training quality depends on the skill and diversity of human raters, who vary by region and domain expertise (e.g., STEM tasks vs. creative writing).
- Stakeholder Alignment:
- OpenAI utilizes a "Model Spec" to navigate conflicting demands between end-users, developers, the platform, and humanity at large.
- The guiding heuristic is to maximize helpfulness to the user while intervening to prevent harm to others, avoiding excessive paternalism while blocking clearly detrimental usage.
- Future alignment will likely rely on distilling complex, subtle human preferences from large datasets of pairwise comparisons rather than explicit instruction manuals.