Sequoia Capital45 min
OpenAI's Noam Brown, Ilge Akkaya and Hunter Lightman on o1 and Teaching LLMs to Reason Better
Noam Brown, Ilge Akkaya, Hunter Lightman, Sonya Huang, Pat Grady
Interview, Fireside Chat, Product Demonstration
Core Concept of O1 (Project Strawberry):
- O1 represents OpenAI's first major implementation of "inference time compute," allowing models to spend extended time reasoning before generating a final response.
- The approach distinguishes between "System 1" (automatic, instinctive) and "System 2" (slow, process-driven) thinking, applying the latter to AI problems where thinking longer yields higher accuracy.
- Reasoning is defined as tasks where there is a significant "generator-verifier gap," meaning it is much harder to generate a correct solution than to recognize one once found (e.g., Sudoku vs. naming the capital of Bhutan).
- Unlike domain-specific reasoning seen in AlphaGo, O1's ability to extend thinking time is generalizable across various domains, including STEM, coding, and biology.
Development History and Research Trajectory:
- The path to O1 involved years of research with significant setbacks; initial methods often failed, but leadership maintained conviction in the direction of extended reasoning.
- Key team members (Noam, Hunter, Ilga) noted shifts in conviction:
- Noam: Had early conviction but noted the lack of a clear immediate path.
- Hunter: Experienced an "aha moment" when observing models successfully backtracking and self-correcting in chain-of-thought outputs.
- Ilga: Initially prioritized robotics/embodied AI for AGI; her conviction shifted after observing the data trends supporting the reasoning paradigm.
- The breakthrough involved observing emergent behaviors like "backtracking" (e.g., the model saying "wait, this is wrong, let me take a step back") rather than pre-programming specific reasoning rules.
- The project leverages Deep Reinforcement Learning (Deep RL) in combination with large-scale data training, moving the field out of the "trough of disillusionment" regarding Deep RL's utility outside of narrow game settings.
Capabilities and Performance Trends:
- STEM Dominance: O1 shows notable improvement in STEM domains because these problems often fit the "hard to generate, easy to verify" reasoning spectrum.
- IOI Competition: In the International Olympiad in Informatics, O1 solved problems humans struggled with, though it also failed on problems humans solved well, indicating a non-human approach to problem-solving (e.g., brute-force methods vs. human intuition).
- Scientific Research: Users report using O1 as a brainstorming partner in cancer research and gene discovery, though the model cannot execute the research itself.
- Human-Interpretable Chains of Thought: The model produces reasoning steps that are understandable to humans, allowing researchers to analyze its "aha moments" and logic flow.
- Interview Performance: O1 passed OpenAI's research engineer interview with high rates and has contributed code to the company's repository, acting as a "coding partner."
- Inference Scaling Laws: The team published data showing a clear trend line for performance improvements as inference time (thinking time) increases, suggesting a higher ceiling than traditional pre-training scaling alone.
Strategic Decisions and Philosophy:
- Hidden Chain of Thought: The decision to hide the internal reasoning process was driven by competitive concerns and safety risks, similar to withholding model weights, rather than internal debate.
- AGI Goal: The ultimate objective is Artificial General Intelligence (AGI), defined operationally as the ability to perform a significant proportion of economically valuable jobs.
- Gap Management: There is a recognized gap between raw model intelligence and practical utility; the ecosystem of developers is expected to bridge this by building tools on top of the API.
- O1 vs. O1 Mini:
- O1: Slower, heavier, designed for complex reasoning tasks.
- O1 Mini: Faster, cheaper, designed for rapid iteration and universal utility; preserves reasoning capabilities for broader access.
- Usage Guidance: Developers are encouraged to experiment with O1 for non-STEM tasks (e.g., writing, strategy) to discover new use cases, as the model's utility extends beyond just math and code.
Limitations and Future Outlook:
- Performance Gaps: O1 is not universally superior; it underperforms in creative writing and tasks requiring high-speed interaction compared to previous models like GPT-4.
- Scaling Limits: It is unclear how far inference time can be scaled; while infinite compute theoretically allows solving any formalizable math problem via exhaustive search, diminishing returns and practical constraints exist.
- Future Models: The team plans to iterate on the current findings, using user feedback to extend the trend line to new domains and potentially improve non-STEM capabilities in O2 or O3.
- Misconceptions: The codename "Strawberry" was a random coincidence (someone eating strawberries) and not related to the popular "count the Rs" prompt trick; the "Q-Stack" or "thinking mode" names were rejected for branding reasons.
Notable User Interactions and Feedback:
- Users have successfully used O1 to solve mathematical proofs that were previously unsolvable by AI, suggesting a potential breakthrough in novel math research assistance.
- The model has demonstrated "surprising" geometry solutions where it visualizes problems verbally, mimicking human cognitive strategies.
- Some users have found "funky edges" where specific prompting strategies significantly improve performance, indicating the interface is still being optimized.