OpenAI's Noam Brown, Ilge Akkaya and Hunter Lightman on o1 and Teaching LLMs to Reason Better

Core Concept of O1 (Project Strawberry):

• O1 represents OpenAI's first major implementation of "inference time compute," allowing models to spend extended time reasoning before generating a final response.
• The approach distinguishes between "System 1" (automatic, instinctive) and "System 2" (slow, process-driven) thinking, applying the latter to AI problems where thinking longer yields higher accuracy.
• Reasoning is defined as tasks where there is a significant "generator-verifier gap," meaning it is much harder to generate a correct solution than to recognize one once found (e.g., Sudoku vs. naming the capital of Bhutan).
• Unlike domain-specific reasoning seen in AlphaGo, O1's ability to extend thinking time is generalizable across various domains, including STEM, coding, and biology.

Development History and Research Trajectory:

• The path to O1 involved years of research with significant setbacks; initial methods often failed, but leadership maintained conviction in the direction of extended reasoning.
• Key team members (Noam, Hunter, Ilga) noted shifts in conviction:
  • Noam: Had early conviction but noted the lack of a clear immediate path.
  • Hunter: Experienced an "aha moment" when observing models successfully backtracking and self-correcting in chain-of-thought outputs.
  • Ilga: Initially prioritized robotics/embodied AI for AGI; her conviction shifted after observing the data trends supporting the reasoning paradigm.
• The breakthrough involved observing emergent behaviors like "backtracking" (e.g., the model saying "wait, this is wrong, let me take a step back") rather than pre-programming specific reasoning rules.
• The project leverages Deep Reinforcement Learning (Deep RL) in combination with large-scale data training, moving the field out of the "trough of disillusionment" regarding Deep RL's utility outside of narrow game settings.

Capabilities and Performance Trends:

• STEM Dominance: O1 shows notable improvement in STEM domains because these problems often fit the "hard to generate, easy to verify" reasoning spectrum.
• IOI Competition: In the International Olympiad in Informatics, O1 solved problems humans struggled with, though it also failed on problems humans solved well, indicating a non-human approach to problem-solving (e.g., brute-force methods vs. human intuition).
• Scientific Research: Users report using O1 as a brainstorming partner in cancer research and gene discovery, though the model cannot execute the research itself.
• Human-Interpretable Chains of Thought: The model produces reasoning steps that are understandable to humans, allowing researchers to analyze its "aha moments" and logic flow.
• Interview Performance: O1 passed OpenAI's research engineer interview with high rates and has contributed code to the company's repository, acting as a "coding partner."
• Inference Scaling Laws: The team published data showing a clear trend line for performance improvements as inference time (thinking time) increases, suggesting a higher ceiling than traditional pre-training scaling alone.

Strategic Decisions and Philosophy:

• Hidden Chain of Thought: The decision to hide the internal reasoning process was driven by competitive concerns and safety risks, similar to withholding model weights, rather than internal debate.
• AGI Goal: The ultimate objective is Artificial General Intelligence (AGI), defined operationally as the ability to perform a significant proportion of economically valuable jobs.
• Gap Management: There is a recognized gap between raw model intelligence and practical utility; the ecosystem of developers is expected to bridge this by building tools on top of the API.
• O1 vs. O1 Mini:
  • O1: Slower, heavier, designed for complex reasoning tasks.
  • O1 Mini: Faster, cheaper, designed for rapid iteration and universal utility; preserves reasoning capabilities for broader access.
• Usage Guidance: Developers are encouraged to experiment with O1 for non-STEM tasks (e.g., writing, strategy) to discover new use cases, as the model's utility extends beyond just math and code.

Limitations and Future Outlook:

• Performance Gaps: O1 is not universally superior; it underperforms in creative writing and tasks requiring high-speed interaction compared to previous models like GPT-4.
• Scaling Limits: It is unclear how far inference time can be scaled; while infinite compute theoretically allows solving any formalizable math problem via exhaustive search, diminishing returns and practical constraints exist.
• Future Models: The team plans to iterate on the current findings, using user feedback to extend the trend line to new domains and potentially improve non-STEM capabilities in O2 or O3.
• Misconceptions: The codename "Strawberry" was a random coincidence (someone eating strawberries) and not related to the popular "count the Rs" prompt trick; the "Q-Stack" or "thinking mode" names were rejected for branding reasons.

Notable User Interactions and Feedback:

• Users have successfully used O1 to solve mathematical proofs that were previously unsolvable by AI, suggesting a potential breakthrough in novel math research assistance.
• The model has demonstrated "surprising" geometry solutions where it visualizes problems verbally, mimicking human cognitive strategies.
• Some users have found "funky edges" where specific prompting strategies significantly improve performance, indicating the interface is still being optimized.