The inside story of how ChatGPT was built – OpenAI cofounder John Schulman

Origins and Strategic Shift

• OpenAI initially focused on "instruction following" base models designed to be easier to prompt than raw autocomplete systems, aiming to answer questions directly rather than continuing text.
• Competitors like Google explored chatbots (e.g., LaMDA, Mina) that specialized in conversational personas and fun applications rather than functional tasks like code refactoring.
• Internal projects like WebGPT established the necessity of conversational flow for question-answering tasks, as follow-up questions and clarifying queries are essential for ambiguous topics.
• The decision to build a dedicated conversational assistant was made to leverage the capabilities of the GPT-3.5 model, which showed strong proficiency in both language and coding.
• While the initial prototype included web browsing features, this functionality was later de-emphasized because the model's internal knowledge base proved sufficiently robust for most use cases.

Development and GPT-4 Integration

• ChatGPT development began in early 2022 based on GPT-3.5, with a specific focus on improving coding assistance capabilities.
• By August of that year, training on GPT-4 concluded, prompting a strategic pivot where the instruction-following model stack became the primary production deployment.
• Initial fine-tuned versions of GPT-4 demonstrated high quality but suffered from significant unreliability, including frequent hallucinations and "unhinged" outputs, rendering them unready for public release.
• The team resolved these issues by combining instruction-following and chat-specific datasets to create a hybrid model that balanced the strengths of both approaches.

Rationale for the Chat Architecture

• The "chat" format provided a clearer objective for both the model and human data labelers compared to the vaguely defined "helpful text completion" task of instruction models.
• Defining "what a helpful robot should be" offered an intuitive framework that resulted in more coherent personality traits and consistent, sensible behavior.
• The final architecture ensured the model better understood its own limitations, reducing the risk of presenting unverified information as fact.
• Public release beta testing with friends and family served as a critical validation step before the full launch.

Replicability and Technical Constraints

• Recreating ChatGPT using only public fine-tuning APIs would be non-trivial; a simple single-iteration fine-tuning on human-written data is insufficient for alignment.
• Successful replication would likely require multiple iterations of supervised fine-tuning where humans edit model-generated outputs to bridge the gap between human capability and model output.
• This iterative process functions similarly to Reinforcement Learning (RL), a step that cannot be bypassed if aiming for high-quality alignment without RL.
• Alternative approaches using an RL-trained instruction model wrapped in a chat interface could yield decent results but would lack specific strengths, such as the ability to recognize factual limitations, which are inherent to the ChatGPT training data.
• Pre-existing instruction models possessed different strengths, such as superior performance in creative writing and poetry, but lacked the robust factuality and limitation-awareness of the chat-optimized models.