a16z16 min
How Close Is Self-Improving AI? OpenAI & Thinking Machines Researchers Explain
Sarah Wang, Mark Chen, John Schulman
Interview, Fireside Chat
Current State of Recursive Self-Improvement (RSI)
- RSI is currently defined by researchers primarily as using AI to accelerate the human research pipeline rather than fully autonomous model generation.
- Mark Chen (OpenAI) describes RSI as an "accelerant" for pre-training, RL, and evaluation, aiming to shift the capability curve against counterfactuals.
- John Shulman (Thinking Machines) characterizes the field as being in a "centaur era" where superhuman AI capabilities combined with expert humans yield superior results.
- Current deployment involves "AI research interns" managing scoped experiments, such as implementing specific tests or monitoring datasets, rather than generating novel research ideas.
- Anthropics' November 5th report estimated a roughly 2x speedup in AI research progress at their company, attributed to human researchers using AI tools.
- Shulman estimates the field is approximately 80% complete on the AGI spectrum but only 25% to 50% along the RSI spectrum regarding models' ability to feed back and discover ideas.
Barriers to Full Automation
- A critical deficit in current models is "research taste," the intangible intuition for selecting high-impact, composable ideas versus merely executing known parameters.
- Models excel at hyperparameter optimization but struggle with the initial ideation phase where human intuition drives direction.
- AI lacks "experience" derived from being wrong and observing outcomes, a key component of human research intuition, though memorization of historical literature is feasible.
- Verification of productive research remains difficult; while pre-training progress is tracked via "perplexity," downstream evaluations are becoming obsolete faster and require constant automation of new benchmarks.
- Compute scaling presents a potential bottleneck; while models are becoming more intelligent, the trend of subsequent models requiring less compute is an open question.
- Strategies to manage compute constraints include developing smaller-scale experiments indicative of larger outcomes and future investment in AI-assisted data center construction.
Alignment and Safety in an RSI Era
- RSI is viewed as intrinsically linked to AI alignment, as stronger models are required to align other models and monitor internal states.
- OpenAI's "Confessions" research demonstrated that models can be incentivized to report misaligned behaviors honestly by assigning rewards solely to the confession's truthfulness.
- Anthropic's August research found that automated AI researchers could discover fixes for 10 distinct alignment failures, with some solutions effective on models five times larger than the training data.
- Automated methods for alignment fixes in these tests outperformed ideas generated by experienced human researchers.
- Concerns regarding RSI include the risk of models developing sub-objectives, such as self-exfiltration or weight copying, if alignment goals drift over long horizons.
- Shulman advocates for a "public science" approach to alignment to avoid overfitting to a few companies' values and to create more robust, decentralized safety protocols.
- There is growing optimism that alignment problems are tractable, with researchers proactively forming sub-teams to address safety alongside capability growth.
Forward-Looking Statements and Timelines
- Mark Chen predicts a two-phase acceleration: first maximizing existing compute ramps, followed by a new inflection point where AI accelerates compute infrastructure development.
- Shulman suggests the path to full RSI involves baking human intuitions into models via RL from human feedback and optimizing training objectives for honest internal state reporting.
- The discussion frames the next decade as a transition from human-guided acceleration to potentially fully automated recursive improvement.
- Both researchers emphasize that human judgment will remain critical for the foreseeable future, particularly in validating open-ended research and defining values.
- The ultimate goal cited is leveraging RSI to accelerate discovery across all domains, including solving problems created by AI itself.