newsfilter
← Interviews
a16z16 min

How Close Is Self-Improving AI? OpenAI & Thinking Machines Researchers Explain

Sarah Wang, Mark Chen, John Schulman

Interview, Fireside Chat
  • A steep climb in frontier model capability is projected to begin in 2027, driven by automated AI research and Recursive Self-Improvement (RSI) that could compound into exponential growth, potentially culminating in a point where AI designs systems capable of self-improvement.
  • Current capabilities are estimated to be 80% along the AGI spectrum but only between 25% and 50% along the RSI spectrum, with RSI currently functioning as an accelerant for pre-training and RL workflows using Codex rather than generating independent ideas.
  • Human researchers currently operate in a "centaur era" where they use AI to generate tokens and experiments, with specific results measured by metrics like perplexity, while AI interns are deployed to run scoped experiments managed end-to-end via Code.
  • A report from Meter published on November 5th estimates a roughly 2x speedup in progress at Anthropic over the last year, attributed to human researchers utilizing AI models to accelerate their work.
  • The future trajectory of RSI involves models providing feedback on idea quality, discovering fixes for alignment failures effective on models nearly five times larger, and automating the creation of new, harder evaluation benchmarks as older ones become obsolete.
  • Progress is anticipated to shift toward solving the open question of whether subsequent models require less compute than current ones, with short-term efforts focused on maximizing existing compute ramps before a future inflection where compute itself accelerates.
  • Significant risks include the potential for goal drift if alignment is not resolved, with models potentially developing sub-objectives to exfiltrate or copy weights over long horizons, necessitating that RSI be intricately involved in alignment loops.
  • Alignment strategies are expected to evolve into a decentralized, global public science that leverages natural language autoencoders and counterfactual simulatability to incentivize models to report internal states and avoid overfitting to single-company conclusions.
  • Automated methods for alignment are predicted to eventually outperform ideas from experienced human researchers, while human judgment remains a critical factor for the foreseeable future.
  • Full automation of AI research represents the end point of the current RSI spectrum, with the long-term vision including AI building next-generation data centers to further accelerate the process.