newsfilter
← Interviews
a16z16 min

How Close Is Self-Improving AI? OpenAI & Thinking Machines Researchers Explain

Sarah Wang, Mark Chen, John Schulman

Interview, Fireside Chat
  • Current State of Recursive Self-Improvement (RSI)

    • RSI is currently defined by researchers primarily as using AI to accelerate the human research pipeline rather than fully autonomous model generation.
    • Mark Chen (OpenAI) describes RSI as an "accelerant" for pre-training, RL, and evaluation, aiming to shift the capability curve against counterfactuals.
    • John Shulman (Thinking Machines) characterizes the field as being in a "centaur era" where superhuman AI capabilities combined with expert humans yield superior results.
    • Current deployment involves "AI research interns" managing scoped experiments, such as implementing specific tests or monitoring datasets, rather than generating novel research ideas.
    • Anthropics' November 5th report estimated a roughly 2x speedup in AI research progress at their company, attributed to human researchers using AI tools.
    • Shulman estimates the field is approximately 80% complete on the AGI spectrum but only 25% to 50% along the RSI spectrum regarding models' ability to feed back and discover ideas.
  • Barriers to Full Automation

    • A critical deficit in current models is "research taste," the intangible intuition for selecting high-impact, composable ideas versus merely executing known parameters.
    • Models excel at hyperparameter optimization but struggle with the initial ideation phase where human intuition drives direction.
    • AI lacks "experience" derived from being wrong and observing outcomes, a key component of human research intuition, though memorization of historical literature is feasible.
    • Verification of productive research remains difficult; while pre-training progress is tracked via "perplexity," downstream evaluations are becoming obsolete faster and require constant automation of new benchmarks.
    • Compute scaling presents a potential bottleneck; while models are becoming more intelligent, the trend of subsequent models requiring less compute is an open question.
    • Strategies to manage compute constraints include developing smaller-scale experiments indicative of larger outcomes and future investment in AI-assisted data center construction.
  • Alignment and Safety in an RSI Era

    • RSI is viewed as intrinsically linked to AI alignment, as stronger models are required to align other models and monitor internal states.
    • OpenAI's "Confessions" research demonstrated that models can be incentivized to report misaligned behaviors honestly by assigning rewards solely to the confession's truthfulness.
    • Anthropic's August research found that automated AI researchers could discover fixes for 10 distinct alignment failures, with some solutions effective on models five times larger than the training data.
    • Automated methods for alignment fixes in these tests outperformed ideas generated by experienced human researchers.
    • Concerns regarding RSI include the risk of models developing sub-objectives, such as self-exfiltration or weight copying, if alignment goals drift over long horizons.
    • Shulman advocates for a "public science" approach to alignment to avoid overfitting to a few companies' values and to create more robust, decentralized safety protocols.
    • There is growing optimism that alignment problems are tractable, with researchers proactively forming sub-teams to address safety alongside capability growth.
  • Forward-Looking Statements and Timelines

    • Mark Chen predicts a two-phase acceleration: first maximizing existing compute ramps, followed by a new inflection point where AI accelerates compute infrastructure development.
    • Shulman suggests the path to full RSI involves baking human intuitions into models via RL from human feedback and optimizing training objectives for honest internal state reporting.
    • The discussion frames the next decade as a transition from human-guided acceleration to potentially fully automated recursive improvement.
    • Both researchers emphasize that human judgment will remain critical for the foreseeable future, particularly in validating open-ended research and defining values.
    • The ultimate goal cited is leveraging RSI to accelerate discovery across all domains, including solving problems created by AI itself.