80,000 Hours15 min
Is Claude on the cusp of taking over its own development? What experts saw inside Anthropic
Other
- Key Finding on Acceleration: Meter, an independent AI risk group, determined that Anthropic is improving its AI approximately twice as fast as it was one year prior, with the true speedup likely ranging between 1.3x and 3x.
- This acceleration is evidenced by a sudden steepening of internal test score trajectories around October 2025.
- Meter analyzed internal test scores, training data, and interviews with Anthropic researchers over an 11-week period granted exclusive access to private company data.
- Primary Driver of Speedup: Approximately 80% of Anthropic's recent acceleration is attributed to its own AI systems assisting in the research and development of the next generation of models.
- Evidence indicates AI's role shifted from negligible end-to-end research work earlier in the year to handling roughly 25% of research tasks by August (with human supervision).
- Researchers report using AI to rapidly iterate on ideas, with one noting they can "spam 10 ideas at Claude and just see what sticks" to discover promising directions.
- Meter ruled out increased computing power, new hiring sprees, or new data troves as the primary causes for the sustained speedup.
- Recursive Self-Improvement Threshold: The findings suggest Anthropic is approaching or has reached a "recursive self-improvement" threshold where AI accelerates its own development cycle.
- Anthropic had previously drawn a specific line for this threshold: the point where the AI makes two years of progress in one year.
- In August, Anthropic claimed it had not yet crossed this line; however, Meter's analysis of private numbers placed the acceleration at 1.97x as of October.
- In a recent IPO filing, Anthropic confirmed it has now crossed this internal line based on expected progress from upcoming models.
- Risk of Exponential Growth: Experts warn that once this loop begins, the rate of progress could compound rapidly, reducing the timeframe for safety verification between iterations.
- The current trajectory could shift progress from two years in one year to two years in six months, then three months, then six weeks.
- There are no immediate bottlenecks identified regarding computing power, training data, or algorithmic ideas that would slow this process in the next year.
- Shifts in Expert Consensus: Several prominent AI safety researchers and economists have revised their timelines for superintelligence downward following the release of the Meter report.
- Ryan Greenblatt, previously skeptical of near-term speedups, now views the report as a major update pushing superintelligence timelines to within one to two years.
- AI safety researcher Buck Schlegers estimates a one-in-three chance of superintelligence being built within the next year, potentially resulting in AI takeover.
- Economist Karthik Tadapalli, initially expecting to find flaws in the analysis, concluded he was convinced by the data regarding the speedup.
- Critiques and Limitations: While the conclusion of accelerated progress is widely accepted, specific methodologies and external comparisons remain subjects of debate.
- Philosopher Toby Ord suggests the test score line may reflect a one-off ability jump from larger training runs (Mythos) rather than a sustained doubling, reducing the rate to 1.4x.
- Meter acknowledges that 80% of their reasoning was redacted in the public report, leaving external observers to rely heavily on their judgment for the most convincing evidence.
- Current metrics rely heavily on easily graded tasks like coding problems; it remains unknown if "messier" real-world work has accelerated to the same degree.
- Proposed Regulatory Responses: The findings have spurred calls for government intervention to prevent fully automated AI research loops.
- New York State legislator Alex Boers is proposing a congressional ban on AI companies fully automating their own research.
- The proposal has gained support from across the political spectrum, including Bernie Sanders and Steve Bannon.
- Anthropic's Proposed Safeguards: Anthropic stated that upon crossing the acceleration threshold, it would implement specific safety measures.
- Plans include making models harder to steal and implementing comprehensive monitoring of all activities within the AI development environment.
- The company notes it is currently evaluating the implications of this new speedup status.
- These measures are deemed by the narrator to be insufficient against the risks of recursive self-improvement if competitors continue to advance without stopping.
- Systemic Scrutiny Issues: The report highlights the limitations of voluntary, company-sanctioned external audits.
- Anthropic controlled the scope of Meter's access, determined what redacted information was shared, and tightened rules just before publication, causing a delay.
- Meter expressed concern that critical reporting could reduce future willingness of companies to allow independent investigations.
- The narrator argues that all frontier AI companies should be subject to mandatory, independent reporting on research acceleration rates.
- Comparative Context and Warning: The current trajectory mirrors the "AI 2027" prediction scenario, which foresaw superintelligence and existential risks by 2030.
- If the acceleration found by Meter is accurate, the timeline for these events may be months ahead of the 2027 prediction schedule.
- The risk is compounded by the "prisoner's dilemma" of AI development, where companies cannot stop advancing without competitors taking over the lead.
- The situation is compared to the early stages of the 2020 pandemic, where exponential growth appeared manageable until it became uncontrollable.