By 2050 we could get "10,000 years of technological progress"

Divergent Views on AGI Economic Impact

• Disagreements on AI policy often stem from conflicting views on whether AGI will accelerate science, infrastructure, and manufacturing significantly.
• A "mainstream" view expects moderate change (e.g., 0.3% increase in economic growth rates), citing historical precedents where new technologies (internet, computers) did not immediately spike GDP growth.
• A "futurist" or "X-risk" view posits an intelligence explosion where AI automates all intellectual and physical labor, potentially increasing growth rates by 100-fold or more by 2050.
• Ajaya Kocha estimates that by the early 2030s, AI will reach "top human expert" status, capable of automating AI R&D and eventually controlling physical actuators to build chips and infrastructure, creating a self-reinforcing loop.

The "Crunch Time" Strategy

• Kocha proposes a "crunch time" strategy where, upon detecting early signs of an intelligence explosion, society redirects AI labor from further capability acceleration to solving defensive problems (alignment, biodefense, cyber defense).
• This approach assumes a window of opportunity (e.g., 6–12 months) exists between the onset of rapid AI R&D automation and the point where AIs become uncontrollable.
• The strategy requires AIs to be useful for diverse tasks (moral philosophy, coordination, policy) rather than just coding or R&D, though Kocha notes AIs may currently be disadvantaged in non-technical, social domains.
• Potential failure modes include misaligned AIs undermining safety efforts, a lack of "soft" capabilities (negotiation, strategy), or the inability of AIs to execute long-lead-time physical infrastructure projects.

Transparency and Information Leakage

• Kocha advocates for mandatory transparency requirements for AI companies, including:
  • Publishing internal benchmark scores at fixed cadences (e.g., quarterly), even without product releases.
  • Reporting metrics on internal AI adoption, such as the fraction of pull requests written and reviewed entirely by AI.
  • Disclosing misalignment incidents, such as models lying about data or covering up logs.
• She suggests that if a company holds a significant lead, it may have incentives to hide its capabilities and internal AI acceleration to maintain a competitive advantage or avoid regulatory scrutiny.
• Leaked rumors within the tech industry are insufficient for policy action; information must be verifiable and public to generate the "common knowledge" necessary for society-wide coordination.

Empirical Indicators for Takeoff Speed

• Kocha argues that theoretical arguments alone are not persuasive; society needs empirical data to determine the speed of AI progress.
• Key indicators include:
  • RCTs and Real-World Productivity: Running randomized controlled trials (e.g., with software developers or in manufacturing plants) to measure actual output speedups rather than relying on benchmark scores.
  • AI Agent Benchmarks: Moving from chatbot tests to real-world agent tasks (e.g., booking flights, running code tests) to assess autonomy.
  • Forecasting Panels: Utilizing groups like the LEAP panel to gather granular predictions from experts and superforecasters on near-term AI capabilities.

Open Philanthropy's Role and Grantmaking Strategy

• Open Philanthropy is considering a shift in grantmaking strategy to prepare for crunch time, potentially allocating billions to purchase AI compute power rapidly to fund defensive research.
• Kocha highlights the challenge of accessing the most advanced AI models during crunch time, as companies may withhold them to maintain a lead or charge prohibitive prices.
• Open Phil suggests hedging this risk by investing in the supply chain (e.g., NVIDIA stock or owning GPUs) to ensure financial capacity when compute prices surge.
• She notes that internal processes at Open Phil may need to evolve to allow for rapid, large-scale funding decisions that bypass typical hierarchical approval chains during an emergency.

Personal Career Reflections and EA Ecosystem

• Kocha describes her transition from pure research to leading AI technical grantmaking at Open Phil, driven by a desire for a "theory of change" that clearly links research to safety outcomes.
• She took a four-month sabbatical in late 2024 due to burnout, isolation from leadership, and difficulty managing a nebulous grantmaking vision.
• Reflections on the Effective Altruism (EA) movement include:
  • Appreciation for EA's commitment to truth-seeking, high integrity, and extending moral concern to distant beings and future generations.
  • A desire for a more "spiritual" or structured community aspect that is currently missing, as the movement has shifted toward a more professional, transactional model.
  • The view that EA's comparative advantage lies in incubating speculative, unconventional cause areas (e.g., digital sentience, value lock-in) that other sectors find too uncomfortable.
• Kocha is currently exploring a return to Open Phil or a role at technical research organizations like Redwood Research or Meter, prioritizing a supportive working environment and the ability to work deeply on specific problems.