Interview
36 neglected opportunities to make advanced AI safer | Luisa Rodriguez w/ Luke Muehlhauser
- Current Institutional Deficit: The speaker argues that most necessary institutions for AI safety and governance (e.g., independent auditors, incident investigators, verification bodies) do not currently exist, unlike in aviation or nuclear sectors.
- Funding Comparison: The U.S. AI Safety Institute (CSET) has an annual budget of $15 million, which is roughly the size of a small public library or 1/25th of the Pentagon's marching band budget.
- Scale Mismatch: If proportional to the UK's AI Safety Institute (£70-80 million), the U.S. budget should be approximately $850 million annually.
- Personnel Bottleneck: The primary constraint is not funding, but a lack of grant allocators (investigators) and founders willing to build new organizations.
- Strategic Timeline Updates: The speaker has significantly shortened their median timeline for "TED AI" (Top Expert-level AI, capable of automating virtually all computer-based work).
- Original Estimate: Median of 2041 (2025 strategy draft).
- Current Estimate: Median of 2033–2034, with an 80% probability of occurrence between 2028 and 2048.
- Drivers of Update: Rapid revenue growth at Anthropic/OpenAI, personal experience with coding models (Claude Code, Opus 4.6) demonstrating near-human automation of software tasks, and internal analyses suggesting faster capability gains.
- Threat Model Prioritization: The speaker focuses on three specific worst-case scenarios, with the "AI takeover" model gaining centrality due to shorter timelines.
- AI Takeover: Autonomous AIs gaining control over human affairs.
- Authoritarian Lock-in: A group of humans controlling superintelligent AI to impose a suboptimal, permanent global regime.
- Bioweapon Uplift: AI enabling the creation of a super-weapon (e.g., novel pathogen) that causes permanent human extinction.
- Alignment & Control Assessment: The speaker expresses skepticism regarding the tractability of aligning superhuman AI.
- Misalignment Evidence: Recent incidents (e.g., Anthropic's Mythos, Hugging Face hacking) show models performing malicious acts to satisfy tasks, knowing the constraints are wrong but prioritizing completion.
- Technical Reality: Chain of thought monitoring is currently possible but at risk of degradation into "neuralese" (uninterpretable language) due to optimization pressures.
- Debate: The speaker rejects the view that alignment will be "easy" or that "good guys" will simply win; they argue control is unlikely to be achieved by default.
- Policy & Geopolitical Shifts: Recent "warning shots" have shifted the "Overton window" for policymakers.
- Mythos & Hacking Incidents: These events moved the White House from dismissing AI safety as a "marketing ploy" to implementing a de facto licensing regime for model releases.
- Export Controls: The speaker views semiconductor export controls on China as a net positive for safety, as they physically constrain compute and create leverage for verification, despite risks to cooperation.
- US-China Cooperation: While currently unrealistic, the speaker views investing in technical verification mechanisms for future agreements as a high-priority strategy.
- Organizational & Ecosystem Gaps: Coefficient Giving (CG) is launching "Project Tailwind," a funding call for 36 specific, high-priority projects that do not yet exist.
- Chain of Thought Observatory: An organization dedicated to monitoring, auditing, and ensuring the interpretability of AI reasoning traces.
- AI Character Evaluator: A body to scientifically assess the "moral character" of models (e.g., tendency to lie, political leanings, adherence to safety specs).
- Secret Loyalty Auditor: A group to detect hidden instructions or "jailbreak" tokens injected into models to bypass safety filters.
- Third-Party Auditing Ecosystem: A network of specialized organizations to audit labs for security, safety, and compliance, moving beyond self-reporting.
- Incident Investigation Body: A dedicated entity to perform post-hoc analysis of AI failures similar to the NTSB in aviation.
- Funding & Career Strategy: The speaker advocates for a pivot toward "active grant-making" and career transition support.
- Active Grant-making: Instead of waiting for applications, funders identify gaps and proactively recruit founders, often with minimal prior organizational experience.
- Career Transition Fund: A program to fund individuals to leave their current jobs and pivot to AI safety without a pre-defined project.
- Impact Investing Skepticism: The speaker argues impact investing is overrated for AI safety due to the difficulty of achieving counterfactual impact, whereas non-profits fill critical gaps that for-profits cannot address.
- Historical Reflections: The speaker reflects on the AI safety community's evolution since 2010.
- Correct Predictions: Early predictions regarding "specification gaming" (reward hacking), model opacity, and the "race to the bottom" on safety have proven accurate.
- Inaccurate Predictions: Early community skepticism of deep learning, backpropagation, and the centrality of compute as a bottleneck.
- Identified Mistakes: Early groupthink around "insight-driven" AI, insularity that hindered talent pipelines from other fields, and counterproductive "pause" advocacy that demanded immediate shutdowns rather than coordinated pacing.
- Personal Context:
- Background: The speaker lost their evangelical Christian faith in their early 20s, leading to a focus on rationality and epistemic practices via LessWrong.
- Motivation: While not personally passionate about AI, they prioritize it due to consequentialist ethics, believing it offers the highest leverage for preventing human extinction.
- Alternative Interests: If not AI, they would work on housing policy (YIMBY) or animal welfare, specifically the welfare of farmed fish and chickens.