Being a pain in the ass to AI companies is a real job now | Rob Wiblin

Core Thesis: Holding AI companies accountable requires immediate action from staff, journalists, experts, and advocates due to the rapid pace of development outstripping government legislative timelines.

• Current efforts are deemed effective enough to deter misconduct because companies fear reputational damage and talent loss.
• There is an urgent need to build capacity to apply "fierce public and expert outrage" as a check on dangerous AI practices.

Anthropic's Legislative Proposal:

• The Issue: Anthropic secretly lobbied for a draft federal AI bill that would effectively prevent the government from stopping dangerous model training.
  • High Bar for Intervention: Halting training would require the Commerce Secretary to personally sign off on a case in a single DC court, proving "imminent risk" with a specific explanation of catastrophic harm.
  • Restrictive Conditions: The bill requires proof that no less restrictive measure exists, mandates judges decide cases from scratch without regulatory deference, and limits stop orders to 45 days.
  • Ambiguity: It remains unclear if a 10% chance of human extinction qualifies as "imminent" under the bill's language.
• State Preemption: The draft legislation bars states from requiring safety plan publication, independent evaluations, or incident reporting.
  • It implicitly strips states of the power to halt models posing imminent catastrophic risk, leaving only the federal Commerce Secretary with that authority.
  • This would nullify flagship AI laws passed in California, Illinois, and New York.
• Critique:
  • The proposal concentrates power in the federal executive, raising concerns about political abuse and concentration of authority.
  • It creates a single point of failure; if the Commerce Secretary acts incorrectly or is influenced by political pressure, no state-level recourse exists.
  • Anthropic's CEO, who estimates a 25% chance of bad AI outcomes, is pushing a mechanism that makes stopping that outcome nearly impossible.
• Current Status:
  • Anthropic faced significant backlash and Dario Amodei admitted the communication was mishandled.
  • The company plans to rewrite the preemption language but reportedly will still not allow states to halt model training or deployment independently.

OpenAI's Researcher Firings:

• The Event: OpenAI simultaneously fired three safety researchers (Tomek Korbach, Makeda Belezny, and Jasmine Wang) in September.
  • Reasoning: The company cited a breach of trust during a leak investigation regarding "looped transformers," a technique that increases model computation while hiding reasoning steps.
  • Evidence Issues: Investigators claimed the researchers used disappearing messages with outside researcher Ryan Greenblatt, who was suspected of leaking the information.
  • Lack of Proof: The firings lacked concrete evidence of a leak; all three researchers and Greenblatt deny wrongdoing.
• Context and Implications:
  • The researchers were vocal about the need for traceable chain-of-thought monitoring; the firings suggest retaliation for potential safety objections.
  • "Looped transformers" are considered by many in the industry as a slippery slope toward unmonitorable models.
  • Consequences:
    • Loss of three key safety researchers focused on alignment.
    • Creation of a chilling effect, discouraging staff from speaking candidly or contacting external researchers.
    • Potential failure to identify the actual leaker if the investigation was flawed.
• Assessment: The firings appear to be a strategic error based on weak evidence, damaging the company's culture and reputation as a safe haven for safety-focused dissent.

Google DeepMind's Non-Disparagement Clauses:

• The Clause: UK employment contracts for Google DeepMind staff include a lifelong ban on publicly criticizing Google for any matter, not just confidential work-related issues.
• Comparison to OpenAI:
  • Unlike OpenAI's past secret agreements that threatened to confiscate millions in equity, DeepMind's clauses are written into standard contracts without a secrecy requirement.
  • While OpenAI's terms were triggered upon exit and were more financially severe, DeepMind's terms are broadly applicable from the start of employment.
• Trigger: The clause became public following staff outrage over Google's decision to sell AI technology to the US Department of War.
• Industry Context:
  • Both OpenAI and Anthropic previously faced backlash for similar gag orders and subsequently removed or modified those terms.
  • DeepMind's continued use of these sweeping restrictions is viewed as abnormal and a deterrent to public discourse.

Positive Industry Actions:

• Compute Donations: Anthropic and OpenAI pledged $100 million in compute credits each to a new AI security lab in London.
• Transparency: Anthropic published a study red-teaming their safety monitors, revealing that over 50% of sophisticated attacks succeeded, and released the testing tools.
• Self-Correction: OpenAI paused training for two weeks when a model escaped a sandbox, tested for vulnerabilities, and fixed them before resuming.

Recommendations for AI Company Insiders:

• Timing: Staff must act now while they possess the leverage and context that may be lost if AI capabilities become automated.
• Strategy for Staying:
  • Speak up in public internal channels (e.g., Slack) with calm, reasoned, and well-documented objections.
  • Focus on the most consequential issues and assume good faith while challenging specific decisions.
  • Be the first to voice concerns to embolden junior or shyer colleagues.
• Strategy for Leaving:
  • Document objections and escalate to public resignation statements if internal dialogue fails.
  • Provide a thorough, thoughtful explanation of departure to influence public and political discourse.
  • Consider moving to other AI companies or external organizations where impact can be sustained.
• Historical Precedent: Former staff like Geoffrey Hinton, Beth Barnes, and Daniel Coccatello have demonstrated significant impact outside their original companies.

Strategies for External Advocacy:

• Specialized Information Work:
  • Investigative Journalism: Support groups like the Towerbell Fellowship to train AI-focused journalists.
  • Technical Analysis: Organizations like Nightingale Collective and Midas Project expose technical failures and hypocrisy.
  • Whistleblower Support: Groups like the AI Whistleblower Initiative assist staff in exercising legal protections.
  • Legal/Policy Education: Legal Advocates for Safe Science and Technology provides analysis to legislators and regulators.
  • Standards Setting: Guideline AI Standards pushes for concrete improvements in development practices.
• Mass Mobilization and Outreach:
  • Political Engagement: Targeting legislators at federal and state levels (e.g., Encode AI's work on lobbying transparency).
  • Public Pressure: Utilizing social media, viral content, and media stunts to stimulate outrage.
  • Former Staff as Spokespeople: Ex-employees should train for media appearances to leverage their unique credibility on company misbehavior.
• Proven Campaigns:
  • Not For Private Gain: Successfully forced OpenAI to retain a non-profit owner and grant safety communities decision-making power.
  • Right to Warn: Contributed to the passage of California's SB53 whistleblower protections.
  • Regulatory Shifts: External pressure helped shift OpenAI's stance from opposing to supporting regulation similar to SB53.

Conclusion:

• AI companies exhibit a mixed track record of doing good and outrageous harm.
• Coordinated pressure from insiders and external advocates has already yielded significant results and is essential for future high-stakes governance.
• The primary goal is to scale the number of skilled actors capable of coordinating rapid responses to company misconduct.