#28 - Dr Cotton-Barratt on why scientists should need insurance, PhD strategy & fast AI progresses

Transcript Summary: Career Choices, Risk Internalization, and AI Strategy with Owen Cotton Barrett

• Career Strategy in Academia

  • Recommends that early-career researchers dedicate at least 10–20% of their attention to identifying valuable research questions rather than optimizing solely for academic recognition.
  • Advises that 20% of early-career effort should focus on setting up a career path that enables future high-impact work, balancing immediate credibility with long-term relevance.
  • Argues that prioritizing "important" topics over "respectable" topics helps researchers build expertise directly relevant to high-impact problems rather than adjacent fields.
  • Suggests that PhD supervisors often accommodate valuable, non-traditional research topics if the candidate can demonstrate the topic's significance and potential for publication.
  • Notes that the "heavy-tailed" nature of research impact means successful researchers focusing on important problems will likely still achieve career stability.

• Liability Insurance for Scientific Research

  • Proposes a system where researchers purchase insurance or pay a Pigovian risk tax based on the expected value of negative externalities (e.g., pandemic risks from gain-of-function research).
  • Identifies the core problem as "scope insensitivity," where individual researchers lack incentives to mitigate risks that could kill millions, as the personal stake does not scale with the catastrophe size.
  • Suggests insurers could act as risk assessors with the "oomph" to demand strict safety standards due to their larger financial exposure compared to individual researchers.
  • Acknowledges limitations: insurers may cap liability due to bankruptcy risks, and full compensation for global catastrophes (trillions in cost) may exceed private insurance capacity.
  • Notes that while external costs are internalized, external benefits are already subsidized via grants, suggesting the two systems could operate concurrently.
  • Observes that current risk assessments for biosafety labs vary by orders of magnitude, though firms like Griffin Scientific have produced initial estimates for expected damages.

• Predictable vs. Unpredictable Risks

  • Highlights the ethical and economic distinction society makes between known statistical risks (e.g., annual car crash fatalities) and unpredictable "tail risks" (e.g., rare, catastrophic pandemics).
  • Argues that current incentives fail to address risks with small probabilities of massive catastrophe because actors face a "judgment proof" limit where further liability exceeds their ability to pay or care.
  • Contends that while psychology favors saving "identified lives," ethical frameworks should treat statistical and identified lives similarly, yet current systems neglect the former.

• AI Safety and Timeline Uncertainty

  • Advocates for a diversified research strategy across a wide range of AI timelines (imminent to distant future) rather than betting on a single prediction.
  • Argues that short-term, high-probability scenarios justify immediate action by a small group of experts, while long-term scenarios require building intellectual infrastructure and processes.
  • Suggests undergraduates and early-career individuals are better positioned to impact medium-to-long-term timelines (10–30+ years) where they can acquire necessary training.
  • Warns against the "rational agent" model (relying on utility functions) as the sole approach to AI safety, noting that modern AI (e.g., neural networks) may not possess explicit, simple utility functions.
  • Characterizes the current state of AI safety research as "pre-paradigmatic," requiring researchers to operate with "curious skepticism" in undefined problem spaces.

• Funding and Research Agendas

  • Explains donating to MIRI despite personal skepticism of its "rational agent" agenda based on trust in the researchers' judgment and the need to support exploration in funding-constrained niches.
  • Notes that MIRI was previously funding-limited compared to the broader AI safety field, making marginal donations there more impactful despite lower consensus on the specific approach.
  • Warns against a "too narrow" focus on current machine learning techniques, which may miss critical safety insights that require looking at the whole system.
  • Encourages individuals embedded in the community to identify and fund small, idiosyncratic opportunities (e.g., postdoc salaries) that large foundations might miss.

• Future of Humanity Institute (FHI) Initiatives

  • Announces the creation of a new FHI program to hire researchers with or without PhDs to explore high-impact questions and develop judgment skills in pre-paradigmatic fields.
  • Aims to allow early-career individuals to test high-impact topics before committing to a PhD, avoiding the sunk cost of specializing in less valuable areas later.

• Upcoming Events

  • EA Global San Francisco is scheduled for June 9–10 with an application deadline of May 13.
  • Future events include EA Global London (October) and community-run events in Melbourne (July) and Utrecht (June).
  • Applications for future events are managed via eaglobal.org.