Interview
#28 - Dr Cotton-Barratt on why scientists should need insurance, PhD strategy & fast AI progresses
- Upcoming EA Global San Francisco is scheduled for June 9–10, with an application deadline of May 13, while community-run events are planned for Utrecht in June, Melbourne in July, and the next major event in London in October.
- Selection for future research topics is expected to balance immediate fascination with importance and value, acknowledging that scope insensitivity may hinder efforts to reduce high-consequence risks like pandemic flu or laboratory accidents that could cause global catastrophes.
- Liability insurance for high-stakes research faces significant hurdles, requiring a bankroll of tens to hundreds of billions but remaining unable to cover potential trillions in damages, leading to likely unpopularity among scientists who fear no net increase in research budgets.
- AI safety research is viewed as a valuable direction for systems that are safe and robustly beneficial, though specifying safety criteria is challenging due to the intractability of understanding internal mechanisms in neural nets and the likelihood that utility functions may not appear in code.
- Expert consensus indicates uncertainty regarding the timing of transformative AI, with possibilities ranging from unexpected breakthroughs in a few years to arrivals in many decades, creating pressure to prioritize scenarios with probabilities of a percent or less where only current individuals can influence outcomes.
- Impact strategies for undergraduates and early-career researchers are suggested to focus on scenarios at least 10 to 25 years out, as those waiting 30 to 40 years will face different challenges regarding workforce availability and may require building intellectual infrastructure rather than direct intervention.
- A new program is being established to hire researchers without PhDs in messy pre-paradigmatic fields, aiming to provide a space for exploration before doctoral study since traditional PhDs rarely cover AI safety and hiring is often hindered by a lack of research experience.
- Career advice recommends dedicating at least 10–20% of attention to identifying valuable topics and another 20% to career setup strategies early on, noting that delaying such reflection until after obtaining a tenure-track position is likely more costly due to evaluation metrics based on post-PhD output.
- Funding priorities are shifting away from the highly reliable agent design agenda, with funding seen as less of a constraint for many AI safety areas and MIRI's recent large grant reducing its marginal value as a donation target in favor of supporting small, idiosyncratic opportunities.
- The podcast hosts anticipate that a "mix" of research selection approaches will yield the most results, though they note the current listener base familiar with the thesis topic is likely very small, estimated at two or three individuals.