Epistemic systems & layers of defence against global catastrophes | Owen Cotton-Barratt (2020)

The Research Scholars Program (RSP)

• A two-year, fully funded training program at Oxford's Future of Humanity Institute (FHI) for early-career researchers interested in long-term human flourishing.
• Aims to solve the "pivot" problem where researchers must abandon unrelated PhDs to work on existential risk, by providing space to explore topics without immediate pressure to produce a narrow academic thesis.
• Applications are scheduled to open in spring 2021, with new cohorts potentially joining every two years starting in autumn.
• The program emphasizes meta-level thinking, giving participants ownership to critique and reshape the curriculum and research direction.

Defense in Depth Against Human Extinction

• A framework classifying extinction risks by three sequential stages:
  • Origin: How a threat begins (e.g., a lab leak, geopolitical conflict).
  • Scaling: How a small problem grows into a large-scale catastrophe.
  • Termination: How a large catastrophe transitions into human extinction.
• Key Strategic Insight: Reducing the probability of failure at any single stage by half yields an equivalent reduction in total extinction risk.
  • This justifies investing in "later" defense layers (e.g., resilience against starvation) even if the probability of reaching that stage is low, provided interventions there are tractable.
• Diminishing Returns Argument: If a stage has an extremely high probability of leading to the next stage (e.g., 99.99%), the effort required to halve that probability may be disproportionately high compared to reducing lower-probability earlier stages.
• Risk Factors for Weakness: External events like societal collapse or war can degrade defense layers by destroying governance, communication, and incentive structures needed for response.
• Robustness of Abstract Modeling: The framework is designed to remain valid regardless of specific empirical assumptions, highlighting structural vulnerabilities common to all risks (e.g., AI, pandemics, volcanoes).

Estimating Total Existential Risk

• Disagreement exists within the community regarding the prior probability of extinction this century (e.g., Toby Ord's ~17% vs. Will McCaskill's ~1%).
• Refraining from Reference Class Forecasting: Cotton Barrett argues against assuming our century is "average" simply because humanity has survived many centuries; instead, he favors models where risk increases with technological power, potentially making the current century the most dangerous.
• Optimistic Prior Assumption: Even if risk is monotonically increasing, it is strategically rational to bet on the possibility of stabilization, as the payoff of a long future is infinite while the cost of inaction is certain extinction.

Everyday Long-Termism: A "Web of Virtues" Strategy

• Core Proposal: If long-termism is true, the most robust advice for the general population is to cultivate and spread virtues that facilitate good decision-making under uncertainty.
• Target Virtues:
  • Clarity and Truth-Seeking: Prioritizing clear thinking and evidence over dogma.
  • Scope Sensitivity: Weighing decisions by the absolute number of lives affected, not just local impact.
  • Cooperation and Long-Termism: Encouraging institutions and individuals to consider long-term consequences and avoid antagonistic races.
• Rationale for Indirect Action: In a world of high uncertainty, promoting good decision-making processes is more robust than guessing specific future solutions; it "passes the buck" to future generations who will have better information to make the right specific choices.
• Trade-offs: This approach complements direct strategic action; exhibiting virtue is often a necessary component of being effective in high-stakes roles (e.g., on policy committees).

Mathematics in Existential Risk Research

• Utility of Modeling: Mathematical modeling is valuable for forcing explicit assumptions, surfacing disagreements, and testing intuition, though complex models with too many variables are often less useful than simple, robust ones.
• Key Insight from Modeling: Research on risk allocation suggests that people should prioritize risks that are imminent (e.g., within years) because the future workforce capable of addressing them is finite, whereas distant risks have a larger future pool of researchers.
• Advice for Mathematicians:
  • Focus on building "crisp" questions that can be tackled by others.
  • Leverage the habit of "testing definitions" in various scenarios to internalize concepts rather than relying on symbolic manipulation alone.
  • Use simple models to capture essential dynamics, avoiding the trap of over-complication which obscures the signal.

Community and Career Guidance

• Nerd Sniping: The most effective way to engage others is to present specific, crisp, solvable puzzles that feel interesting, rather than vague, overwhelming宏大 concepts.
• Failure Modes of "Virtue" Advocacy: Risks include defaulting to common-sense morality without critical scrutiny and failing to pivot to direct strategic action when the "fog of uncertainty" clears enough to identify high-impact interventions.
• Target Audience: The "Web of Virtues" strategy is particularly valuable for those who do not yet have a clear specific action, serving as a default path to ensure they remain a constructive part of the solution.