Interview, Podcast
Epistemic systems & layers of defence against global catastrophes | Owen Cotton-Barratt (2020)
- Research Scholars Program applications are expected to open in the spring of 2021, followed by a two-year training program at the Future of Humanity Institute designed to support long-termist research and career development for individuals with and without PhDs.
- New cohorts for the Research Scholars Program are projected to arrive every two or three years starting in the autumn, with participants spending two-year contracts in Oxford and having the freedom to influence the program's structure.
- Interviews featuring Toby Ord regarding The Precipice are anticipated by the release of the current episode, with additional interviews on other topics planned for release in the new year.
- The program aims to generate insights into creating environments beneficial for long-termist research, with plans to expand these elements to a wider audience based on operational experience.
- Mathematical modeling is viewed as underapplied in existential risk analysis, yet complexity should be avoided; simple models are considered more accurate due to human cognitive limits, while mathematical training is valued for instilling argumentative care and habit formation.
- Existential risk is predicted to amplify societal good but push back against bad outcomes, creating a puzzle regarding how extinction could occur despite these pressures, though direct extinction by accident is deemed vanishingly unlikely due to the large population size.
- A pandemic with a mortality rate under 100% cannot cause direct extinction but may pose a risk if it triggers a breakdown of societal structures, potentially leading to increased hostility or cooperation depending on social dynamics.
- Risk reduction efforts are expected to be most effective at low-probability stages (e.g., 0.002%), whereas reducing risks at over-determined stages (e.g., 99.99%) yields little proportional benefit, suggesting a strategy to spread focus across stages unless a specific risk is exceptionally high.
- Interventions with positive externalities for large populations are predicted to be undersupplied by rational market actors due to attention being a limited resource focused only on salient events.
- Significant points of disagreement are expected between common sense morality and long-termist virtues, though more concordance is anticipated regarding short-to-medium-term well-being; however, spreading virtues like clear thinking and scope sensitivity is seen as a robust first step for individuals and institutions.
- Cultivating virtues such as scope sensitivity and truth-seeking is recommended as a best initial step for those without clear future plans, with a focus on individuals closer to important leverage points like celebrities and teachers.
- If extinction risks continue to ramp up indefinitely, humanity is expected to face fewer generations and certain doom, whereas stabilizing risks offers a significantly larger payoff for a future where long-termism fosters better decision-making processes.
- Experts anticipate that a 99% chance of extinction from an engineered pandemic might be hard to address in its final power-acquisition stage, whereas slowing down an engineered pandemic in earlier stages may offer low-hanging fruit compared to preventing initial agent releases.
- Societal collapse scenarios are assessed with uncertainty, ranging from a plausible extinction risk exceeding 10% to potentially below 0.1%, though recovery via another industrial revolution is considered moderately feasible if the environment remains habitable.
- Long-termist strategy expects a diverse research community rather than universal focus on AI alignment, utilizing a "web of virtue" to ensure future actors with better context can make superior decisions even if current understanding is incomplete.
- The Research Scholars Program seeks to hire diverse scholars to provide them space to determine important contributions, utilizing a model that differs from standard master's programs by allowing structural changes based on participant feedback.
- Simple models and crisp questions are advocated as "virtuous nerd sniping" to allow others to grasp puzzles and gain traction, while mathematical-minded individuals are encouraged to distinguish between technical system interaction and broader perspective-seeking skills.
- The program will likely include participants with varying backgrounds, including those seeking immediate employment and those considering future PhD studies, with an expectation that sociologists and economists may contribute more to virtue analysis than psychologists.