Interview
Joseph Carlsmith - Utopia, AI, & Infinite Ethics
- Joe Carl Smith's Roles: Senior research analyst at Open Philanthropy focusing on AI existential risk; doctoral student in philosophy at the University of Oxford; author of the "Hands and Cities" blog.
- AI Research Focus: Investigating AI timelines and "takeoff speeds" to assess catastrophe probabilities; aims to inform prioritization strategies based on how quickly AI becomes transformative.
- Probability Thresholds: Smith argues substantive differences in prioritization exist between 1%, 10%, and 90% probabilities of catastrophic AI failure.
- Utopia Definition: Defines utopia as a "profoundly better" future that is actually possible through non-crazy, rational planning, representing a gap in value similar to "being asleep vs. awake" rather than incremental improvements.
- Utopia Alienation: Acknowledges future utopias may initially appear alienating to present humans but argues that with increased cognitive capacity and wisdom, individuals would fully endorse them.
- Wisdom as Prerequisite: Views the path to utopia as a philosophical process of species-wide cognitive enhancement and wisdom accumulation, rather than a static implementation of normative theories.
- Infinite Ethics Motivation: Addresses the difficulty of applying ethical principles to infinite worlds, arguing these principles often "break" when applied to scenarios with infinite value or populations.
- Infinite Influence Hypothesis: Proposes that in an infinite universe, evidential decision theories (relying on correlated copies) could grant individuals causal influence over infinite populations despite physical light-speed constraints.
- Epistemic Caution: Maintains a "middle ground" on infinite ethics, advocating for survival and the acquisition of future wisdom before attempting to resolve these destabilizing philosophical problems.
- Insect Moral Status: Expresses significant uncertainty regarding the moral patienthood of insects (e.g., ants), rejecting both total neglect and "Jane" extremes (avoiding all insect contact) in favor of accepting trade-offs.
- Anthropic Reasoning (SIA vs. SSA): Prefers the Self-Indication Assumption (SIA) over the Self-Sampling Assumption (SSA), arguing SIA better handles the probability of existing in worlds with more observers.
- SSA Critique: Notes SSA implies implausible telekinetic influences on world states (e.g., preventing a boulder from hitting a puppy based on reference class probabilities) and leads to the "Doomsday Argument."
- SIA Limitations: Acknowledges SIA can naively predict the observer lives in an infinite universe or a universe filled with simulations, creating similar infinities-breaking problems in cosmology.
- AI Capability Estimates: Estimates the human brain's task-relevant computational capacity at approximately $10^{15}$ FLOPS, using triangulation of neuroscience data, vision system comparisons, and physical energy limits.
- Training Cost Uncertainty: Cites Open Philanthropy's Ajeya Cotra's models estimating training costs for human-level AI anywhere from $10^{23}$ to $10^{41}$ units of compute, centered on $10^{32}$, with high uncertainty driven by "horizon length" of training.
- Scaling Hypothesis Assumption: Notes current AI timeline estimates rely on the assumption that current deep learning techniques and scaling laws will succeed without major algorithmic breakthroughs, though he assigns weight to the possibility of earlier timelines via such breakthroughs.
- Futurism Critique: Criticizes futurism for becoming "unreal" due to the necessity of using "extremely lossy abstractions" when modeling complex future systems with limited cognitive tools.
- Definite Optimism: Aligns with Peter Thiel's view that vague "indefinite optimism" is insufficient, advocating for concrete, detailed visions of the future to guide action.
- Writing Methodology: Attributes high blog productivity to a "stream of consciousness" approach and avoiding over-editing, prioritizing volume and idea generation over polished perfection.
- Book Recommendations: Recommends The Precipice by Toby Ord for existential risk; Angels in America and Housekeeping for literature; and the corpus of Nick Bostrom for philosophical engagement.