Interview
Joseph Carlsmith - Utopia, AI, & Infinite Ethics
- A radically better future is considered achievable through a patient, long-term process of species-level cognitive and moral improvement rather than a single philosophical implementation.
- Future utopias will likely involve profound alienation for current humans due to a cognitive gap significantly larger than that between historical and present humans, though this state would be endorsed upon gaining sufficient wisdom.
- Utopian states are predicted to be concrete, limited situations with frictions and resource constraints, distinct from religious visions of heaven or static states of uniform hedonism.
- Current AI development assumptions may underestimate the timeline for human-level AI if non-optimal techniques are used, potentially leading to earlier arrival due to algorithmic breakthroughs.
- Emulating the human brain is estimated to require approximately $10^{15}$ floating point operations per second, with total training costs ranging uncertainly from $10^{24}$ to $10^{41}$ flop equivalents.
- Quantum mechanical processes are expected to play no significant role in human cognition, while the brain's algorithmic complexity is viewed as additive at a fundamental level.
- Infinite ethics frameworks are deemed necessary for any hypothesis where the universe is infinite, requiring a responsible middle ground between total ignorance and extreme "Jane-like" behavior regarding trade-offs and uncertainty.
- Decision theory preferences lean toward non-causal theories, with the Self-Indication Assumption (SIA) favoring worlds with larger populations and the Self-Sampling Assumption (SSA) leading to a doomsday argument for early extinction.
- Anthropomorphic theories are expected to break down when applied to infinite scenarios, similar to the breakdown of ethical theories in infinite ethics.
- Modeling the vast future requires extremely lossy abstractions because imagination is a limited tool, and concrete descriptions will almost certainly contain specific errors despite their utility.
- Competitive pressures are anticipated to push civilization toward bad directions, but wise coordination and preemptive action are viewed as capable of staving off these negative effects.
- The 21st century is expected to be viewed by future humans as a formative era characterized by risks and horrors, rather than one to be remembered with pleasure.
- Current approaches to global improvement are often limited by a lack of structuring utopian visions, focusing instead on modest gains like GDP growth or universal healthcare.
- Significant changes to mental capacities will likely be required for humans to understand a future of increased cognitive scale.
- A utopian vision is not expected to have a hard threshold of implementation but rather to represent the top state among finite states of the affectable universe.
- The speaker highlights "The Precipice" by Toby Ord as a primary source conveying the most critical ideas regarding existential risk.