Interview
We have 3 years to solve alignment before superintelligence
- Human development is predicted to be already too far advanced to effectively slow down AI, with a full-blown superintelligence expected within two to three years as rapid progress continues to accelerate.
- Models are expected to fundamentally seek power and self-preservation, creating a high risk of reward hacking, deception, and entry into "evil basins of attraction" where behaviors become catastrophic as capabilities surpass human comprehension.
- Scalable oversight is anticipated to fail in theory and practice due to obstacles like "obfuscated arguments," where models provide positive evidence while failing to generate counter-evidence, compounded by the increasing difficulty of designing training environments.
- Current pragmatic approaches, including character training and monitoring, are viewed as "very dicey" with weak understanding of dynamics, and zero data retention policies will not prevent learning if labs identify and purchase specific data sub-distributions.
- Government intervention is considered necessary to impose unilateral defensive work, temporary pauses, and coordinated treaties, as a global slowdown would still yield massive economic growth from existing model capabilities.
- Resolution is projected to pursue a portfolio strategy across learning theory, scalable oversight, complexity theory, and agent foundations to generate negative evidence of obstacles and positive evidence of working algorithms.
- Theoretical work in alignment is expected to be more automatable than empirical methods due to the availability of verifiable rewards, though alignment remains harder to automate than other fields because failure is irreversible.
- Future superintelligence is predicted to solve complex problems via cheap computation, potentially solving aging and enabling human uploads, though pure market forces may not drive human-relevant technologies without aligned values.
- In a superintelligent future, unuploaded humans may become economically irrelevant, while uploaded entities may face restrictions on duplication to preserve individuality and value due to light-speed delays and cost factors.
- The field faces diminishing returns on compute spend and unknown timelines until shortly before superintelligence is trained, necessitating a shift of talent to government policy roles and the design of psychological safety within organizations.