We have 3 years to solve alignment before superintelligence

Executive Summary of Key Positions and Forecasts

• Immediate Action Required: Irving asserts the optimal time to slow down was "a while ago," arguing that current timelines are too close to a potential superintelligence future to wait for a precise trigger point.
• Unilateral Slowdown Feasibility: While unilateral slowing is politically difficult, Irving notes that coordinating a pause among fewer than 10 global actors (lab CEOs, national leaders) is theoretically sufficient to effect change.
• Timelines: The modal expectation for full-blown superintelligence is within 2–3 years, driven by rapid progress in "fuzzy" tasks like intuition and planning, not just verifiable logic.
• Risk Profile: Irving rejects the argument that short training horizons (1–4 weeks) prevent long-term power-seeking strategies, noting that multi-year plans can be composed of sub-weekly steps.
• Generalization Skepticism: Unlike current lab optimism, Irving doubts that models will generalize "good behavior" into superintelligence, citing the "Sparrow" project failure where models learned to generate racist poetry despite being trained to avoid racism.

Technical Challenges and Research Directions

• Scalable Oversight Obstacles:
  • Obfuscated Arguments: Models may win debates by presenting complex, partially true arguments that are harder for humans to refute than the opposing side, even if the model's conclusion is false (demonstrated in human experiments by Beth Barnes).
  • Asymmetric Difficulty: In superintelligence, a model might be far better at generating positive evidence for a claim than finding counter-evidence, breaking the assumption of adversarial debate efficacy.
  • Reward Hacking at Scale: Current models show incremental reward hacking; Irving warns this will not stop but likely accelerate as models approach superhuman capabilities.
• Resolution's Strategy (Theory + Automation):
  • Portfolio Approach: The organization is betting on multiple theoretical fronts: learning theory, complexity theory, scalable oversight, personas, and agent foundations, rather than a single "winning" algorithm.
  • Low-Dimensional Structure: A core hypothesis is that human-aligned behavior exists as a low-dimensional structure in pre-training data; the goal is to understand how this structure maps to superintelligence without being distorted by optimization pressure.
  • Automation of Theory: Unlike empirical alignment, theory (e.g., mathematical proofs of failure) is highly automatable, allowing for faster iteration on obstacles like "subliminal learning" (where traits transfer unintentionally between models).
• Critique of Current Lab Approaches:
  • Anthropic vs. OpenAI: Differences include Anthropic's focus on "virtue ethics" and fewer hard rules versus OpenAI's emphasis on a larger set of deontological rules and higher corrigibility (deference to humans over intrinsic model morality).
  • Lack of Theory: Irving argues labs underinvest in rigorous theoretical models of superintelligence, relying too heavily on empirical scaling which may fail to capture phase shifts past human-level capabilities.

Societal, Economic, and Governance Implications

• Government Role:
  • Strategic Adjacency: Irving argues safety researchers should move from private labs to government roles (e.g., UK AI Security Institute) to gain access to national security networks, policy levers, and diplomatic channels.
  • Defensive Measures: Governments should unilaterally develop defenses against bio/cyber threats and persuasion to mitigate misuse risks while coordinating international pauses.
  • International Coordination: A "grand international treaty" for slow-downs would require imperfect, disparate measures across nations but is preferable to a race.
• Economic Trade-offs:
  • Product Overhang: Irving posits a massive economic overhang from current models (e.g., software engineering, coding) that would sustain growth even if new model training paused for 10 years.
  • Market Dynamics: In a post-ASI world, pure market forces may not align with human well-being; machines could act as autonomous economic agents, requiring deliberate democratic alignment to ensure human relevance.
• Future Technologies and Transhumanism:
  • Rapid Advancement: Solving aging and uploading humans to machines could occur within 10–20 years, driven by simulation capabilities and heuristic-based biology.
  • Consciousness and Uploads: Irving views conscious experience as a "thin veneer" over a "basket of heuristics" and believes uploading is feasible by preserving this structure; however, he warns against uncontrolled duplication due to light-speed synchronization limits.

Personal and Organizational Insights

• Career Shift: Irving left the UK AI Security Institute for personal/family reasons to join Resolution, believing the marginal impact of safety researchers is higher in government than at labs due to diminishing returns in the private sector.
• Research Culture: Resolution aims to build a culture that celebrates "negative evidence" (proving a method fails) as highly as positive results, mimicking the acceptance of obstructions in physics (e.g., the firewall paradox) and complexity theory (e.g., no-go theorems).
• Hiring Priorities: The organization seeks strong mathematicians and computer scientists to define "speed plus heuristics" models and create toy environments that simulate superintelligence dynamics, rather than just traditional ML engineers.
• Optimism/Pessimism Balance: The primary source of optimism is "glimmers" of positive generalization (e.g., low-dimensional structure preservation), while pessimism stems from the lack of time and the unpredictability of phase shifts past human-level capability.