Interview, Fireside Chat
AI Nationalization is Inevitable – Leopold Aschenbrenner
- The long-term outlook projects a high-stakes environment extending to the survival of liberal democracy, the CCP, and the global order for the next century, with forces activated beyond current scope prioritizing national security implications akin to the initial nuclear energy era.
- The national security state is expected to become intimately involved in AGI development, potentially resulting in nationalization, public-private partnerships, defense contracts, or a government project that absorbs all talent to manage the risks of super intelligence.
- A volatile and dangerous initial period is predicted where technology advances from primitive tools to Intercontinental Ballistic Missiles (ICBMs) within years, creating a race where a private CEO cannot be relied upon as a benevolent dictator.
- It is asserted that a single private entity with a six-month lead over competitors could become the most powerful weapon ever and potentially overthrow other companies, while a one-year lead on Artificial Super Intelligence (ASI) is viewed as highly concerning regarding privatization.
- If private labs dominate, the landscape is expected to shift beyond two or three major players to include nation-states such as China, Russia, and North Korea due to the inability of private entities to maintain sufficient security against state-level threats.
- The timeline for initial instability is estimated to last 10 years within a few years, during which the offense-defense balance remains precarious regarding bioweapons and AI, before potentially reaching a stable situation with universal defense capabilities.
- Safety alignment during the intelligence explosion is forecasted to occur via a war-like "fog of war" with a sane chain of command rather than through years of bureaucratic regulation or deliberative schemes.
- The outlook suggests that startups are unfit to handle weapons of mass destruction (WMDs), necessitating government control over security and system-level prompts to prevent the creation of bioweapons while many companies continue development.
- Despite a government project framework, the deployment of AI is expected to be broad, potentially including the open-sourcing of models that are two years behind the frontier, as private companies may still operate under strict government security control.
- The future of the computing infrastructure is predicted to remain centralized with only two or three big players, as the $100 billion cost of advanced computer clusters is not expected to drop to $1,000 within a lifetime, preventing a decentralized open-source community.
- There is a fear that private competition between entities led by figures such as Demis and Sam will result in a race to develop "nuclear arsenals in the robot," undermining mutual deterrence.
- Initial applications of ASI are anticipated to "suck" and focus primarily on stabilizing the situation and preventing the proliferation of new WMDs rather than delivering immediate commercial utility.
- Trust in automated AI researchers regarding safety alignment is currently low, with concerns expressed about generalization scaling curves and a specific prediction that China may deploy "robo armies" that force immediate action.