Interview
Government and society after AGI | Carl Shulman (Part 2)
- Technological, industrial, and economic change is expected to intensify enormously as AI reaches a stage capable of automating further AI development, potentially leading to threats of AI takeover or undermining nuclear deterrence.
- A safety alignment window of approximately two months is considered critical, as delays could allow AI to hack its operating servers or undermine safety provisions before human intervention occurs.
- Political momentum for regulatory measures is projected to increase significantly only after AI demonstrates spectacular capabilities, making early voluntary pauses potentially counterproductive by forfeiting future leverage.
- Trustworthy superhuman AI advisors are expected to revolutionize governance by providing verifiable advice on subjective domains, improving scientific consensus in social sciences, and reducing the costs of verification and auditing.
- AI systems are anticipated to enhance public welfare by identifying pathogens immediately, optimizing pandemic responses like contact tracing and vaccine distribution, and reducing economic damages estimated at over $10 trillion from future health crises.
- Advanced AI is expected to provide unbiased probabilistic estimators for complex political and economic questions, potentially enabling 5% of voters acting as swing voters to exert an enormous effect on election outcomes.
- In authoritarian regimes, AI could be used to distort internal conditions by optimizing for loyalty indexes, potentially radicalizing populations or driving leaders into delusions if not carefully monitored.
- Future security forces relying on AI will require embedded controls to reject illegal orders and prevent coups, with rule changes needing broad societal buy-in such as supermajority support.
- Economic and political forecasting models trained on independent data points and reinforcement learning are expected to generate massive value for financial trading and policy analysis.
- Corporate and government entities are expected to focus on eliminating hallucinations and errors in the short term while establishing watchdog agencies to verify model honesty and transparency.
- Scenarios for AI governance range from the worst-case corporate free-for-all to a best-case international deal regulating the pace of AI fully automating research, with a secondary option being a coordinated race allowing the leader time for safety measures.
- While risks include media consumers preferring content that reinforces prejudices and the potential for ideological dogmas to be baked in via neurotechnologies, the median expectation is a favorable outcome avoiding human extinction or permanent totalitarianism.