Career review: AI governance & policy | Cody Fenwick

Core Context and Risk Landscape

• The field of AI governance is expanding rapidly due to widespread attention on advanced AI capabilities gained between late 2022 and 2024.
• Key risks driving this focus include misaligned AI, uncontrollable systems, potential global catastrophes, human disempowerment, and extinction.
• There is currently no consensus on the optimal governance strategies, with significant uncertainty regarding which policies will effectively mitigate risks.
• Public policy and corporate governance are deemed necessary to ensure advanced AI remains beneficial and low-risk.

Primary Aims of AI Governance

• Prevent the deployment of AI systems posing significant, direct threats of catastrophe.
• Mitigate the negative impact of AI on other catastrophic risks, such as nuclear weapons and biotechnology.
• Guide the integration of AI into society and the economy to minimize harms and maximize universal benefit.
• Reduce the risk of an AI arms race between nations and corporations.
• Ensure AI developers are incentivized to cooperate and prioritize safety.
• Slow development and deployment if advancements outpace safety and control capabilities.

Six Broad Categories of Work

• Government Work:
  • The U.S. federal government is expected to be the most significant player due to its jurisdiction over major AI companies (e.g., Anthropic, OpenAI, Google DeepMind) and the chip supply chain.
  • The U.S. and U.K. announced new AI safety institutes in 2023, and the EU established an AI office in 2024.
  • Career trajectories include the standard seniority track, specialization via movement between sectors, and direct impact roles without long-term career capital accumulation.
  • U.S. citizenship is often a strict requirement for key national security and executive branch roles.
• Research on AI Policy and Strategy:
  • High-priority research needs include concrete policy development, public/expert opinion surveys, and legal feasibility studies.
  • Technical research areas include compute governance and theoretical questions regarding societal implications.
  • Organizations like Epoch AI focus on forecasting future AI development trajectories to inform governance.
  • Researchers are warned that much existing governance research may prove useless without rigorous reflection and field input.
• Industry Work (Internal Policy):
  • Internal governance roles involve deciding board composition, setting organizational incentives, and modeling threats.
  • Key tasks include implementing risk-reduction proposals, red teaming, and ensuring compliance with increasing government oversight.
  • Industry leaders often express concern about catastrophic risks, creating potential for cooperation on regulation.
  • Companies face incentives to commercialize products, which may conflict with risk-reducing policies requiring significant buy-in.
• Advocacy and Lobbying:
  • Advocacy aims to shape public discourse and promote specific ideas, while lobbying targets specific legislation and officials.
  • Approaches include collaborating with industry to advocate for beneficial regulation or joining independent organizations to push for halting development.
  • The May 2023 Center for AI Safety statement, supported by over 100 signatories (including OpenAI, Google DeepMind, and researchers like Jeffrey Hinton), clarified that extinction risk is a global priority.
  • Advocacy carries risks of backlash, polarization, or discrediting the cause if rhetoric is mismanaged.
• Third-Party Auditing and Evaluation:
  • Future regulatory frameworks will likely require external auditors to verify compliance due to government lack of technical expertise.
  • The non-profit METER (formerly ArcEvals) has partnered with OpenAI and Anthropic to evaluate model capabilities prior to release.
  • Audits may assess model alignment, compute access, information security practices, and training data usage.
  • This field requires individuals with skills to manage complex relationships between industry and government.
• International Work and Coordination:
  • Improving coordination between the U.S. and China on safe AI development is identified as a particularly impactful career path.
  • The U.K. government announced an AI safety institute in 2023, complementing U.S. efforts.
  • The EU's General Data Protection Regulation (GDPR) demonstrates the potential for the "Brussels effect" to shape global corporate behavior.
  • Future scenarios may involve AI treaties and international bodies similar to the International Atomic Energy Agency or the IPCC.

Career Capital and Entry Strategies

• Early-career professionals should focus on building "career capital" (skills, experience, judgment, networks, credentials) rather than seeking immediate high-impact roles.
• Technical expertise (machine learning, AI hardware, information security) combined with policy knowledge is highly valued.
• Useful educational pathways include degrees in political science, economics, law, and international relations, alongside STEM backgrounds.
• Testing fit can be achieved through policy internships (e.g., Semester in D.C., Virtual Student Federal Service), fellowships, and entry-level research projects.
• A mix of technical and policy credentials is necessary; many successful policy professionals hold STEM degrees like computer science or master's in machine learning.
• Specific fellowships exist to support individuals with STEM backgrounds entering policy careers.
• Communication skills, organizational abilities, and diplomatic experience are also critical forms of transferable capital.

Risks and Pitfalls

• Accidental Harm: Actions may inadvertently exacerbate risks by ratcheting up geopolitical tensions, suppressing beneficial technology, or creating precedents exploitable by dangerous actors.
• Backfire: Advocacy strategies can polarize public opinion or discredit the cause if messages are mistaken or rhetoric is ineffective.
• Burnout: Government and policy roles can be fast-paced, high-stress, and slow to yield tangible progress, leading to career attrition.
• Compensation: Policy roles in government are often less well-paid than comparable positions in the private sector.
• Irreversibility: Practitioners are advised to avoid hard-to-reverse actions and maintain humility due to incomplete information.

Forward-Looking Statements and Resource Allocation

• Despite increased attention on AI, the field remains neglected regarding the most critical risks, maintaining a high priority for impact.
• Increased attention makes interventions potentially more tractable as policymakers become more open to regulation.
• Career trajectories and impactful institutions are expected to shift over time; current recommendations may differ in five years.
• 80,000 Hours offers free one-on-one advice to help individuals assess fit and identify job or funding opportunities.