Can AI make society and government smarter? (article by Zershaaneh Qureshi)

Context & Stakes:

• AGI deployment could compress a century of progress into a single decade, drastically increasing the stakes and reducing the time available for critical decision-making.
• Human institutions currently suffer from systematic failures, including ignoring climate warnings, ineffective pandemic responses, and engaging in regrettable wars due to poor coordination or fact misinterpretation.
• Future AI systems may drive economic progress, develop weapons autonomously, and create geopolitical tensions that escalate into conflict without human-aligned oversight.

Proposed Solution: AI Decision-Making Tools:

• The article categorizes promising AI applications into two distinct types designed to correct specific human decision-making flaws.
  • Epistemic Tools: Designed to improve understanding of truth and predictability.
    • AI Fact-Checkers: Provide more reliable and impartial evaluation of information to counter misinformation.
    • Forecasting Systems: Enable institutions to model policy effects and predict world events with higher accuracy.
    • Moral Progress Tools: Assist in reasoning through complex ethical questions to foster societal agreement.
  • Coordination Tools: Designed to facilitate collective action among groups with competing interests.
    • AI Negotiation Agents: Simulate thousands of hours of negotiation to identify mutually beneficial agreements missed by humans.
    • Verification Systems: Offer impartial monitoring of compliance to overcome trust barriers in cooperation.
    • Structured Transparency: Enable controlled information sharing to detect threats (e.g., weaponization) without broad privacy invasions.

Strategic Argument: Differential Technology Development:

• Opportunity for Speed: Current commercial incentives focus on broadly capable agents, leaving a gap in specific decision-making tools that could be filled by a small, focused group.
• Impact of Acceleration: Deploying safety-promoting tools just months ahead of dangerous AI capabilities could allow for critical safety commitments to be established before catastrophic risks materialize.
• Low-Hanging Fruit: Very few projects currently target this specific niche compared to the billions invested in general AI development.

Counterarguments & Risks:

• Default Development: Market forces may eventually build these tools anyway, potentially reducing the counterfactual impact of new entrants; however, speed-ups and targeting under-incentivized gaps (e.g., ethical reasoning over financial trading) remain viable.
• Accelerating AGI Risks: Developing advanced AI capabilities might inadvertently speed up AGI arrival or enhance dangerous planning skills, though the article argues this risk is manageable by targeting lower-risk applications like fact-checking.
• Misuse Potential: Malicious actors could use these tools to gain strategic advantages or execute power grabs; mitigating this requires making tools widely accessible rather than restricting them to powerful elites.
• Hypothetical Assumption: The strategy assumes actors are generally well-intentioned; while most harmful decisions stem from errors rather than malice, the risk of deliberate misuse must be factored into project design.

Recommendations for Practitioners:

• Target Audience: The authors estimate that 200–300 "thoughtful and entrepreneurial" individuals are needed to advance this field effectively without overwhelming the area.
• Ideal Candidate Profile: Success requires high tolerance for ambiguity, an entrepreneurial mindset, and strong judgment in prioritizing projects that are market-disincentivized yet low-risk.
• Implementation Strategies:
  • Direct Development: Join or found organizations building these tools, focusing on engineering, product design, and user interface for decision-makers.
  • Support Infrastructure: Create benchmarks for epistemic capabilities, curate training data (e.g., past forecasting failures), or build databases for resource sharing.
  • Integration: Work within key institutions (government, international bodies) to educate stakeholders and integrate AI tools into existing decision processes.
  • Preparation: Found non-harmful companies to build foundational skills or gain expertise in high-impact domains like diplomacy and forecasting.

Forward-Looking Statements:

• The field is described as "not mature," with significant uncertainty regarding which specific work will yield the most useful results.
• The authors note that other researchers, such as those behind AI Tools for Existential Security, may hold more optimistic views regarding the capacity for many people to work in this space.
• Practitioners are advised to remain ready to pivot as the landscape of AGI challenges evolves.