Noam Brown: AI vs Humans in Poker and Games of Strategic Negotiation | Lex Fridman Podcast #344

Libratus (Heads-up Poker, 2017):

• A bot that approximates Nash equilibrium and defeated four top human players in heads-up No-Limit Texas Hold'em.
• The bot did not attempt to exploit human psychology or perform "mind games"; it relied solely on game-theoretic optimality.
• Over 120,000 hands, the bot won approximately $2 million in simulated prize money.
• The victory solidified the dominance of Game Theory Optimal (GTO) play over exploitative strategies in high-stakes poker.
• Humans initially lost by a wide margin in a 2015 competition due to the bot's lack of real-time search capabilities.
• Key improvements leading to the 2017 win included the introduction of real-time search (planning moves during play) rather than relying solely on pre-computed strategies.
• The bot utilized Counterfactual Regret Minimization (CFR) to converge on an optimal strategy without needing neural networks.
• The game state space for heads-up Hold'em is estimated at $10^{161}$ decision points, far exceeding the number of atoms in the universe.
• Libratus introduced "over-betting" (betting significantly larger than the pot), a strategy later adopted by human professionals to put opponents in difficult spots.

Pluribus (Multi-player Poker, 2019):

• Achieved superhuman performance in six-player No-Limit Texas Hold'em, a more complex, non-zero-sum environment.
• The final training cost was under $150 on AWS, a drastic reduction from Libratus's estimated $100,000 training cost.
• Performance gains were driven by algorithmic efficiency, specifically "depth-limited search," rather than just increased compute power.
• Techniques developed for two-player games were shown to work in practice for multi-player poker due to the game's inherently adversarial nature.

Cicero (Diplomacy, 2024):

• An AI system that achieved human-level performance in the board game Diplomacy, which involves seven players and unstructured natural language negotiation.
• The bot ranked second out of ~80 players in a tournament of 40 games against humans, surpassing expert-level human performance.
• Unlike previous game AIs, Cicero could not rely purely on self-play; it required human data to understand social conventions and trust dynamics.
• The system combines a strategic reasoning model (using reinforcement learning and planning) with a controllable large language model (LLM).
• It generates dialogue based on specific "intents" (strategic goals) rather than imitating human speech patterns directly.
• The bot learned that minimizing lies was strategically superior, as trust is the core currency for winning the game.
• The project highlights that in cooperative games, AI must model human irrationality and emotional reactions to succeed.
• All code, models, and a dataset of 50,000 human games (10 million messages) have been open-sourced for research.
• The research demonstrates that conditioning language on strategic intents allows AI to generate human-compatible negotiation tactics.

Core Technical Insights & Trends:

• Search vs. Neural Networks: Neural networks alone (policy/value functions) are insufficient for superhuman play; real-time search (planning) is the critical differentiator.
  • Removing search from AlphaZero reduces its ELO rating from ~5,200 to ~3,000 (approximate human grandmaster level).
  • Human players also utilize search, but it is a sequential, language-based planning process rather than Monte Carlo tree search.
• Game Theory Application:
  • Nash equilibrium guarantees no loss in expectation in two-player, zero-sum games (perfect information or imperfect like poker).
  • In non-zero-sum games (like Diplomacy), Nash equilibrium does not guarantee a win, and self-play from scratch often fails without human alignment.
• Human-Like AI:
  • Techniques developed for Diplomacy are being applied to chess and Go to create strong, yet human-like, opponents for training purposes.
  • AI can be tuned to mimic specific human player styles (e.g., a Magnus Carlsen-style chess bot) to aid human preparation.
  • Creating human-like bots poses challenges for cheat detection, as moves become harder to distinguish from top-tier human play.

Future Directions & Ethical Considerations:

• Data Efficiency: Current AI requires massive datasets compared to humans; improving data efficiency is identified as the next major hurdle for AGI.
• Real-World Application: Diplomacy represents a significant step toward applying AI to real-world negotiation, where natural language and cooperation are central.
• Trust and Deception: The research forces an examination of AI lying; while effective in games, deploying deceptive AIs in consumer products raises ethical questions.
• Bias and Social Dynamics: AI systems face inherent human anti-AI bias; humans in Diplomacy tend to form coalitions to eliminate the perceived AI threat.
• Long-term Safety: The difficulty of defining "optimal" reward functions (e.g., "live a good life") is highlighted as a central challenge in aligning AI with human values.
• Advice for Researchers: Beginners are encouraged to build strong foundations in math and statistics but should also pursue diverse, non-standard backgrounds to bring unique perspectives to complex problems.

Game-Specific Nuances:

• No-Limit vs. Limit Poker: No-Limit allows for extreme bet sizes that induce psychological "jumpiness" and risk aversion in humans, whereas Limit poker rewards more calculated strategy.
• Diplomacy Structure: The game is set in pre-WWI Europe; France is statistically the strongest starting power, but the game is self-balancing as others cooperate to stop the leading power.
• Negotiation Mechanics: Moves are simultaneous, and promises are binding only until the next turn; the core tension is between forming alliances and the inevitability of backstabbing.