newsfilter
← Interviews
Dwarkesh Patel18 min

Which AI is the best liar?

Webinar, Other
  • Project Scope & Motivation: The creator built a $20,000 API-cost benchmark using the social deception game Blood on the Clock Tower to evaluate AI capabilities in long-horizon strategizing, multi-agent coordination, and creative bluffing, areas not clearly covered by existing benchmarks.
  • Game Mechanics: The game involves good and evil teams with secret roles; the evil team includes a "Demon" who kills players at night, while the good team votes to execute the Demon during the day.
  • Strategic Complexity: Evil players must maintain consistent lies and coordinate subtle conspiracies, while good players must synthesize fragmented, often misleading evidence to identify the Demon without falling for intentional misinformation.
  • Methodology: 368 games were run where entire teams consisted of identical AI models to isolate performance differences between model families over a period of seven months.
  • Performance Rankings: GPT-6 Astra (released September 2026) is currently the top model, ranking approximately 430 ELO points higher than Claude Opus 4.6 (released February 2026).
  • Win Probability: Due to the ELO gap, GPT-6 Astra is projected to win 92% of games against Claude Opus 4.6.
  • Rapid Improvement: Over the last six months, OpenAI models gained 580 ELO points, rising from trailing Anthropic models in March to significantly outperforming them by September.
  • Operational Divergence: OpenAI models demonstrate superior "good team" performance by treating evidence holistically and skeptically, whereas Anthropic models tend to accept reported information at face value.
  • Reasoning Failure Example (Anthropic): In one scenario, Claude models accepted a player's innocence after a failed "Slayer" shot, failing to realize the evil team had sabotaged the Slayer's ability the previous night.
  • Reasoning Success Example (OpenAI): GPT models correctly inferred the sabotage context, refused to treat the failed shot as conclusive proof of innocence, and subsequently identified and executed the actual Demon.
  • Accuracy Metrics: When privately queried on their confidence regarding the Demon's identity, OpenAI models identified the correct target faster and with higher accuracy than Anthropic models.
  • Team Synergy Hypothesis: A hybrid team consisting of 50% Opus 5.5 and 50% GPT 5.6 Sol was tested in 130 games but performed slightly worse than homogeneous teams, though the result was not statistically significant.
  • AI-Driven Environment Design: The benchmark harness was developed using AI agents to iterate on game rules and probability tables over 20+ revisions, replacing the human "Storyteller" role to eliminate human bias.
  • Future Benchmark Viability: Unlike static benchmarks, this dynamic evaluation is unlikely to saturate, as the difficulty bar rises automatically with the emergence of smarter models.
  • Open Source & Expansion: The code is publicly available at clocktowerbench.com, where users can also play against AIs or review game logs; the creator invites extensions to include other game scripts or explore learning across repeated games.