Interview
Clay Graubard and Robert de Neufville on forecasting the war in Ukraine (2022)
- Episode Context: Recorded on April 27, 2022, this episode features super forecasters Robert de Nuffel and Clay Graubart analyzing the Russian invasion of Ukraine and the broader state of the forecasting community.
- Forecasting Renaissance: The field is experiencing a resurgence driven by increased funding interest, particularly from the FTX Foundation's donation initiatives, which has led to a scramble for grants to support full-time forecasting roles.
- Startup Activity: Analysis by Global Guessing indicates a peak in forecasting and prediction market startups between 2016–2017, followed by a lull, with a new surge of companies emerging recently focused on geopolitics and prediction markets.
- Robert's Work: Robert de Nuffel recently secured a grant from the Effective Altruism Infrastructure Fund to write "Telling the Future" on Substack, focusing on forecasting research and the Ukraine war.
- Clay's Work: Clay Graubart is a master's student at Oxford studying geopolitical forecasting and co-founder of baserate.io, which operates Global Guessing (aggregating forecasts on Ukraine, JCPOA) and CrowdMoney (prediction markets).
- Super Forecaster Definition: The term "super forecaster" is a trademarked credential from Good Judgment Open, typically awarded to individuals consistently performing in the top 2% of forecasting pools.
- Methodology Differences: Robert notes that many platforms rely on low-cost volunteer participation for "wisdom of the crowds" effects, whereas Clay advocates for deep-dive analysis, spending 8–9 hours daily on single complex questions to refine probability estimates.
- Pre-Invasion Predictions (January): Robert initially underestimated the invasion risk, assuming it was strategically irrational for Putin and citing "mirror imaging" of Western logic; he only shifted his view in February as Russian military movements escalated.
- Pre-Invasion Predictions (February): Clay assigned a 62% probability to invasion on Jan 1st, which rose to 88% by February, 90%+ by Feb 12, and hit 99% by Feb 20/21 before the Feb 24 invasion.
- Forecasting Errors: Robert attributes his early failure to a lack of granular domain theory regarding Putin's specific incentives and a failure to account for the possibility of irrational state behavior.
- Military Performance Expectations: Most forecasters, including Robert, overestimated Russian military dominance; they underestimated Ukrainian resilience, morale, and the extent of Western intelligence and logistical support.
- Western Intelligence Role: Clay highlights that US intelligence provided real-time targeting data to Ukraine from day one, a factor often overlooked in pre-war assessments that significantly bolstered Ukrainian defense capabilities.
- Russian Military Assessment: While the Russian military failed to take Kyiv, Clay notes they have performed better in the south and east, where their artillery-heavy doctrine and logistics are better suited for the terrain.
- NATO Response: Both guests were surprised by the cohesion and speed of the NATO response, which they view as a strategic blunder by Putin that revitalized the alliance rather than fracturing it.
- Forecast Confidence Risks: Robert cautions that assigning 96–99% probabilities to complex geopolitical events is dangerous due to "unknown unknowns," noting his own past overconfidence regarding COVID variants.
- Information Sources (Pre-War): Clay relied heavily on curated Twitter lists to ingest a high volume of open-source intelligence (OSINT) and diverse viewpoints, allowing for rapid signal-to-noise filtering.
- Information Sources (Post-War): Robert and Clay agree that post-invasion Twitter is less useful for forecasting as Ukraine dominates the information narrative, whereas pre-invasion data was more balanced regarding Russian troop movements.
- Nuclear Risk Estimates (Early Phase): Initially, the group estimated a ~4% probability of tactical nuclear weapon use within the first few months of the conflict; Robert now considers this figure likely too high given Putin's constraints.
- Current Nuclear Risk Estimates: Clay currently estimates an 8–12% probability of tactical nuclear use before a peace deal or the fall of Kyiv, driven by the risk of Russian desperation or accidental escalation.
- Escalation Ladders: Forecasts suggest the most likely path to nuclear use is a limited tactical strike within Ukraine to "shake up" the conflict, rather than immediate strategic strikes against NATO cities.
- Signal and Response Risks: Robert warns that ambiguous Western responses to red lines (e.g., varying levels of aid) increase the risk of miscalculation, as Putin may not trust US signaling consistency.
- Background Nuclear Risk: Clay notes a persistent background risk of nuclear strikes on major cities globally, independent of the Ukraine conflict, which slightly elevates the baseline probability of events like a strike on London.
- Strategic Flashpoints: Clay identifies China-Taiwan, Iran-Israel, and Transnistria as critical secondary flashpoints where the Ukraine conflict could trigger broader NATO-Russia confrontations.
- China's Role: Robert believes China likely signed off on the Ukraine invasion but views a full-scale Chinese invasion of Taiwan as less likely in the immediate future due to the global fallout and China's "salami-slicing" strategy.
- Domain Expertise Debate: The guests agree that combining domain experts (e.g., political scientists) with generalist super forecasters creates superior outcomes; experts provide context and sanity checks, while forecasters provide probabilistic discipline.
- Future Forecasting Trends: Robert predicts the post-2025 world will be significantly more multipolar and authoritarian, with inflation and geopolitical instability pressuring democratic institutions globally.
- Learning Strategy: New forecasters are advised to read Superforecasting by Tetlock and Panter, then engage in practice by either doing high-volume predictions or deep-dive pre-mortem analyses on select topics.
- Collaborative Forecasting: Both guests emphasize that forecasting with partners increases accuracy by challenging individual biases and uncovering overlooked variables, despite the risk of becoming socially "obsessive."