Fallible: The Risk of Predictions and the Limits of Expertise

Panelists and Expertise:

• Sam Arbisman (Kauffman Foundation): Expert in complexity science, focusing on how scientific/technological change informs the future of knowledge.
• Zachary Carabon (Investnet): Head of global strategy, writes "The Edgy Optimist" for Slate, skeptical of predictive power in economics.
• Bethany McLean (Vanity Fair): Investigative journalist known for exposing business failures (e.g., Enron); emphasizes the limitations of inputs in storytelling and modeling.
• John Rohal (Man Group): Chairman of North American operations; focuses on absolute return products and the difficulty of predicting financial markets.
• Paul Kudrowski (Bloomberg/SK Ventures): Managing partner and contributing editor; highlights the "gentleman amnesia effect."

Core Concept: The "Gentleman Amnesia Effect":

• Coined by Michael Crichton, describing the tendency for individuals to trust an expert on a new topic immediately after recalling an expert's previous error on a familiar topic.
• Demonstrates a fundamental trust in the "word of experts" that overrides direct first-hand experience of their fallibility.

Complex Systems and Predictive Limits:

• Tightly Coupled Systems: Highly interconnected (like crystals) allow for accurate prediction.
• Loosely Coupled Systems: Independent parts (like a gas) allow for accurate statistical aggregation despite individual unpredictability.
• Intermediate Complexity (The "Sweet Spot"): Systems with nonlinear interactions (small changes yield large outputs) are the hardest to predict; social and biological systems fall here.
• Expert Role Redefined: In complex systems, experts should place "bounds on surprise" rather than making precise predictions.
• Statistical Reality: Large statistical regularities may only explain 30% of variation, leaving 70% of important outcomes unpredictable.

Failures in Economics and Finance:

• Economic Projections: The Congressional Budget Office (CBO) 10-20 year projections are flawed because they assume static variables ("all things being equal") in a fluid reality where "nothing is ever static."
• Historical Analogy: Historians often repeat each other looking for patterns; history does not repeat itself, though it may "rhyme."
• Binary vs. Fluid: Societal and economic outcomes are fluid, not binary, making precise prediction "non-gameable."
• Market Reality: Financial markets act as a "falsification engine," constantly testing and refuting predictions through price movements.
• Survival Mechanism: Investors often bet negatively (e.g., against liquidity) but fail to "survive the path" (e.g., Julian Robertson in 1999/2000) due to short-term volatility before long-term theses play out.

Journalism and Input Quality:

• Garbage In, Garbage Out: Both financial modeling and journalism rely on inputs; accuracy is limited by the availability of truthful, diverse perspectives.
• Expert Bias: Experts often deny being wrong because feeling wrong feels identical to feeling right until outcomes prove otherwise.
• Scaffolding of Truth: Bethany McLean argues that transparency regarding the "facts" used as a scaffold for narratives is more valuable than the conclusion itself.
• Enron Case Study: Even sophisticated insiders at Enron lacked a holistic understanding of the fraud, realizing pieces were nonsensical but unable to see the whole picture.

Information Silos and "Undiscovered Public Knowledge":

• Don Swanson's Discovery: Information scientist Don Swanson found that combining disparate literature (e.g., A implies B, B implies C) could reveal hidden truths (A implies C).
• Example: Swanson discovered fish oil treats circulatory disorders by linking unrelated medical papers that never crossed paths.
• Outdated Data: Large literature sizes lead to outdated information being applied to new fields (e.g., medical errors due to wrong population counts).
• Reproducibility Crisis: Scientific methods sections are often now "advertisements" rather than transparent guides; a movement exists to make code and data open for verification.

Hacking Expertise: How to Evaluate Predictors:

• John Rohal's Framework: Evaluate experts by:
  • Credibility: How were they credentialed?
  • Failure Rate: How have they failed previously?
  • Incentives: What is their reward system (financial vs. psychic vs. academic)?
• Bethany McLean's Approach:
  • Seek bias on all sides of an argument, including contrarian voices (e.g., short sellers).
  • Prioritize volume of inputs ("more, more, more") to avoid consensus traps.
  • Scrutinize the basis of generalizations (e.g., the "7.2% unemployment rule" for re-elections was statistically insignificant despite being repeated as fact).
• Sam Arbisman's Warning: Academia lacks accountability for incorrect predictions; models can be internally consistent yet factually wrong.
• The "Big Data" Trap: More data is not always better; "medium-sized good data" is preferable to "big bad data," which often reinforces bias.

Cultural and Structural Drivers of Fallibility:

• Certainty Premium: Media and markets disproportionately reward binary certainty over nuanced, probabilistic reasoning (e.g., Obama's specific "3.5 million jobs" promise).
• Regulatory Lag: Regulations (Sarbanes-Oxley, Dodd-Frank) often solve yesterday's problems while failing to anticipate future system evolutions.
• Ideological Enfranchisement: Specialization and the internet have made it easier for people to find "like-minded" experts, reinforcing confirmation bias (e.g., Enron's internal silos).
• Taleb's Antifragility: Accepting failure and building redundancy is often superior to trying to predict every outcome in complex systems.

Behavioral Finance and Human Nature:

• Theoretical vs. Real: Behavioral studies (e.g., Trolley Problem) often fail to predict actual human behavior under pressure, as people react differently to real crises than hypothetical ones.
• Measurement Challenges: There is no agreed-upon metric or time cycle to measure the development of "behavioral skills" in investors.
• Human Desire for Order: Humans crave certainty in a fluid world, leading to the acceptance of false precision in political and economic promises.

Prediction Markets:

• Effectiveness: Prediction markets work well for discrete, short-term outcomes determined by group consensus (e.g., elections).
• Limitations: They are ineffective for complex system outcomes where the crowd is not the determinant of the result (e.g., stock market fundamentals).

Closing Stances on Expertise:

• Paul Kudrowski: More cynical; experts often hide their non-consensus correct views (e.g., hackers, traders).
• Bethany McLean: Consistently cynical; early career experience with stock recommendations going wrong established a baseline of skepticism.
• Zachary Carabon: Distinguishes between "hard-won expertise" (e.g., craftsmanship, math) which is admirable, and "expertise" in complex systems prediction, which is often misleading.
• Sam Arbisman: Hopeful; views the scientific enterprise as a framework for continuous correction, where individual errors drive collective progress.