Panel
Fallible: The Risk of Predictions and the Limits of Expertise
- Experts in complex systems are expected to only place bounds on surprise rather than predict precise outcomes, as small nonlinear changes can yield significant outputs or vice versa, with statistical regularities potentially explaining only about 30% of variation while the remaining 70% represents unpredictable, important factors.
- Long-term projections by entities like the Congressional Budget Office regarding 10- to 20-year spending effects are anticipated to be inaccurate due to the fluid nature of human societies, and replicating accurate predictions after a spectacular event is viewed as incredibly rare because complex systems are non-gameable.
- Future analysis of fluid systems is predicted to rely more on probabilities, humility, and intuitive "feel" rather than precise modeling, contrasting with the expectation that the deep human need to make sense of the world persists despite limited or "garbage in" inputs for models and stories.
- Journalists may frequently remain "completely off base" in their storytelling because clarity is often accidental and dependent on identifying the right informants, such as company assistants rather than executives, to clarify topics.
- Financial market analysis suggests that while short-term "voting" dominates immediate discussions, fundamentals will rule over longer periods, though many investors cannot survive the path to correctness even when ultimate outcomes are right, citing failures during 1999–2000 and prior to the global financial crisis.
- Regulators are expected to continue solving for yesterday's problems rather than addressing emerging threats, leading to predictions that the financial crisis is not the last of its kind.
- The desire for numerical specificity in predictions, such as President Obama's 2009 promise to create 3.5 million jobs from a $787 billion spend, is expected to create massive problems compared to stating directionality, driven by a high currency paid for certainty in business and media that discourages acknowledging complexity.
- The half-life of knowledge varies by field, with a long time anticipated before new discoveries percolate between siloed fields, potentially leading to the use of outdated information in critical areas like medicine where lives are on the line.
- Computational techniques are predicted to eventually bridge gaps in literature to find "undiscovered public knowledge," though this remains difficult currently due to the sheer size of existing literature, while academic models often lack accountability because internal consistency does not require correctness.
- Using big data in academia is expected to be problematic as it prioritizes measuring what is capable of being measured rather than what is desired, potentially introducing biased datasets.
- Evaluating experts is expected to require analyzing credentials, failure rates, and reward structures to understand incentive structures, while finding bias on all sides of an argument is viewed as a useful strategy since no one is entirely unbiased.
- Ideologies are predicted to become more entrenched as it is easier for individuals to find like-minded people now, creating self-reinforcing belief systems that contradict the human DNA-embedded desire for certainty about future outcomes.
- Prediction markets are expected to be ineffective for complicated system outcomes like stock markets, which are better modeled by factors against assets to understand interactions, but remain effective for election outcomes where group decisions determine the result.
- Cynicism about experts is expected to increase as valuable non-consensus views are increasingly kept secret by hackers and financial professionals who no longer wish to be known, contrasting with the view that hard-won expertise in specific fields like mathematics or carpentry remains awe-inspiring.
- Science as an enterprise is expected to move closer to truth even if individual experts are often wrong, and it is predicted to be harder to escape information about wrongness now than previously due to markets and systems signaling errors more quickly.
- No one is expected to ever possess true expertise regarding future outcomes in complicated systems, despite remaining in awe of specific hard-won skills, leading to the expectation that deep human needs for certainty will continue to clash with reality.