Judea Pearl: Correlation and Causation | AI Podcast Clips

Correlation, Causation, and Conditional Probability

• Correlation is defined as the phenomenon where two or more variables vary together over a long period, often driven by an underlying, unobserved causal mechanism.
• While human intuition assumes correlation implies causation, the speaker argues that statistics alone cannot prove causality without external logic.
• Conditional probability differs from causation by representing how variables vary when a third variable is held constant, a choice often made by the experimenter rather than a natural law.
  • Conditioning on a third variable can artificially create or destroy correlations between variables that are physically independent.
  • The "Simpson's Paradox" illustrates this, where selecting specific subsets of data reverses or creates trends not present in the aggregate.
• A major flaw in scientific inference occurs when researchers impose causal logic on observational correlations without rigorous controls.
  • The speaker identifies psychology and fields relying on Structural Equation Modeling (SEM) as disciplines prone to making unjustified leaps from correlation to causation.
  • Applied psychology studies in semi-autonomous vehicles face similar challenges due to the inability to control all environmental variables.

Case Study: Semi-Autonomous Vehicle Research

• Research questions regarding whether drivers fall asleep more frequently in autonomous vehicles require observational studies because randomized controlled trials (turning the system on/off) are unethical on public roads.
• In these observational studies, "uncontrolled" variables introduce confounding factors; for example, drivers may disable autonomous features specifically when they are already fatigued.
• These uncontrolled confounders make it difficult to infer whether the autonomous system causes sleepiness or if pre-existing fatigue drives both the usage of the system and the act of sleeping.

Historical and Mathematical Context of Causality

• The debate over inferring causality from observational data is described as being at least 2,000 years old, citing the biblical story of Daniel and the Babylonian king.
  • Daniel proposed an experiment to test if a vegetarian diet improved mental performance compared to the king's meat-based diet.
  • This historical account is characterized as an early, albeit simple, instance of a controlled experiment.
• While the concept of discovering causes dates to ancient thinkers like Democritus, the mathematical formalization to distinguish cause from effect was not developed until the 1920s.
• A critical limitation of current mathematical science is that most physical equations are symmetrical (algebraic equalities), meaning they do not inherently distinguish between "X causes Y" and "Y causes X."
• The speaker asserts that science has failed to provide a robust mathematical framework to capture the asymmetry of causation, leaving researchers to rely on intuition or imperfect observational logic.