Feb 25, 2020, 12:30 PM ETInterview, Fireside Chat

What is Statistics? (Michael I. Jordan) | AI Podcast Clips

Statistics is defined as a hybrid discipline between mathematics, science, and technology, providing principles for making reasoned inferences and minimizing decision errors.
The field's formal history spans approximately 250 years, originally termed "inverse probability" to address problems where observed outcomes are used to infer unknown underlying mechanisms (the inverse of predicting outcomes from known mechanisms).
The modern term "statistics" was adopted by Pierre-Simon Laplace, who analyzed French census data to inform government policy, deriving the name from the French word état (state).
Formalization accelerated in the 1930s alongside the concurrent development of game theory and decision theory by figures such as Von Neumann and Abraham Wald.
Advanced statistical curricula currently prioritize decision theory as the foundational framework, which later branches into Bayesian and frequentist methodologies.

The field exhibits a fundamental duality akin to wave-particle duality in quantum physics, with Bayesian and frequentist approaches offering distinct, sometimes conflicting, interpretations of the same problems.
Frequentist Approach:
- Treats data ($X$) as random and the parameter ($\theta$) as fixed, averaging the loss function over all possible data sets to establish a "risk" metric.
- Provides guarantees valid across infinite repetitions, making it the preferred standard for software deployment where robustness against unseen data distributions is critical (e.g., ensuring a procedure is correct 95% of the time).
Bayesian Approach:
- Treats the parameter ($\theta$) as the random variable, incorporating prior human expertise and subjective probability distributions to condition on the specific observed data ($X$).
- Focuses on the specific instance of data available rather than hypothetical repetitions, aligning well with domain-specific scientific inference.
Empirical Bayes:
- A hybrid methodology that utilizes the Bayesian framework but estimates unknown prior parameters directly from the data using plug-in estimates.
- Balances the philosophical rigor of Bayesian reasoning with the practical need to estimate unknown quantities in real-world scenarios.

James-Stein Estimation: A counterintuitive statistical finding noted for its paradoxical nature, which challenges standard intuitions about estimation efficiency and requires specialized perspective to understand.
False Discovery Rate (FDR):
- A metric developed around 1960 (associated with Herbert Robbins) designed for multiple hypothesis testing, measuring the proportion of false discoveries among all declared discoveries.
- Unlike traditional frequentist metrics (accuracy, precision, recall) which condition on the truth of the hypothesis, FDR conditions on the data (the set of discoveries made).
- The concept has been advanced through work by Brad Efron, Ben Yekutieli, and John Storey, who provided Bayesian interpretations of the framework.
Decision Theory Framework: The loss function, defined by data ($X$) and parameters ($\theta$), serves as the common ground for both schools; the divergence arises in whether one averages over $X$ (frequentist risk) or $\theta$ (Bayesian expected loss).

The speaker advocates for a synthesis of the two paradigms, arguing that while Bayesian methods incorporate expert knowledge effectively, frequentist guarantees remain essential for ensuring reliability in deployed systems.
Future statistical practice should view these approaches not as mutually exclusive but as complementary forces that must "fight and blend" to address complex decision-making problems.