Vladimir Vapnik: Statistical Learning | Lex Fridman Podcast #5

Philosophical Foundations of Learning

• Instrumentalism vs. Realism: Vapnik distinguishes between two scientific approaches:
  • Instrumentalism: Creating theories solely for prediction (e.g., finding a classification rule) without claiming the model represents absolute truth.
  • Realism: Attempting to understand the fundamental nature of reality (e.g., learning conditional probability to understand "how God plays dice").
• Role of Mathematics:
  • Vapnik views mathematics not merely as a tool but as a "language used by God" that reveals simple, underlying principles of reality often hidden from human intuition.
  • Human intuition is described as "primitive" and unable to see simple mathematical truths without rigorous derivation from axioms.
  • He dismisses "imagination" in machine learning (e.g., deep learning architectures) as fantasy or incorrect interpretation rather than a source of insight.
• Limits of Knowledge:
  • The nature of the "great teacher" who generates effective predicates remains an open problem; while they can introduce invariants to reduce observation requirements by 100x, the mechanism of this selection is unknown.

Critique of Current Machine Learning Paradigms

• Critique of Deep Learning:
  • Vapnik characterizes deep learning as "fantasy" and "interpretation" rather than rigorous mathematical science, noting that mathematics contains no concept of "neurons" or "depth."
  • He argues that the success of systems like AlphaGo does not validate deep learning's theoretical optimality; rather, it proves the problem (Go) is not as difficult as previously thought.
  • He asserts that deep learning often fails to create a proper "admissible set of functions" (small VC dimension), necessitating "zillions" of training data instead of a theoretically sound approach.
• Theoretical Superiority of Invariants:
  • Vapnik proposes that optimal learning relies on two mechanisms: strong convergence (standard statistical learning) and weak convergence (requiring human-provided predicates).
  • He suggests that "shallow networks" (specifically "shadow networks" in representation theory) are theoretically more optimal than deep architectures for solving mathematical learning problems.
• The Data Efficiency Gap:
  • Current systems require vast datasets because they lack "invariants" (predicates) that effectively reduce the space of admissible functions.
  • Vapnik claims that with proper invariants, tasks like digit recognition could be solved with 100x fewer examples than current deep learning methods require.

The Nature of Intelligence and Open Problems

• Defining Intelligence:
  • Intelligence is defined not by the ability to imitate human behavior (Turing's imitation game) but by the ability to generate "predicates" (e.g., "swims like a duck") that carry significant information about reality.
  • The "open problem" of intelligence is determining how a teacher selects specific, informative predicates versus useless ones (e.g., "jump like a duck").
• Separation of Statistical and Intelligent Components:
  • Statistical learning theory can solve problems efficiently if provided with correct invariants, but it cannot generate the invariants itself.
  • Vapnik formulates a specific challenge: solving the MNIST digit recognition problem using only 1/100th of the standard training data by incorporating human-derived invariants.
• Externalization of Intelligence:
  • Vapnik speculates that intelligence may not reside solely within the individual but could be a connection to a broader "world intelligence," evidenced by simultaneous mathematical discoveries by different thinkers (e.g., geometry by Lobachevsky, Gauss, and Bolyai).

Personal Reflections and Historical Context

• VC Theory and Support Vector Machines:
  • Vapnik confirms he recognized the profundity and permanence of Support Vector Machines (SVM) and VC theory immediately upon their development, despite initial skepticism (notably from Dudley at MIT).
  • He views the recent invariance theory as the "ultimate learning story" because it mathematically separates statistical limitations from the problem of intelligence.
• Sources of Truth:
  • Vapnik identifies "ground truth" as the guiding principle in his work, finding resonance in the structural clarity of Bach's music and classical poetry, which he views as empirical evidence of this truth.
  • He emphasizes rigorous self-correction over interpretation, stating that most of his early work was "wrong" until he could align it with ground truth.
• Complexity Theory:
  • He critiques the focus on worst-case complexity (P vs. NP), arguing it is an "edge case" analysis that fails to capture the reality of learning, where average-case performance driven by invariants is more relevant.
  • He notes the mathematical difficulty of applying the "uniform law of large numbers" (requiring small VC dimension) versus the standard law of large numbers.