Interview, Fireside Chat
Vladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence | Lex Fridman Podcast #71
- Vapnik distinguishes between engineering (imitating human behavior to create useful devices) and the science of understanding (deciphering the "vault of pure ideas" that constitute intelligence).
- He proposes that human intelligence relies on a small set of universal "predicates" (abstract functions or units) rather than a vast array of specific rules.
- Drawing parallels to Vladimir Propp's analysis of Russian folklore, Vapnik suggests that just as 31 structural units can explain narrative progression, a small number of mathematical predicates could explain complex visual recognition.
- In the context of digit recognition, Vapnik identifies "symmetry" (including vertical, diagonal, and anti-symmetry) and "structuring" as key predicates for defining invariance.
- Vapnik defines a "predicate" formally as a function that acts as an inner product, measuring specific integral properties of data rather than point-wise values.
- He contrasts "strong convergence" (convergence of a function everywhere) with "weak convergence" (convergence of the function's properties/inner products), arguing that weak convergence is the more powerful tool for defining "admissible sets of functions."
- The goal of intelligent learning, per Vapnik, is to reduce the VC dimension (capacity) of the hypothesis space by selecting functions that satisfy specific invariants derived from weak convergence.
- Vapnik challenges the field to achieve state-of-the-art MNIST digit recognition results using approximately 60 examples per class (roughly 100 times fewer than the standard 60,000), arguing that current deep learning methods are overly dependent on data volume.
- He asserts that neural networks, particularly convolutional networks, are essentially engineering implementations of specific invariants (like translation invariance) rather than a fundamental scientific explanation of intelligence.
- Vapnik believes that discovering "good predicates" is a human-centric endeavor that cannot be fully automated by machines due to the infinite sea of possible functions in Hilbert space.
- He suggests a methodology for discovering new predicates by identifying contradictions where current invariants fail to explain data, similar to hypothesis testing in physics.
- Vapnik argues that logic-based symbolic AI is insufficient because it lacks the necessary connection to "reality" and common sense that allows humans to select useful predicates.
- He posits that solving the handwritten digit challenge would immediately provide insights applicable to broader 2D image understanding, as the underlying principles of invariance are universal to visual data.
- Vapnik expresses skepticism regarding the necessity of recurrence or sequential reasoning in machine learning for classification tasks, maintaining that a well-constructed admissible set of functions can solve problems without memory loops.
- He views music critics' descriptions of Bach and Chopin as a source of "privileged information" that could help formalize the abstract concepts (predicates) needed for visual recognition.
- Vapnik advocates for moving away from heuristics and empirical risk minimization toward closed-form solutions derived from the mathematical theory of weak convergence.
- He emphasizes that "understanding" requires knowing the abstract ideas (forms) that project onto reality, rather than merely processing the data (shadows) directly.
- Vapnik notes that while engineering (like self-driving cars) does not require a deep understanding of the underlying science, true intelligence demands the discovery of the underlying laws of invariance.
- He mentions a specific collaboration involving "learning with privileged information," where poetic descriptions of digits were used as additional data to significantly improve recognition performance.
- Vapnik cites "uniform convergence" as a critical concept in statistical learning theory, distinguishing it from the law of large numbers to ensure generalization across all functions in a set.
- He believes that the path to general artificial intelligence lies in identifying a small set of universal predicates that can be applied across different domains, from literature and music to vision.
- Vapnik advises that solving a specific, simple problem (like digit recognition) is a better path to understanding than attempting to solve a more general problem as an intermediate step.