François Chollet: Measures of Intelligence | Lex Fridman Podcast #120

François Chollet defines intelligence not as a collection of skills, but as the efficiency with which an agent acquires new skills for tasks it has not previously encountered, prepared for, or anticipated.

• He distinguishes between the process of intelligence (the agent's ability to adapt and generalize) and the output (a static program or skill artifact).
• Citing Einstein, Chollet notes that the measure of intelligence is the ability to change or adapt to a novel environment.
• Systems that cannot deviate from hard-coded rules or training data distributions are classified as non-intelligent, functioning essentially as lookup tables.

Chollet contrasts two historical views of the mind to contextualize current AI progress:

• The "Static Program" View: Historically held by early AI researchers like Marvin Minsky, this views the mind as a collection of hard-coded, special-purpose mechanisms and logical rules operating over a database, where learning was merely memorization.
• The "Blank Slate" View: Rooted in John Locke's tabula rasa and dominant today via deep learning, this views the mind as an information sponge that absorbs complexity from experience, lacking innate structure.
• Chollet argues that current deep learning models often conflate the output of intelligence (memorized skills) with the process (generalization), failing to distinguish between a system that knows how to play chess and a system that knows how to learn to play chess.

Regarding GPT-3 and large-scale language models, Chollet offers the following assessment:

• He believes GPT-3 likely engages in pattern matching and recognition of tasks from its training data rather than true abstraction or model development.
• Scaling these models (e.g., to 100 trillion parameters) will not resolve fundamental flaws like factual inconsistency, self-contradiction, or lack of reasoning, as the bottleneck is the quality and scope of training data, not model size or compute.
• He predicts that while text generation will become more plausible, these models will struggle with genuinely novel situations that require adapting to "unknown unknowns" rather than interpolating known data distributions.
• Chollet suggests the path forward involves using self-supervised models to generate a "pool of knowledge" and then writing explicit reasoning programs over that latent space to enforce constraints.

Chollet introduces the concept of "Developer-Aware Generalization" as a rigorous metric for intelligence:

• Robustness: The ability to generalize to variations within a known distribution (e.g., handling noise or slight environmental shifts).
• Flexibility: The ability to handle unknowns within a specific domain (e.g., a self-driving car encountering a unique edge case).
• Extreme Generalization: The ability to apply skills across arbitrary domains with minimal training (e.g., a robot trained on cooking quickly learning to garden without explicit reprogramming).
• He posits that true human-level intelligence is defined by the capacity for extreme generalization with high efficiency and low data requirements.

The ARC (Abstraction and Reasoning Corpus) Challenge is presented as a concrete benchmark designed to measure fluid, general intelligence rather than specific skills:

• The test uses grid-based input/output tasks that rely exclusively on four innate "core knowledge" priors identified by psychologist Elizabeth Spelke:
  • Objectness: The intuition that coherent motion defines an object.
  • Agentness: The ability to infer goals and intentions from movement.
  • Basic Geometry/Topology: Innate understanding of space, distance, and navigation.
  • Number: Intuitive counting and quantity comparison.
• Unlike traditional IQ tests, ARC tasks are designed to be novel to both the system and the developers, preventing brute-force memorization of the test set.
• Chollet notes that while humans solve these tasks easily, machine performance has historically hovered near zero, though recent progress has reached approximately 20% on the test set.
• The challenge is intended to force the co-evolution of problem design and AI solutions to clarify the nature of abstraction.

Chollet offers a critical evaluation of existing intelligence testing paradigms:

• The Turing Test: He rejects this as a "cop-out" that outsources the definition of intelligence to biased human judges and incentivizes trickery (anthropomorphism) rather than genuine understanding.
• The Hutter Prize (Compression): He argues that while cognition utilizes compression, it is not synonymous with it. Intelligence requires hedging against future uncertainty, often necessitating the retention of information that seems useless in the past but crucial for future novelty, which contradicts the goal of maximum compression.

On the topic of human-cybernetic integration and intelligence augmentation:

• Chollet is skeptical of neural interfaces (like Neuralink) as a primary solution, arguing that information bandwidth (input/output speed) is not currently the bottleneck for human cognition; rather, the limit lies in processing speed and the brain's internal architecture.
• He views culture, language, and external tools (internet, books) as "externalized cognition," which is the true mechanism for scaling human intelligence beyond biological limits.

Chollet outlines his philosophical view on the meaning of life:

• He defines human existence as a "collection of ripples" in culture, where individuals are composed of concepts and words inherited from the past.
• The meaning of life is found in the creation of new "ripples"—ideas, art, or theories—that propagate into the future, influencing subsequent generations and future AI systems.
• Ethical behavior is framed as the conscious curation of these ripples, choosing to generate positive rather than negative impacts on the collective cultural edifice.