Interview, Fireside Chat
Francois Chollet — Why the biggest AI models can't solve simple puzzles
- François Chollet defines the ARC (Abstraction and Reasoning Corpus) benchmark as an "IQ test for machine intelligence" designed specifically to resist memorization and measure the ability to synthesize novel programs from minimal examples.
- Chollet argues that current Large Language Models (LLMs) function as "big interpolative databases" or "interpolative memories" that excel at memorizing static programs and retrieving pre-existing solution templates rather than performing true on-the-fly reasoning.
- The primary distinction Chollet draws between intelligence and skill is that intelligence is the capacity to adapt to novel, unseen situations efficiently, whereas scaled models only improve their "skill" (performance on memorization-based benchmarks) without increasing general intelligence.
- Human children possess "core knowledge" (basic physics, object permanence, counting, geometry) which allows them to solve ARC puzzles immediately, whereas LLMs struggle because each puzzle is novel and requires program synthesis, not pattern matching.
- Jack Chole's approach achieved 35% on ARC by fine-tuning a 240-million-parameter model on millions of generated ARC-like tasks and utilizing "test-time fine-tuning" (active inference) to adapt the model for each specific puzzle, a capability absent in standard static LLM inference.
- Chollet posits that while larger models may achieve higher performance on memorization-heavy benchmarks (e.g., GSM-8k), they will fail ARC because the benchmark space is effectively infinite and the world is non-stationary, making brute-force pre-training impossible.
- OpenAI is criticized for setting back progress toward AGI by approximately 5 to 10 years, primarily due to the "closing down of frontier research publishing" and LLMs "sucking the oxygen out of the room," which has discouraged the exploration of alternative AI paradigms.
- Chollet proposes a future architecture that merges deep learning (System 1, intuition/memory) with discrete program search (System 2, reasoning), where LLMs guide the search in program space to avoid combinatorial explosion.
- Mike Knouf, co-founder of Zapier, launched a $1 million prize in collaboration with Chollet to incentivize the development of AGI-capable systems, noting that few experts were aware of the ARC benchmark until recently.
- The ARC Prize offers a $500,000 reward to the first team that achieves an 85% benchmark score on the private test set, representing the average human performance level.
- A $100,000 "progress prize" is available this year ($50,000 for top scores, $50,000 for the best conceptual paper), with the plan to run the contest annually until the 85% goal is met.
- To ensure transparency and prevent "secret sauce" monopolies, winners must release their solutions or papers into the public domain to allow the community to reproduce and iterate on the results.
- The competition restricts submissions to open-source models running on limited hardware (NVIDIA Tesla T4 or P100) with a 12-hour runtime limit to prevent trivial brute-force scaling, though a parallel private track allows unlimited resources for testing scaling hypotheses.
- Chollet distinguishes between "cheating" (brute-forcing millions of synthetic tasks to create overlap with the test set) and legitimate progress (using code interpreters or test-time fine-tuning to synthesize programs on the fly).
- An ARC 2 version is planned for release later this year to address redundancy in the current task set and improve resistance to training data contamination.
- The prize aims to verify whether new ideas are required to solve ARC, as current trends suggest that adding more compute to existing LLM architectures has reached a plateau on tasks requiring genuine generalization.
- If a multimodal model can solve 85% of ARC tasks while being trained only on core knowledge and without explicit exposure to ARC-like tasks during pre-training, Chollet considers it a significant milestone on the path to AGI.