How Intelligent Is AI, Really?

ARC Prize Foundation Core Mission

• Operates as a tech-forward non-profit dedicated to accelerating open progress toward systems that generalize like humans.
• Adopts François Chollet's 2019 definition of intelligence as the ability to learn new things efficiently, rather than relying on static knowledge retrieval (e.g., SAT scores or math problems).
• Focuses on "novelty" and long-horizon learning rather than solving increasingly difficult but static problems (often termed "PhD Plus" problems).

Benchmark Evolution and Performance Data

• Arc AGI v1 (2019): Created by François Chollet with 800 manually designed tasks; established the baseline for measuring generalization.
• Arc AGI v2 (March 2025): Released as a "deeper version" of v1, maintaining a static benchmark format.
• Performance Trajectory:
  • Pre-2024 large language models achieved approximately 4% accuracy (specifically GPT-4 base with no reasoning).
  • Post-integration of reasoning paradigms (e.g., O1), performance jumped to approximately 21% within a short timeframe.
• v3 (Upcoming):
  • Will transition to an interactive format featuring approximately 150 video game-like environments.
  • No Instructions: The benchmark will provide no text, symbols, or English instructions; agents must deduce goals through action and feedback.
  • Human Validation: Environments are excluded if they do not meet a solvability threshold for a minimum of 10 regular humans (e.g., accountants, Uber drivers).
  • Efficiency Metrics: Will measure success not just by accuracy, but by the number of actions required relative to the human average, penalizing brute-force approaches common in past Ataris benchmarks.

Industry Adoption and Strategic Shifts

• Major Lab Endorsements: In the past 12 months, Arc has been adopted as a standard performance metric by OpenAI, xAI (Grok-4), Google (Gemini 3 Pro/DeepThink), and Anthropic (Opus 4.5).
• Critical Caveat: Foundation leadership warns that industry adoption does not equate to mission completion; high scores can sometimes represent "vanity metrics" or short-term gains from specific RL environment tuning ("whack-a-mole").
• Rejection of Pure RL Tuning: The foundation argues that training specifically for RL environments fails to test true generalization, as real-world intelligence requires handling novel problems without prior environment-specific training.

Forward-Looking Statements on AGI Declaration

• Solvability vs. AGI: Solving Arc AGI v1/v2 is a "necessary" condition for AGI but not "sufficient"; v3 is projected to provide the "most authoritative evidence to date" of a system capable of generalization.
• Future Criteria: The foundation intends to declare AGI only when a system demonstrates true generalization, defined by efficiency in both training data requirements and energy consumption, matching human-level metrics.
• Response to 100% Scores: If a model achieves 100% on Arc benchmarks tomorrow, the foundation plans to immediately analyze failure points in that system rather than immediately declaring AGI, maintaining a rigorous stance on the definition of intelligence.

Critique of Current Evaluation Trends

• Wall Clock Time: Dismissed as an arbitrary metric because performance is primarily a function of available compute rather than inherent intelligence.
• Key Efficiency Drivers: Intelligence is defined by the ratio of training data needed and energy consumed to execute a task, both of which have established human baselines.
• Risk of False Positives: Distinguishes between "economically valuable" products that may hit benchmarks via specific RL tuning and the "romantic pursuit" of general intelligence that requires handling unknown inputs.