Ezra's Bookshelf

On the Measure of Intelligence

by Francois Chollet · 154 pages · ~3 hrs

Science & TechnologyPhilosophy

Recommended on

Francois Chollet, the creator of the Keras deep learning framework, presents a rigorous examination of how we should define and measure intelligence in artificial systems. The paper argues that current AI benchmarks fundamentally fail to capture what we mean by intelligence because they conflate skill at specific tasks with general cognitive ability. Chollet distinguishes between crystallized skills that can be achieved through memorization or narrow optimization and fluid intelligence that enables adaptation to genuinely novel situations. He proposes that true intelligence should be measured by sample efficiency—how quickly a system can adapt to new tasks given limited experience—rather than by peak performance on any particular benchmark. The paper introduces the Abstraction and Reasoning Corpus (ARC), a set of tasks designed to test exactly this kind of fluid intelligence: problems that humans find intuitive but that resist solution through brute-force pattern matching or massive training data. Chollet's framework draws on psychometric research, cognitive science, and algorithmic information theory to build a mathematically precise definition of intelligence as skill-acquisition efficiency. For readers following AI development, this work provides essential conceptual tools for evaluating claims about artificial general intelligence and understanding why systems that achieve superhuman performance on games or benchmarks may still lack capacities that any human child possesses. The analysis has implications for how we should direct AI research and what milestones would indicate genuine progress toward general intelligence.

For fans of