In an unusual crossover between AI benchmarking and psychometrics, researchers have published the first validation study of the ARC-AGI benchmark in human participants, finding that performance on the AI-oriented test correlates substantially with fluid intelligence. The study tested 100 human participants on a compilation of ARC-AGI items alongside established measures of figural reasoning and creativity.
The ARC-AGI benchmark, developed by François Chollet and designed to measure an AI system’s ability to induce novel rules from minimal examples, was originally proposed as a measure of fluid intelligence, the capacity to solve novel problems independent of acquired knowledge. Yet until this study, it had never been psychometrically validated in humans.
The results showed a substantial correlation (ρ = .63) between ARC-AGI performance and a standard figural fluid intelligence test. Associations with figural originality, a measure of creative thinking, were weak. The ARC-AGI items themselves demonstrated good psychometric properties, supporting their reliability as a measurement instrument.
The findings are significant for two reasons. First, they provide empirical support for ARC-AGI’s claim to measure something related to human fluid intelligence, strengthening its validity as a benchmark for AI systems. Second, they represent a broader methodological argument: that AI benchmarks should be systematically embedded within the nomological network of human cognitive abilities to enable more rigorous evaluation and foster interdisciplinary cooperation between AI research and psychology.
The study is a first step, the sample size of 100 is modest, and the authors note the need for larger studies with additional covariates including working memory measures. But the approach itself, taking a benchmark designed to test machines and running it on humans, opens a path toward more interpretable evaluations of artificial intelligence.
Sources: Bringing Back Rule Induction to Fluid Intelligence Research? An Initial Validation of the ARC-AGI Benchmark in Humans (arXiv, July 2026)