ARCAGI3 Benchmark Reveals AI Generalization Limits

The ARC Prize Foundation released ARCAGI3 this week, a new benchmark that challenges artificial intelligence systems to demonstrate human-like generalization in unfamiliar interactive environments, revealing stark limitations in leading models amid recent industry claims of achieving artificial general intelligence.

Developed by François Chollet and the ARC Prize Foundation, which established an in-house game studio, ARCAGI3 consists of 135 original turn-based game environments. AI agents are placed into these pixelated worlds without instructions, goals, or rule descriptions. They must explore the space, form hypotheses about objectives, develop plans, and execute actions to succeed. Humans, tested without prior exposure, solved all environments efficiently, achieving perfect performance. In contrast, frontier models tested via official APIs scored below 1 on the benchmark's Relative Human Action Efficiency scale, which measures not just correctness but efficiency relative to human attempts, applying a squared penalty for excess actions to discourage brute-force approaches.

Google's Gemini 3.1 Pro achieved the highest score at 0.37, followed by OpenAI's GPT-5.4 at 0.26 and...