Google updates Android Bench with Harbor and new models

Google has updated its Android Bench, a specialized benchmark for evaluating how artificial intelligence models perform at real-world Android development tasks, introducing a new testing framework, expanded model coverage, and community collaboration features. The changes, part of a July release, aim to provide developers with a more accurate and transparent tool for selecting AI coding assistants.

The most significant update is the adoption of the Harbor framework, replacing the previous evaluation system based on a more general-purpose tool. Google states that Harbor provides a standardized foundation, making it easier for developers and researchers to reproduce evaluations, compare different models, and share results consistently. Because of this methodological shift, the company has rerun the benchmark for every model on the leaderboard, establishing a fresh baseline. While scores have shifted modestly, historical results remain accessible through an archive for comparison.

The refreshed leaderboard now includes eight new large language models: Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus, and Qwen 3.7 Max. In the new...