Google AI Overviews 90% Accurate but Millions of Errors Daily

A recent analysis commissioned by The New York Times has found that Google's AI Overviews, the artificial intelligence-generated summaries that appear at the top of search results, deliver accurate information about 90 percent of the time. However, with Google handling billions of searches daily, this error rate translates to tens of millions of potentially misleading responses each day, raising questions about the reliability of AI-driven search tools.

The study, conducted by AI startup Oumi, evaluated AI Overviews using the SimpleQA benchmark, a set of over 4,000 questions with verifiable answers developed by OpenAI in 2024. Tests in October, when the feature relied on Google's Gemini 2 model, showed an accuracy rate of 85 percent. After an upgrade to the more advanced Gemini 3 model by February, accuracy improved to 91 percent. Oumi analyzed 4,326 searches in each round, focusing on the AI's ability to provide correct answers and link to supporting sources.

Despite the overall high accuracy, the analysis highlighted persistent issues. In cases where responses were correct, more than half lacked full support from the cited websites, a problem that worsened after the Gemini 3...