Visa benchmark finds AI shopping agents pass 57.2% of tests

Visa’s first benchmark of AI shopping agents found that the systems passed 57.2% of simulated shopping runs, while 54% of failed runs still ended in a purchase that missed the shopper’s requirements or was made before those needs were fully identified. The Visa Agentic Commerce Benchmark tested 22 models from 12 labs across 15 shopping challenges involving televisions, laptops and personal-care appliances. Visa said each challenge required an agent to identify a shopper’s needs through conversation, gather enough information to justify its choice and refrain from buying if no option met the requirements. The simulated assignments withheld some preferences at the outset. In a television scenario, for example, a shopper’s requirements included Dolby Vision compatibility, a 30-day return policy and use with a child’s gaming console. The agent had to ask questions to uncover the relevant details before selecting a product. Visa reported that the five highest-cost models passed 67.1% of runs, compared with 50.2% for the five lowest-cost models. The results show a difference between those groups but do not establish that model cost caused it. Visa also said the agents often bought...