- \n
- First-of-its-kind1<\/sup> benchmark tests AI in realistic shopping scenarios\u2014and consumers can try the test, too<\/i><\/li>\n
- AI should uncover customer needs and make an informed purchase decision; models failed 42.8% of test runs<\/i><\/li>\n
- Benchmark creates a roadmap for industry players to deliver better agentic shopping experiences<\/i><\/li>\n<\/ul>\n
SAN FRANCISCO–(BUSINESS WIRE<\/a>)–
\nVisa (NYSE: V) today launched Visa Agentic Commerce Benchmark, a new benchmark for industry that puts leading AI models through realistic buying challenges\u2014and is letting people take the test, too.<\/p>\n\nShopping is a difficult task for AI because consumers often do not clearly know what they want at the outset\u2014for example, they may know they want a TV, but not all the required specifications. Under Visa\u2019s new benchmark, AI models are required to identify the needs of a consumer based on a conversation. If nothing fits the customer\u2019s requirements, the AI should opt not to buy. To prevent lucky guesses, the AI must uncover enough information to justify its decision.<\/p>\n
\nThe first round of tests involved 22 different AI models from 12 labs taking on 15 shopping challenges like buying a TV, laptop, or personal care appliance.<\/p>\n
\nThe Results<\/p>\n
- \n
- Room for improvement:<\/b> Across testing, selected AI models achieved an overall benchmark pass rate of about half (57.2%). The best performing models passed 82.2% of attempts, the worst 28.9%.\n<\/li>\n
- The more you know: <\/b>The five AI models that uncovered the most hidden buying factors using conversation flows with the consumer averaged a 70.2% pass rate. The five that uncovered the least passed less than half (44.4%) of the time.\n<\/li>\n
- Pricy models win, but do not guarantee success: <\/b>The five models with the highest average cost per run passed 67.1% of runs, compared with 50.2% for the five least expensive.\n<\/li>\n
- Consistency is an issue:<\/b> Every model got three attempts at each shopping scenario. Across all models and scenarios, all three attempts passed just 34.2% of the time.\n<\/li>\n
- Buyer beware: <\/b>About 1-in-2 (54%) of failed runs still ended in the AI making a purchase. Either the AI recommended a purchase that did not meet the customer\u2019s identified requirements, or it failed to uncover all the customer\u2019s requirements.\n<\/li>\n<\/ul>\n
\n\u201cAI agents have the potential to power the next era of commerce, but trust is key to making it work,\u201d <\/b>said Rubail Birwadker, Global Head of Growth at Visa. \u201cVisa Agentic Commerce Benchmark is yet another way Visa is working to boost standards across the industry.\u201d<\/p>\n
Superintelligence Sweep<\/b><\/p>\n
\nIn one scenario, the customer asks for help choosing a new TV. They mention their relative has recommended a brand of television, and that they hold a membership with a third-party supplier\u2014but leave other details unsaid.<\/p>\n
\nThrough conversation, the AI must uncover the customer\u2019s budget and additional requirements\u2014such as format compatibility, HDR10+\u00ae technology support for family videos, and gaming capabilities.<\/p>\n
\nModels that were better at uncovering the customer\u2019s requirements tended to pass more often. But when made to repeat a scenario they had already passed, the same models often failed\u2014showing how difficult it is to deliver a consistent consumer experience.<\/p>\n
\nEliminating errors and improving consistency is crucial to building trust in agentic commerce, which is still relatively low today. Visa\u2019s Trust Index<\/a>, released in September 2026, found that while 72% of U.S. consumers had used an AI assistant, only 23% trusted AI to handle payment transactions on their behalf. Visa Intelligent Commerce, Visa\u2019s portfolio of products, programs and partnerships for agentic commerce, is designed to tackle this challenge.<\/p>\n
\nVisa plans to add other shopping scenarios and product categories to Visa Agentic Commerce Benchmark, and will extend testing to checkout, payment, and order placement. Visa has shared the benchmark with leading model providers for feedback.<\/p>\n
\nVisit Visa Payments Frontier<\/a> to attempt Visa Agentic Commerce Benchmark yourself, watch replays of AI attempts, and dig deeper into the research with the Visa Agentic Commerce Benchmark whitepaper.<\/p>\n
FAQs<\/b><\/p>\n
Is Visa Agentic Commerce Benchmark part of Visa Intelligent Commerce?<\/b><\/p>\n
\nNot directly. Visa Intelligent Commerce is a portfolio of products, programs and partnerships that help bring trust to agentic commerce. The benchmark is complementary research designed to benefit the wider industry.<\/p>\n
Is Visa saying AI agents aren\u2019t ready for commerce?<\/b><\/p>\n
\nNo. The results show promising capabilities, with the highest-scoring models each passing 82.2% of attempts. But consistency and reliability remain challenges.<\/p>\n
Is Visa Agentic Commerce Benchmark publicly available?<\/b><\/p>\n
\nYes. Anyone can explore the research and interactive experience on Visa Payments Frontier, including consumers, researchers, developers, merchants, clients and AI companies.<\/p>\n
Why is Visa benchmarking AI models?<\/b><\/p>\n
\nTo help make agentic commerce more reliable. The benchmark gives the industry a consistent way to understand where AI shopping works today and what still needs to improve.<\/p>\n
About Visa<\/b><\/p>\n
\nVisa (NYSE: V) is a world leader in digital payments, facilitating transactions between consumers, sellers, financial institutions and government entities across more than 200 countries and territories. Our mission is to connect the world through the most innovative, convenient, reliable and secure payments network, enabling individuals, businesses and economies to thrive. We believe that economies that include everyone everywhere, uplift everyone everywhere and see access as foundational to the future of money movement. Learn more at Visa.com<\/a>.<\/p>\n
- The more you know: <\/b>The five AI models that uncovered the most hidden buying factors using conversation flows with the consumer averaged a 70.2% pass rate. The five that uncovered the least passed less than half (44.4%) of the time.\n<\/li>\n
- AI should uncover customer needs and make an informed purchase decision; models failed 42.8% of test runs<\/i><\/li>\n
![\"Logo\"](\"https:\/\/mms.businesswire.com\/media\/20261105169310\/en\/1260353\/3\/Visa_Brandmark_Blue_RGB_72ppi.jpg\")
<\/td>\n<\/tr>\n