Putting AI Shopping Assistants to the Test

Casey Lewis, who writes the Gen-Z trendspotting newsletter After School, has spent years fruitlessly searching online for a simple shift mini dress. The closest she’s come to her ideal, she said in one newsletter, was an Everlane dress from 2020 that’s no longer available.

Recently, she tried a new approach. She asked ShopWith AI, an AI shopping assistant that mixes OpenAI’s ChatGPT and proprietary technology, to recommend a black minidress.

“It actually did surface some brands that I’d never heard of before,” Lewis said in an interview. “Honestly, I might order one of these dresses because it was so cute and seemingly exactly what I was looking for and probably wouldn’t find otherwise.”

In recent months, businesses including Shopify, Mercari and KNXT — an online shop operated by Kering — have debuted ChatGPT-powered shopping assistants. Zalando plans to release its own soon.

These chatbots are a leap forward from the previous generation of automated online assistants. Those drew from a set of scripted responses, but the hope for this new wave is that they can be smart, flexible and responsive, like an actual salesperson. A customer could ask questions like “What should I wear to an October wedding in New York?” and receive a reply that takes into account current trends, the location and the weather, along with links to products from the retailer’s catalogue. The customer could give the bot feedback and direction too in order to get closer to the products they’re after.

That future is still largely theoretical. While these generative AI bots are able to give coherent answers personalised to a shopper’s questions, complete with recommendations of relevant products, they can still feel robotic and have limitations.

In her use of ShopWith AI, which lets you shop by asking questions to celebrity personas like Kim Kardashian, Lewis said she still felt like she was talking to a bot. The brand recommendations are more useful, but that’s in large part the work of a human: ShopWith AI’s co-founder, Sarah Stark, who chose the list of brands and boutiques from which the AI selects products.

“It’s still a very curated approach. It’s not the mainstream brands you’re getting on the first three pages of your Google search results,” Stark said.

To get a better sense of how well the new crop of AI shopping assistants work, BoF staffers tested some. We fed the same prompts into different bots to see how the results compared and asked a variety of questions to get an overall sense of their abilities. Here’s what we found.

The Reviews

The bots understood the prompts and were mostly able to recommend appropriate items, knowing that linen is good for warmer weather and having a general understanding of the dress code for various events. But overall, the reviewers were left underwhelmed by the product suggestions themselves. At times the AI’s effectiveness came down to the non-technological issue of whether a retailer stocked clothes that fit the query. Every store won’t have items for every occasion, of course, but at least a human associate could definitively say whether a shopper should look elsewhere rather than leaving them guessing whether the problem was how they worded their question and whether the associate knew the answer.

0 of 4

One tester, who asked what to wear to a boat party in Europe this summer after getting the question from a friend, said the recommendations across the bots generally weren’t very useful, though she felt Shopify’s suggestions seemed a bit more curated. She ended up sending her friend one of them, a white button-front maxi dress.

KNXT’s Madeline bot, which offers products from Kering brands and those in which the holding company of the Pinault family behind Kering owns a stake, seemed to struggle the most with her search. It offered some out-there Balenciaga looks she believed weren’t a fit for a summer boat party. (When another reviewer asked about Bottega Veneta menswear, it still suggested Balenciaga.)

A reviewer looking for graduation dresses for a June ceremony in Chicago liked KNXT’s suggestions best in terms of style but found the interface clunky. Shopify’s was quick and easy but she didn’t like the products it recommended, which included festival outfits that left her confused.

On the other hand, she and another tester, who asked about clothes her husband could wear to a cricket match around London, noted Shopify’s bot was the only one that appeared to take weather into account. For the London-area cricket match, it selected a raincoat as one of its options.

Multiple reviewers said the product recommendations by Mercari’s Merchat didn’t feel very curated, perhaps because Mercari is a secondhand marketplace and dependent on the listings of individual sellers. They also noted that it asked a lot of questions, apparently trying to get more context — “too many questions for my Gen-Z attention span,” the youngest reviewer noted.

The tester shopping for her husband, however, appreciated that it wanted more information on him, since he’s 6-foot-3-inches, and had ideas on what would fit him best. Where it fell short was its product suggestions, including shorts, which she noted people are less likely to wear around London. She generally found its point of view, including its language, to be US-centric. “Can’t say I’m convinced,” she concluded about shopping with an AI assistant.

The reviewer looking for graduation dresses wasn’t thrilled with ShopWith AI’s options either, though she was more impressed when she asked its Lenny Kravitz persona what to wear to a concert. The outfit actually looked like something Kravitz might pick out, she said.

The conversational interface used by these shopping assistants did offer some benefit in that it allowed the reviewers to ask for suggestions and not just search for products. But the text replies often sounded like marketing copy, and any conversational capabilities won’t much matter if the bots can’t nail the job of offering up the right products.

It’s Still Early for Generative-AI Shopping Assistants

Rough as these bots may currently be, they represent the early days of generative-AI shopping assistants and should theoretically improve as the technology advances, which it’s doing rapidly. The companies behind them acknowledge they’re more for testing and learning than acting as finished products.

“One of the reasons for getting out early was to be able to see what that user feedback was, how people would interact with it and how they would make sense of it,” said Miqdad Jaffer, Shopify’s director of product. “The places we would want to make improvements are on the type of results that come back, the types of conversations that take place and then the ability to filter further.”

Kering similarly called KNXT a place to “test innovative digital experiences” in a statement. John Lagerling, chief executive of Mercari US, said in an email they’re learning and iterating as they go and expect to keep improving on the quality of their product recommendations.

But the theoretical strength of these bots is that the large language models underlying them are designed in a way that makes them good at predicting a user’s intent. Robert Hetu, vice president and analyst at Gartner, a technology research and advisory firm, said previous chatbots work well for straightforward tasks like telling you an order’s status but haven’t been great for shopping because they need a lot of specificity from users to guide them. Shopping assistants powered by generative AI, on the other hand, are more likely to be successful because they’re much better at accurately predicting the intent of a customer.

Still, that’s not an easy skill to get right.

“Transparently, it’s a work in progress,” Jake Stark, the other co-founder of ShopWith AI, said. “It’s hard stuff, personalised search.”

Lewis, who described herself as a “tech optimist,” said AI shopping assistants won’t supplant conventional search for her at this stage. But that could change if tech giants like Google and Microsoft, who are incorporating generative AI into their products, direct their huge amounts of data, expertise and resources toward the matter.

“It hasn’t replaced search, but I’m confident that in the future it might,” she said. “Maybe they’ll be the ones to do it.”