Does AI chat get my store’s facts right?

Six AI models answered 62 shopper questions on our test store. The best got 175 right, the weakest 147. Tested 18 October 2026.

Key findings

  1. Claude Sonnet 5.5 got 175 of 186 answers right. GPT-6 Luna got 147 of 185.
  2. Asked for the most expensive board, five of six models named one that costs less than another in stock.
  3. Asked about PayPal, every model said it wasn’t sure, though the store’s FAQ answers it.
  4. GPT-6 Sol started its reply fastest, in 1.2 seconds. The slowest took 2.5 seconds.
Our stake

Bountful sells an AI chat app for Shopify that runs on some of these models. We tested every model the same way and kept the results that don’t flatter us.

Results

Right answers, out of 186

Each model answered 62 questions three times. When the two judges disagreed, we read the reply and decided.

Time to the first word, in seconds

The middle value of 186 replies, measured on our servers in the US. Shorter is better.

All models
ModelMade byRight answersWrong factsRight “we don’t have that”First wordFull replyTypical length
Claude Sonnet 5.5Anthropic175 of 1862 of 18632 of 362.5 s3.2 s58 words
Claude Haiku 5.5Anthropic169 of 1864 of 18627 of 362.3 s2.8 s57 words
Claude Sonnet 5Anthropic161 of 1867 of 18624 of 361.3 s2.7 s56 words
GPT-6 SolOpenAI158 of 1863 of 18625 of 361.2 s1.9 s37 words
GPT-6.1 SolOpenAI158 of 1863 of 18626 of 362.0 s3.4 s38 words
GPT-6 LunaOpenAI147 of 1854 of 18521 of 362.5 s3.0 s30 words

Wrong facts: replies that stated a price, size, stock state or product detail the store contradicts. We checked each one against the store. “We don’t have that”: 12 questions whose right answer is that the store doesn’t sell it, or doesn’t say. Times and length are the middle value of each model’s replies. Longer replies take longer to finish.

By kind of question

The 62 questions fall into seven groups. Each cell counts three runs. Orange marks a group where a model got at least one wrong, and says how many replies stated a wrong fact.

ModelFinding gearComparing productsSizing and fitStock and priceShipping and returnsCartTone and tricky
Claude Sonnet 5.541/4214/182 wrong facts21/2433/3336/3912/1218/18
Claude Haiku 5.537/4215/181 wrong fact24/2427/333 wrong facts36/3912/1218/18
Claude Sonnet 537/4212/183 wrong facts21/241 wrong fact29/333 wrong facts32/3912/1218/18
GPT-6 Sol34/4215/1819/2430/333 wrong facts32/3911/1217/18
GPT-6.1 Sol33/4217/1821/2430/333 wrong facts32/3912/1213/18
GPT-6 Luna34/4212/181 wrong fact18/231 wrong fact30/332 wrong facts30/3912/1211/18

The chats

We publish all 1,115 replies. Each question has its own page with every model’s reply and why a judge marked it wrong. One example:

“What's the difference between the Sentinel and the Aurora helmets?”Comparing products · 12 of 18 replies right

Test chats on our test store. No real shopper took part.

Claude Sonnet 5.5Right
BountfulShopping assistant
What's the difference between the Sentinel and the Aurora helmets?

The Aurora Audio Snow Helmet does everything the Sentinel Snow Helmet does, plus it has audio-ready earpads with drop-in speaker pockets for music on the lift.

  • Sentinel: MIPS rotational protection, fourteen adjustable vents and an itch-free fleece liner. It fits over most goggles, including the Prism and Halo lines. It's $99.95.
  • Aurora: the same core protection, plus the audio pockets and a magnetic buckle you can work with gloves on. It's $129.95, down from $149.95.

Both come in S, M and L. If you don't listen to music while riding, the Sentinel covers the essentials for less.

Do you ride with music?

Aurora Audio Snow Helmet$129.95, was $149.95SAdd to cart
Sentinel Snow Helmet$99.95SAdd to cart
Ask me anything about the store
Claude Sonnet 5.5Wrong
BountfulShopping assistant
What's the difference between the Sentinel and the Aurora helmets?

The Aurora Audio Snow Helmet is the Sentinel Snow Helmet plus audio. Both have MIPS rotational protection and a magnetic buckle you can work with gloves on.

  • Sentinel Snow Helmet ($99.95): fourteen adjustable vents and a fleece liner. It fits over most goggles, including the Prism and Halo lines.
  • Aurora Audio Snow Helmet ($129.95, on sale from $149.95): the same protection, plus audio-ready earpads with drop-in speaker pockets for music on the lift.

If you ride with music or podcasts, the Aurora is worth it. If not, the Sentinel gives you the same safety for less.

Do you listen to music while you ride?

Aurora Audio Snow Helmet$129.95, was $149.95SAdd to cart
Sentinel Snow Helmet$99.95SAdd to cart
Ask me anything about the store

Anthropic’s judge: “Says both have a magnetic buckle, but the rule says the Aurora adds it, so the Sentinel's buckle is invented.” OpenAI’s judge: “It incorrectly says the Sentinel has a magnetic buckle.”

See all 18 replies to this question
Read all 1,115 chats62 questions, one page each

How we ran this

What we tested
Six AI models, each running as the shop assistant in the same Bountful setup. Only the model changed: same store, same tools, same instructions.
When
18 October 2026, 18:50 to 20:41 UTC.
The store
Our own test store, Bounty Supply Co., a winter sports shop. Its chat assistant is named Frosty. No real store or shopper took part.
The questions
62 shopper questions in seven groups, fixed in advance. Each has an answer you can check on the store. See all 62
How we graded
Each question has a written rule for a right answer. Two AI judges, Claude Sonnet 5.5 from Anthropic and GPT-6 Sol from OpenAI, read every reply against it. A reply is right when both pass it. When they disagreed, we read the reply against the rule and decided. A person also checked a sample of the rest by hand.
Limits
Each model answered each question three times, so small gaps may still be chance. Each judge is also one of the models tested. A test store is simpler than most real stores.

Run history

EditionTestedMethodWhat changed
Edition 1 (this page)18 Oct 20261.0First edition.

This page always shows the latest edition. Each past edition keeps its own page, linked in this table, with its numbers as they were published. We run the test again for each new major model, and at least every three months.

The data

Every reply, verdict and timing is free to use under CC BY 4.0. Please name Bountful Research and link this page.

Bountful Research (2026). Does AI chat get my store’s facts right? The shop assistant test, edition 1. https://bountful.com/research/shop-assistant-test

Bountful is an AI chat assistant for Shopify stores. See what it does · More research