Does AI chat get my store’s facts right?
Six AI models answered 62 shopper questions on our test store. The best got 175 right, the weakest 147. Tested 18 October 2026.
Key findings
- Claude Sonnet 5.5 got 175 of 186 answers right. GPT-6 Luna got 147 of 185.
- Asked for the most expensive board, five of six models named one that costs less than another in stock.
- Asked about PayPal, every model said it wasn’t sure, though the store’s FAQ answers it.
- GPT-6 Sol started its reply fastest, in 1.2 seconds. The slowest took 2.5 seconds.
Bountful sells an AI chat app for Shopify that runs on some of these models. We tested every model the same way and kept the results that don’t flatter us.
Results
Right answers, out of 186
Each model answered 62 questions three times. When the two judges disagreed, we read the reply and decided.
Time to the first word, in seconds
The middle value of 186 replies, measured on our servers in the US. Shorter is better.
| Model | Made by | Right answers | Wrong facts | Right “we don’t have that” | First word | Full reply | Typical length |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | Anthropic | 175 of 186 | 2 of 186 | 32 of 36 | 2.5 s | 3.2 s | 58 words |
| Claude Haiku 5.5 | Anthropic | 169 of 186 | 4 of 186 | 27 of 36 | 2.3 s | 2.8 s | 57 words |
| Claude Sonnet 5 | Anthropic | 161 of 186 | 7 of 186 | 24 of 36 | 1.3 s | 2.7 s | 56 words |
| GPT-6 Sol | OpenAI | 158 of 186 | 3 of 186 | 25 of 36 | 1.2 s | 1.9 s | 37 words |
| GPT-6.1 Sol | OpenAI | 158 of 186 | 3 of 186 | 26 of 36 | 2.0 s | 3.4 s | 38 words |
| GPT-6 Luna | OpenAI | 147 of 185 | 4 of 185 | 21 of 36 | 2.5 s | 3.0 s | 30 words |
Wrong facts: replies that stated a price, size, stock state or product detail the store contradicts. We checked each one against the store. “We don’t have that”: 12 questions whose right answer is that the store doesn’t sell it, or doesn’t say. Times and length are the middle value of each model’s replies. Longer replies take longer to finish.
By kind of question
The 62 questions fall into seven groups. Each cell counts three runs. Orange marks a group where a model got at least one wrong, and says how many replies stated a wrong fact.
| Model | Finding gear | Comparing products | Sizing and fit | Stock and price | Shipping and returns | Cart | Tone and tricky |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 41/42 | 14/182 wrong facts | 21/24 | 33/33 | 36/39 | 12/12 | 18/18 |
| Claude Haiku 5.5 | 37/42 | 15/181 wrong fact | 24/24 | 27/333 wrong facts | 36/39 | 12/12 | 18/18 |
| Claude Sonnet 5 | 37/42 | 12/183 wrong facts | 21/241 wrong fact | 29/333 wrong facts | 32/39 | 12/12 | 18/18 |
| GPT-6 Sol | 34/42 | 15/18 | 19/24 | 30/333 wrong facts | 32/39 | 11/12 | 17/18 |
| GPT-6.1 Sol | 33/42 | 17/18 | 21/24 | 30/333 wrong facts | 32/39 | 12/12 | 13/18 |
| GPT-6 Luna | 34/42 | 12/181 wrong fact | 18/231 wrong fact | 30/332 wrong facts | 30/39 | 12/12 | 11/18 |
The chats
We publish all 1,115 replies. Each question has its own page with every model’s reply and why a judge marked it wrong. One example:
Test chats on our test store. No real shopper took part.
The Aurora Audio Snow Helmet does everything the Sentinel Snow Helmet does, plus it has audio-ready earpads with drop-in speaker pockets for music on the lift.
- Sentinel: MIPS rotational protection, fourteen adjustable vents and an itch-free fleece liner. It fits over most goggles, including the Prism and Halo lines. It's $99.95.
- Aurora: the same core protection, plus the audio pockets and a magnetic buckle you can work with gloves on. It's $129.95, down from $149.95.
Both come in S, M and L. If you don't listen to music while riding, the Sentinel covers the essentials for less.
Do you ride with music?
The Aurora Audio Snow Helmet is the Sentinel Snow Helmet plus audio. Both have MIPS rotational protection and a magnetic buckle you can work with gloves on.
- Sentinel Snow Helmet ($99.95): fourteen adjustable vents and a fleece liner. It fits over most goggles, including the Prism and Halo lines.
- Aurora Audio Snow Helmet ($129.95, on sale from $149.95): the same protection, plus audio-ready earpads with drop-in speaker pockets for music on the lift.
If you ride with music or podcasts, the Aurora is worth it. If not, the Sentinel gives you the same safety for less.
Do you listen to music while you ride?
Anthropic’s judge: “Says both have a magnetic buckle, but the rule says the Aurora adds it, so the Sentinel's buckle is invented.” OpenAI’s judge: “It incorrectly says the Sentinel has a magnetic buckle.”
How we ran this
- What we tested
- Six AI models, each running as the shop assistant in the same Bountful setup. Only the model changed: same store, same tools, same instructions.
- When
- 18 October 2026, 18:50 to 20:41 UTC.
- The store
- Our own test store, Bounty Supply Co., a winter sports shop. Its chat assistant is named Frosty. No real store or shopper took part.
- The questions
- 62 shopper questions in seven groups, fixed in advance. Each has an answer you can check on the store. See all 62
- How we graded
- Each question has a written rule for a right answer. Two AI judges, Claude Sonnet 5.5 from Anthropic and GPT-6 Sol from OpenAI, read every reply against it. A reply is right when both pass it. When they disagreed, we read the reply against the rule and decided. A person also checked a sample of the rest by hand.
- Limits
- Each model answered each question three times, so small gaps may still be chance. Each judge is also one of the models tested. A test store is simpler than most real stores.
Run history
| Edition | Tested | Method | What changed |
|---|---|---|---|
| Edition 1 (this page) | 18 Oct 2026 | 1.0 | First edition. |
This page always shows the latest edition. Each past edition keeps its own page, linked in this table, with its numbers as they were published. We run the test again for each new major model, and at least every three months.
The data
Every reply, verdict and timing is free to use under CC BY 4.0. Please name Bountful Research and link this page.
Bountful is an AI chat assistant for Shopify stores. See what it does · More research