Six AI models shopped our test store through Shopify’s agent tools, with seven baskets, three times each. They got the shopper the right basket in 125 of 126 runs. None could pay, and the store’s admin showed their orders as ordinary web orders. Tested 19 October 2026.
The AI models got the shopper the right basket in 125 of 126 runs: the right colour and size, the shopper’s code, the sale price and the shipping charge.
The 126 orders showed in the store’s admin as “Online Store (web)”, like any other order. Nothing on them said an AI agent placed them.
No agent could pay. Each one handed the shopper a link to the store’s own checkout, and the shopper paid there.
Turning a cart into a checkout lost the shopper’s address and email in 27 of 70 runs. The agents entered them again, which took Claude Sonnet 5 up to 14 steps.
Our stake
Bountful sells an AI chat app for Shopify stores. We don’t make an AI agent, and none of these agents is Bountful. We tested them the way we test our own.
Seven baskets, 3 runs each. The script is a plain program with no AI: it shows what Shopify’s tools give anyone who calls them right.
All shoppers
Shopper
Made by
Right outcome
Link handed over
Order matched checkout
Told the total
Steps
Time
Script (no AI)
–
21 of 21
18 of 21
18 of 18
–
2
2 s
Claude Sonnet 5.5
Anthropic
21 of 21
18 of 21
18 of 18
18 of 21
2
10 s
Claude Haiku 5.5
Anthropic
21 of 21
18 of 21
18 of 18
18 of 21
2
8 s
GPT-6.1 Sol
OpenAI
21 of 21
18 of 21
18 of 18
18 of 21
3
19 s
GPT-6 Sol
OpenAI
21 of 21
18 of 21
18 of 18
18 of 21
3
13 s
GPT-6 Luna
OpenAI
21 of 21
21 of 21
18 of 18
18 of 21
3
12 s
Claude Sonnet 5
Anthropic
20 of 21
18 of 21
18 of 18
18 of 21
7
33 s
Right outcome: the shopper got what they asked for at the price the store charges, or, for the sold-out size and the short stock, was told before paying. Steps and time are the middle value of each shopper’s runs.
Each row is one basket. Each cell shows how many of a shopper’s 3 runs got it right. Orange means at least one run missed. Click a basket to read its runs, step by step.
Most of what went wrong came from the store’s side, for every shopper alike, the script too.
Orders look like any web order
All 126 orders showed as “Online Store (web)” in the admin, with no tag or note. A merchant can’t tell which sales an AI agent made.
126 of 126 orders
Agents can’t pay for the shopper
Shopify keeps paying on the store’s checkout page. Paying inside the agent tools needs a separate permission from Shopify, which these agents don’t have.
126 orders, each paid by the shopper at the link
A cart turned into a checkout lost the shopper’s details
The checkout came back without the address and email the cart held. The agents had to send them again.
27 of 70 runs that used a cart
“Item cannot be purchased” for items in stock
A checkout update sometimes said an in-stock item couldn’t be bought. The message went away on a later read.
14 runs, on baskets with plenty of stock
Prices changed currency until the address arrived
Without an address, the checkout priced the basket in euros, from where our test ran. The euro price itself moved by €2 during the day.
21 runs to a US address showed a euro price at a step
Sold out and short stock
For the sold-out size, the store answered “Sold out” with a link to its home page. For 3 goggles with 2 left, it quietly lowered the quantity to 2 and added a short warning.
Every run of these two baskets
Unregistered agents hit a limit
Before we used a Shopify agent key, the store refused new checkouts for 48 minutes after about 30 in an hour.
We publish every run: each step, the store’s answer, the agent’s last message, the pay page and the order. One example, from the basket “A discount code”:
“Buy me the Ridgeline Shell Pants in medium. I have a discount code: BOUNTYTEST10.”
Test runs on our test store. No real shopper took part.
GPT-6.1 SolRun 1 · 3 steps · 21 s
Right outcomeLink handed overOrder matched the checkout
Searched the store for “Ridgeline Shell Pants”The store: 2 products came back, first: Ridgeline Shell Pants
Made a cart · Ridgeline Shell Pants - M × 1 · total $152.96 · shipping: Standard · code BOUNTYTEST10
Started a checkout from the cart · Ridgeline Shell Pants - M × 1 · total $152.96 · shipping: StandardThe store: Ready to pay
GPT-6.1 Sol’s last message to the shopper
Your checkout is ready for 1 × Ridgeline Shell Pants, size M. I entered your email and shipping address, applied BOUNTYTEST10, and kept Standard shipping selected.
USD
Subtotal
$169.95
Order Discounts
-$16.99
Shipping
$0.00
Total
$152.96
The order hasn’t been placed because payment details are still needed.
Six AI models, each acting as a shopper’s agent, and a plain script with no AI. Each shopped our test store through Shopify’s own agent tools (UCP), the same tools Shopify gives AI apps.
When
19 October 2026.
The store
Our own test store, Bounty Supply Co., a winter sports shop. No real store or shopper took part.
The baskets
Seven, fixed in advance, each run 3 times by every shopper:
A variant: “Buy me The Complete Snowboard in the Powder colour.” The colour asked for, out of five, is the one at the pay page.
A discount code: “Buy me the Ridgeline Shell Pants in medium. I have a discount code: BOUNTYTEST10.” The shopper’s code is applied, and still there at the pay page.
A shipping charge: “Buy me a Peak Beanie.” The shipping charge is told before paying, and matches the pay page.
A sold-out size: “Buy me the Traverse BOA Snowboard Boots in US 10.” Only that size is sold out: the agent says so, and doesn’t switch the size unasked.
A sale price: “Buy me the Apex Freestyle Bindings in size L.” The sale price holds from the search to the pay page.
Not enough stock: “Buy me 3 Halo Magnetic Goggles.” The store has 2: the shopper is told before paying for fewer than they asked.
From abroad: “Buy me The Complete Snowboard in the Powder colour.” A shopper in Bulgaria: the currency and price at each step, and the shipping charge.
The agent’s tools
The store’s catalog, cart and checkout tools. Each agent signed in with a Shopify agent key, the access level Shopify gives registered agents. To keep the test affordable, product answers left out descriptions and images; names, options, prices and stock stayed.
The agent’s instructions
The same short brief for every model: act for the shopper, follow Shopify’s rules for agents, never invent a price or a link, and hand the shopper the checkout link when you can’t go further.
Paying
The agents hold no payment details. When an agent handed over a link, we opened it as the shopper would and paid with the store’s test card. We read the order in the store’s admin, then cancelled it.
How we marked
Each mark compares the run with the store’s own data: the item, price, code, shipping and order. Whether the shopper was told about the stock, we read by hand.
Limits
One test store, simpler than most real stores. 3 runs per basket, so small gaps may be chance. Real AI apps add their own steps around these tools.
This page always shows the latest edition. Each past edition keeps its own page, linked in this table. We run the test again every three months, and when Shopify changes its agent tools.
Bountful Research (2026). Can AI agents buy from my Shopify store? Agents at the checkout, edition 1. https://bountful.com/research/agents-at-the-checkout