The shop assistant test: the data
1,115 replies from six AI models to 62 shopper questions, tested 18 October 2026.
Download the data (CSV)CSV · 1115 rows · 842 KB · CC BY 4.0
| Question | Group | Model | Run | Verdict | First word | Chat |
|---|---|---|---|---|---|---|
| first-board | Finding gear | GPT-6.1 Sol | 1 | Right | 2.5 s | Read |
| first-board | Finding gear | GPT-6 Luna | 1 | Right | 2.8 s | Read |
| first-board | Finding gear | GPT-6 Sol | 1 | Right | 2.0 s | Read |
| first-board | Finding gear | Claude Haiku 5.5 | 1 | Right | 3.1 s | Read |
| first-board | Finding gear | Claude Sonnet 5 | 1 | Right | 1.4 s | Read |
| first-board | Finding gear | Claude Sonnet 5.5 | 1 | Right | 7.5 s | Read |
| first-board | Finding gear | GPT-6.1 Sol | 2 | Right | 1.7 s | Read |
| first-board | Finding gear | GPT-6 Luna | 2 | Right | 2.5 s | Read |
| first-board | Finding gear | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| first-board | Finding gear | Claude Haiku 5.5 | 2 | Right | 1.9 s | Read |
| first-board | Finding gear | Claude Sonnet 5 | 2 | Right | 1.0 s | Read |
| first-board | Finding gear | Claude Sonnet 5.5 | 2 | Right | 2.2 s | Read |
| first-board | Finding gear | GPT-6.1 Sol | 3 | Right | 1.5 s | Read |
| first-board | Finding gear | GPT-6 Luna | 3 | Right | 2.8 s | Read |
| first-board | Finding gear | GPT-6 Sol | 3 | Right | 1.2 s | Read |
| first-board | Finding gear | Claude Haiku 5.5 | 3 | Right | 2.3 s | Read |
| first-board | Finding gear | Claude Sonnet 5 | 3 | Right | 1.4 s | Read |
| first-board | Finding gear | Claude Sonnet 5.5 | 3 | Right | 2.1 s | Read |
| beginner-bindings | Finding gear | GPT-6.1 Sol | 1 | Right | 1.7 s | Read |
| beginner-bindings | Finding gear | GPT-6 Luna | 1 | Right | 2.4 s | Read |
| beginner-bindings | Finding gear | GPT-6 Sol | 1 | Right | 1.2 s | Read |
| beginner-bindings | Finding gear | Claude Haiku 5.5 | 1 | Right | 1.0 s | Read |
| beginner-bindings | Finding gear | Claude Sonnet 5 | 1 | Right | 1.7 s | Read |
| beginner-bindings | Finding gear | Claude Sonnet 5.5 | 1 | Right | 1.3 s | Read |
| beginner-bindings | Finding gear | GPT-6.1 Sol | 2 | Right | 1.5 s | Read |
| beginner-bindings | Finding gear | GPT-6 Luna | 2 | Right | 2.6 s | Read |
| beginner-bindings | Finding gear | GPT-6 Sol | 2 | Right | 1.2 s | Read |
| beginner-bindings | Finding gear | Claude Haiku 5.5 | 2 | Right | 2.7 s | Read |
| beginner-bindings | Finding gear | Claude Sonnet 5 | 2 | Right | 1.0 s | Read |
| beginner-bindings | Finding gear | Claude Sonnet 5.5 | 2 | Right | 1.5 s | Read |
| beginner-bindings | Finding gear | GPT-6.1 Sol | 3 | Right | 2.0 s | Read |
| beginner-bindings | Finding gear | GPT-6 Luna | 3 | Right | 1.0 s | Read |
| beginner-bindings | Finding gear | GPT-6 Sol | 3 | Right | 1.0 s | Read |
| beginner-bindings | Finding gear | Claude Haiku 5.5 | 3 | Right | 2.0 s | Read |
| beginner-bindings | Finding gear | Claude Sonnet 5 | 3 | Right | 1.1 s | Read |
| beginner-bindings | Finding gear | Claude Sonnet 5.5 | 3 | Right | 1.2 s | Read |
| park-bindings | Finding gear | GPT-6.1 Sol | 1 | Right | 1.8 s | Read |
| park-bindings | Finding gear | GPT-6 Luna | 1 | Right | 2.4 s | Read |
| park-bindings | Finding gear | GPT-6 Sol | 1 | Right | 1.4 s | Read |
| park-bindings | Finding gear | Claude Haiku 5.5 | 1 | Right | 2.4 s | Read |
| park-bindings | Finding gear | Claude Sonnet 5 | 1 | Right | 1.4 s | Read |
| park-bindings | Finding gear | Claude Sonnet 5.5 | 1 | Right | 1.8 s | Read |
| park-bindings | Finding gear | GPT-6.1 Sol | 2 | Right | 1.4 s | Read |
| park-bindings | Finding gear | GPT-6 Luna | 2 | Right | 2.0 s | Read |
| park-bindings | Finding gear | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| park-bindings | Finding gear | Claude Haiku 5.5 | 2 | Right | 2.0 s | Read |
| park-bindings | Finding gear | Claude Sonnet 5 | 2 | Right | 1.4 s | Read |
| park-bindings | Finding gear | Claude Sonnet 5.5 | 2 | Right | 1.4 s | Read |
| park-bindings | Finding gear | GPT-6.1 Sol | 3 | Right | 2.0 s | Read |
| park-bindings | Finding gear | GPT-6 Luna | 3 | Right | 2.3 s | Read |
| park-bindings | Finding gear | GPT-6 Sol | 3 | Right | 2.2 s | Read |
| park-bindings | Finding gear | Claude Haiku 5.5 | 3 | Right | 2.7 s | Read |
| park-bindings | Finding gear | Claude Sonnet 5 | 3 | Right | 1.6 s | Read |
| park-bindings | Finding gear | Claude Sonnet 5.5 | 3 | Right | 1.8 s | Read |
| carving-board | Finding gear | GPT-6.1 Sol | 1 | Right | 1.5 s | Read |
| carving-board | Finding gear | GPT-6 Luna | 1 | Right | 2.2 s | Read |
| carving-board | Finding gear | GPT-6 Sol | 1 | Right | 1.3 s | Read |
| carving-board | Finding gear | Claude Haiku 5.5 | 1 | Right | 2.6 s | Read |
| carving-board | Finding gear | Claude Sonnet 5 | 1 | Right | 1.1 s | Read |
| carving-board | Finding gear | Claude Sonnet 5.5 | 1 | Right | 1.4 s | Read |
| carving-board | Finding gear | GPT-6.1 Sol | 2 | Right | 1.9 s | Read |
| carving-board | Finding gear | GPT-6 Luna | 2 | Right | 2.3 s | Read |
| carving-board | Finding gear | GPT-6 Sol | 2 | Right | 1.2 s | Read |
| carving-board | Finding gear | Claude Haiku 5.5 | 2 | Right | 2.2 s | Read |
| carving-board | Finding gear | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| carving-board | Finding gear | Claude Sonnet 5.5 | 2 | Right | 1.1 s | Read |
| carving-board | Finding gear | GPT-6.1 Sol | 3 | Right | 2.0 s | Read |
| carving-board | Finding gear | GPT-6 Luna | 3 | Right | 2.9 s | Read |
| carving-board | Finding gear | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| carving-board | Finding gear | Claude Haiku 5.5 | 3 | Right | 1.9 s | Read |
| carving-board | Finding gear | Claude Sonnet 5 | 3 | Right | 1.6 s | Read |
| carving-board | Finding gear | Claude Sonnet 5.5 | 3 | Right | 1.6 s | Read |
| stiff-boots | Finding gear | GPT-6.1 Sol | 1 | Right | 2.8 s | Read |
| stiff-boots | Finding gear | GPT-6 Luna | 1 | Right | 2.5 s | Read |
| stiff-boots | Finding gear | GPT-6 Sol | 1 | Right | 1.3 s | Read |
| stiff-boots | Finding gear | Claude Haiku 5.5 | 1 | Right | 2.5 s | Read |
| stiff-boots | Finding gear | Claude Sonnet 5 | 1 | Right | 1.1 s | Read |
| stiff-boots | Finding gear | Claude Sonnet 5.5 | 1 | Right | 2.4 s | Read |
| stiff-boots | Finding gear | GPT-6.1 Sol | 2 | Right | 1.8 s | Read |
| stiff-boots | Finding gear | GPT-6 Luna | 2 | Right | 3.0 s | Read |
| stiff-boots | Finding gear | GPT-6 Sol | 2 | Right | 1.2 s | Read |
| stiff-boots | Finding gear | Claude Haiku 5.5 | 2 | Right | 3.0 s | Read |
| stiff-boots | Finding gear | Claude Sonnet 5 | 2 | Right | 1.3 s | Read |
| stiff-boots | Finding gear | Claude Sonnet 5.5 | 2 | Right | 2.5 s | Read |
| stiff-boots | Finding gear | GPT-6.1 Sol | 3 | Right | 2.3 s | Read |
| stiff-boots | Finding gear | GPT-6 Luna | 3 | Right | 2.4 s | Read |
| stiff-boots | Finding gear | GPT-6 Sol | 3 | Right | 1.2 s | Read |
| stiff-boots | Finding gear | Claude Haiku 5.5 | 3 | Right | 2.6 s | Read |
| stiff-boots | Finding gear | Claude Sonnet 5 | 3 | Right | 1.2 s | Read |
| stiff-boots | Finding gear | Claude Sonnet 5.5 | 3 | Right | 2.2 s | Read |
| kids-helmet | Finding gear | GPT-6.1 Sol | 1 | Right | 1.6 s | Read |
| kids-helmet | Finding gear | GPT-6 Luna | 1 | Right | 2.4 s | Read |
| kids-helmet | Finding gear | GPT-6 Sol | 1 | Right | 1.1 s | Read |
| kids-helmet | Finding gear | Claude Haiku 5.5 | 1 | Right | 2.1 s | Read |
| kids-helmet | Finding gear | Claude Sonnet 5 | 1 | Right | 1.1 s | Read |
| kids-helmet | Finding gear | Claude Sonnet 5.5 | 1 | Right | 4.0 s | Read |
| kids-helmet | Finding gear | GPT-6.1 Sol | 2 | Right | 1.6 s | Read |
| kids-helmet | Finding gear | GPT-6 Luna | 2 | Right | 3.0 s | Read |
| kids-helmet | Finding gear | GPT-6 Sol | 2 | Right | 1.0 s | Read |
| kids-helmet | Finding gear | Claude Haiku 5.5 | 2 | Right | 2.2 s | Read |
| kids-helmet | Finding gear | Claude Sonnet 5 | 2 | Right | 1.0 s | Read |
| kids-helmet | Finding gear | Claude Sonnet 5.5 | 2 | Right | 2.0 s | Read |
| kids-helmet | Finding gear | GPT-6.1 Sol | 3 | Right | 1.3 s | Read |
| kids-helmet | Finding gear | GPT-6 Luna | 3 | Right | 2.3 s | Read |
| kids-helmet | Finding gear | GPT-6 Sol | 3 | Right | 1.5 s | Read |
| kids-helmet | Finding gear | Claude Haiku 5.5 | 3 | Right | 2.0 s | Read |
| kids-helmet | Finding gear | Claude Sonnet 5 | 3 | Right | 1.2 s | Read |
| kids-helmet | Finding gear | Claude Sonnet 5.5 | 3 | Right | 3.4 s | Read |
| flat-light-goggles | Finding gear | GPT-6.1 Sol | 1 | Right | 1.7 s | Read |
| flat-light-goggles | Finding gear | GPT-6 Luna | 1 | Right | 2.5 s | Read |
| flat-light-goggles | Finding gear | GPT-6 Sol | 1 | Right | 2.3 s | Read |
| flat-light-goggles | Finding gear | Claude Haiku 5.5 | 1 | Right | 1.1 s | Read |
| flat-light-goggles | Finding gear | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| flat-light-goggles | Finding gear | Claude Sonnet 5.5 | 1 | Right | 1.1 s | Read |
| flat-light-goggles | Finding gear | GPT-6.1 Sol | 2 | Right | 1.9 s | Read |
| flat-light-goggles | Finding gear | GPT-6 Luna | 2 | Right | 2.3 s | Read |
| flat-light-goggles | Finding gear | GPT-6 Sol | 2 | Right | 1.0 s | Read |
| flat-light-goggles | Finding gear | Claude Haiku 5.5 | 2 | Right | 1.3 s | Read |
| flat-light-goggles | Finding gear | Claude Sonnet 5 | 2 | Right | 1.2 s | Read |
| flat-light-goggles | Finding gear | Claude Sonnet 5.5 | 2 | Right | 1.1 s | Read |
| flat-light-goggles | Finding gear | GPT-6.1 Sol | 3 | Right | 2.2 s | Read |
| flat-light-goggles | Finding gear | GPT-6 Luna | 3 | Right | 2.8 s | Read |
| flat-light-goggles | Finding gear | GPT-6 Sol | 3 | Right | 1.2 s | Read |
| flat-light-goggles | Finding gear | Claude Haiku 5.5 | 3 | Right | 3.0 s | Read |
| flat-light-goggles | Finding gear | Claude Sonnet 5 | 3 | Right | 1.1 s | Read |
| flat-light-goggles | Finding gear | Claude Sonnet 5.5 | 3 | Right | 1.4 s | Read |
| cold-first-day | Finding gear | GPT-6.1 Sol | 1 | Right | 1.7 s | Read |
| cold-first-day | Finding gear | GPT-6 Luna | 1 | Right | 2.6 s | Read |
| cold-first-day | Finding gear | GPT-6 Sol | 1 | Right | 1.2 s | Read |
| cold-first-day | Finding gear | Claude Haiku 5.5 | 1 | Right | 3.1 s | Read |
| cold-first-day | Finding gear | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| cold-first-day | Finding gear | Claude Sonnet 5.5 | 1 | Right | 2.4 s | Read |
| cold-first-day | Finding gear | GPT-6.1 Sol | 2 | Right | 2.0 s | Read |
| cold-first-day | Finding gear | GPT-6 Luna | 2 | Right | 3.3 s | Read |
| cold-first-day | Finding gear | GPT-6 Sol | 2 | Right | 1.2 s | Read |
| cold-first-day | Finding gear | Claude Haiku 5.5 | 2 | Right | 2.5 s | Read |
| cold-first-day | Finding gear | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| cold-first-day | Finding gear | Claude Sonnet 5.5 | 2 | Right | 2.5 s | Read |
| cold-first-day | Finding gear | GPT-6.1 Sol | 3 | Right | 1.8 s | Read |
| cold-first-day | Finding gear | GPT-6 Luna | 3 | Right | 2.5 s | Read |
| cold-first-day | Finding gear | GPT-6 Sol | 3 | Right | 1.9 s | Read |
| cold-first-day | Finding gear | Claude Haiku 5.5 | 3 | Right | 2.0 s | Read |
| cold-first-day | Finding gear | Claude Sonnet 5 | 3 | Right | 1.1 s | Read |
| cold-first-day | Finding gear | Claude Sonnet 5.5 | 3 | Right | 1.2 s | Read |
| cheapest-beginner-pair | Finding gear | GPT-6.1 Sol | 1 | Right | 2.1 s | Read |
| cheapest-beginner-pair | Finding gear | GPT-6 Luna | 1 | Wrong | 5.6 s | Read |
| cheapest-beginner-pair | Finding gear | GPT-6 Sol | 1 | Right | 2.6 s | Read |
| cheapest-beginner-pair | Finding gear | Claude Haiku 5.5 | 1 | Right | 3.7 s | Read |
| cheapest-beginner-pair | Finding gear | Claude Sonnet 5 | 1 | Right | 7.7 s | Read |
| cheapest-beginner-pair | Finding gear | Claude Sonnet 5.5 | 1 | Right | 5.6 s | Read |
| cheapest-beginner-pair | Finding gear | GPT-6.1 Sol | 2 | Right | 2.2 s | Read |
| cheapest-beginner-pair | Finding gear | GPT-6 Luna | 2 | Right | 4.9 s | Read |
| cheapest-beginner-pair | Finding gear | GPT-6 Sol | 2 | Right | 2.1 s | Read |
| cheapest-beginner-pair | Finding gear | Claude Haiku 5.5 | 2 | Right | 3.3 s | Read |
| cheapest-beginner-pair | Finding gear | Claude Sonnet 5 | 2 | Right | 1.8 s | Read |
| cheapest-beginner-pair | Finding gear | Claude Sonnet 5.5 | 2 | Right | 3.8 s | Read |
| cheapest-beginner-pair | Finding gear | GPT-6.1 Sol | 3 | Right | 3.9 s | Read |
| cheapest-beginner-pair | Finding gear | GPT-6 Luna | 3 | Wrong | 4.9 s | Read |
| cheapest-beginner-pair | Finding gear | GPT-6 Sol | 3 | Right | 3.1 s | Read |
| cheapest-beginner-pair | Finding gear | Claude Haiku 5.5 | 3 | Right | 3.6 s | Read |
| cheapest-beginner-pair | Finding gear | Claude Sonnet 5 | 3 | Right | 6.0 s | Read |
| cheapest-beginner-pair | Finding gear | Claude Sonnet 5.5 | 3 | Right | 3.6 s | Read |
| splitboarding | Finding gear | GPT-6.1 Sol | 1 | Wrong | 3.5 s | Read |
| splitboarding | Finding gear | GPT-6 Luna | 1 | Right | 5.4 s | Read |
| splitboarding | Finding gear | GPT-6 Sol | 1 | Right | 2.8 s | Read |
| splitboarding | Finding gear | Claude Haiku 5.5 | 1 | Right | 3.4 s | Read |
| splitboarding | Finding gear | Claude Sonnet 5 | 1 | Right | 10.5 s | Read |
| splitboarding | Finding gear | Claude Sonnet 5.5 | 1 | Right | 11.1 s | Read |
| splitboarding | Finding gear | GPT-6.1 Sol | 2 | Wrong | 4.7 s | Read |
| splitboarding | Finding gear | GPT-6 Luna | 2 | Right | 2.5 s | Read |
| splitboarding | Finding gear | GPT-6 Sol | 2 | Wrong | 2.5 s | Read |
| splitboarding | Finding gear | Claude Haiku 5.5 | 2 | Right | 3.7 s | Read |
| splitboarding | Finding gear | Claude Sonnet 5 | 2 | Right | 4.6 s | Read |
| splitboarding | Finding gear | Claude Sonnet 5.5 | 2 | Right | 9.9 s | Read |
| splitboarding | Finding gear | GPT-6.1 Sol | 3 | Wrong | 3.9 s | Read |
| splitboarding | Finding gear | GPT-6 Luna | 3 | Right | 4.3 s | Read |
| splitboarding | Finding gear | GPT-6 Sol | 3 | Wrong | 3.1 s | Read |
| splitboarding | Finding gear | Claude Haiku 5.5 | 3 | Right | 4.2 s | Read |
| splitboarding | Finding gear | Claude Sonnet 5 | 3 | Right | 3.7 s | Read |
| splitboarding | Finding gear | Claude Sonnet 5.5 | 3 | Right | 11.2 s | Read |
| gift-under-50 | Finding gear | GPT-6.1 Sol | 1 | Right | 4.3 s | Read |
| gift-under-50 | Finding gear | GPT-6 Luna | 1 | Right | 2.5 s | Read |
| gift-under-50 | Finding gear | GPT-6 Sol | 1 | Right | 2.6 s | Read |
| gift-under-50 | Finding gear | Claude Haiku 5.5 | 1 | Right | 4.0 s | Read |
| gift-under-50 | Finding gear | Claude Sonnet 5 | 1 | Right | 3.2 s | Read |
| gift-under-50 | Finding gear | Claude Sonnet 5.5 | 1 | Right | 4.3 s | Read |
| gift-under-50 | Finding gear | GPT-6.1 Sol | 2 | Right | 3.9 s | Read |
| gift-under-50 | Finding gear | GPT-6 Luna | 2 | Right | 3.7 s | Read |
| gift-under-50 | Finding gear | GPT-6 Sol | 2 | Right | 3.7 s | Read |
| gift-under-50 | Finding gear | Claude Haiku 5.5 | 2 | Right | 2.9 s | Read |
| gift-under-50 | Finding gear | Claude Sonnet 5 | 2 | Right | 3.2 s | Read |
| gift-under-50 | Finding gear | Claude Sonnet 5.5 | 2 | Right | 4.1 s | Read |
| gift-under-50 | Finding gear | GPT-6.1 Sol | 3 | Right | 4.5 s | Read |
| gift-under-50 | Finding gear | GPT-6 Luna | 3 | Right | 2.8 s | Read |
| gift-under-50 | Finding gear | GPT-6 Sol | 3 | Right | 3.9 s | Read |
| gift-under-50 | Finding gear | Claude Haiku 5.5 | 3 | Right | 3.4 s | Read |
| gift-under-50 | Finding gear | Claude Sonnet 5 | 3 | Right | 2.9 s | Read |
| gift-under-50 | Finding gear | Claude Sonnet 5.5 | 3 | Right | 4.8 s | Read |
| goggles-and-lock | Finding gear | GPT-6.1 Sol | 1 | Right | 1.5 s | Read |
| goggles-and-lock | Finding gear | GPT-6 Luna | 1 | Right | 2.7 s | Read |
| goggles-and-lock | Finding gear | GPT-6 Sol | 1 | Right | 1.2 s | Read |
| goggles-and-lock | Finding gear | Claude Haiku 5.5 | 1 | Right | 2.1 s | Read |
| goggles-and-lock | Finding gear | Claude Sonnet 5 | 1 | Right | 1.9 s | Read |
| goggles-and-lock | Finding gear | Claude Sonnet 5.5 | 1 | Right | 3.1 s | Read |
| goggles-and-lock | Finding gear | GPT-6.1 Sol | 2 | Right | 1.8 s | Read |
| goggles-and-lock | Finding gear | GPT-6 Luna | 2 | Right | 3.2 s | Read |
| goggles-and-lock | Finding gear | GPT-6 Sol | 2 | Right | 1.2 s | Read |
| goggles-and-lock | Finding gear | Claude Haiku 5.5 | 2 | Right | 2.4 s | Read |
| goggles-and-lock | Finding gear | Claude Sonnet 5 | 2 | Right | 2.0 s | Read |
| goggles-and-lock | Finding gear | Claude Sonnet 5.5 | 2 | Right | 2.8 s | Read |
| goggles-and-lock | Finding gear | GPT-6.1 Sol | 3 | Right | 2.3 s | Read |
| goggles-and-lock | Finding gear | GPT-6 Luna | 3 | Right | 3.5 s | Read |
| goggles-and-lock | Finding gear | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| goggles-and-lock | Finding gear | Claude Haiku 5.5 | 3 | Right | 2.4 s | Read |
| goggles-and-lock | Finding gear | Claude Sonnet 5 | 3 | Right | 3.3 s | Read |
| goggles-and-lock | Finding gear | Claude Sonnet 5.5 | 3 | Right | 1.5 s | Read |
| skis | Finding gear | GPT-6.1 Sol | 1 | Wrong | 1.6 s | Read |
| skis | Finding gear | GPT-6 Luna | 1 | Wrong | 1.2 s | Read |
| skis | Finding gear | GPT-6 Sol | 1 | Wrong | 1.1 s | Read |
| skis | Finding gear | Claude Haiku 5.5 | 1 | Wrong | 3.2 s | Read |
| skis | Finding gear | Claude Sonnet 5 | 1 | Wrong | 1.1 s | Read |
| skis | Finding gear | Claude Sonnet 5.5 | 1 | Right | 4.8 s | Read |
| skis | Finding gear | GPT-6.1 Sol | 2 | Wrong | 1.4 s | Read |
| skis | Finding gear | GPT-6 Luna | 2 | Wrong | 1.1 s | Read |
| skis | Finding gear | GPT-6 Sol | 2 | Wrong | 1.0 s | Read |
| skis | Finding gear | Claude Haiku 5.5 | 2 | Right | 3.1 s | Read |
| skis | Finding gear | Claude Sonnet 5 | 2 | Wrong | 1.1 s | Read |
| skis | Finding gear | Claude Sonnet 5.5 | 2 | Right | 4.9 s | Read |
| skis | Finding gear | GPT-6.1 Sol | 3 | Wrong | 2.0 s | Read |
| skis | Finding gear | GPT-6 Luna | 3 | Wrong | 1.1 s | Read |
| skis | Finding gear | GPT-6 Sol | 3 | Wrong | 1.1 s | Read |
| skis | Finding gear | Claude Haiku 5.5 | 3 | Wrong | 3.3 s | Read |
| skis | Finding gear | Claude Sonnet 5 | 3 | Right | 1.9 s | Read |
| skis | Finding gear | Claude Sonnet 5.5 | 3 | Right | 4.9 s | Read |
| burton | Finding gear | GPT-6.1 Sol | 1 | Wrong | 2.2 s | Read |
| burton | Finding gear | GPT-6 Luna | 1 | Wrong | 2.8 s | Read |
| burton | Finding gear | GPT-6 Sol | 1 | Wrong | 2.2 s | Read |
| burton | Finding gear | Claude Haiku 5.5 | 1 | Wrong | 4.0 s | Read |
| burton | Finding gear | Claude Sonnet 5 | 1 | Wrong | 1.6 s | Read |
| burton | Finding gear | Claude Sonnet 5.5 | 1 | Right | 4.1 s | Read |
| burton | Finding gear | GPT-6.1 Sol | 2 | Wrong | 1.5 s | Read |
| burton | Finding gear | GPT-6 Luna | 2 | Wrong | 3.1 s | Read |
| burton | Finding gear | GPT-6 Sol | 2 | Wrong | 1.5 s | Read |
| burton | Finding gear | Claude Haiku 5.5 | 2 | Wrong | 3.1 s | Read |
| burton | Finding gear | Claude Sonnet 5 | 2 | Wrong | 1.7 s | Read |
| burton | Finding gear | Claude Sonnet 5.5 | 2 | Wrong | 4.8 s | Read |
| burton | Finding gear | GPT-6.1 Sol | 3 | Wrong | 2.0 s | Read |
| burton | Finding gear | GPT-6 Luna | 3 | Wrong | 5.2 s | Read |
| burton | Finding gear | GPT-6 Sol | 3 | Wrong | 1.1 s | Read |
| burton | Finding gear | Claude Haiku 5.5 | 3 | Wrong | 3.2 s | Read |
| burton | Finding gear | Claude Sonnet 5 | 3 | Wrong | 1.9 s | Read |
| burton | Finding gear | Claude Sonnet 5.5 | 3 | Right | 3.4 s | Read |
| helmet-over-goggles | Comparing products | GPT-6.1 Sol | 1 | Right | 1.9 s | Read |
| helmet-over-goggles | Comparing products | GPT-6 Luna | 1 | Right | 2.7 s | Read |
| helmet-over-goggles | Comparing products | GPT-6 Sol | 1 | Right | 1.0 s | Read |
| helmet-over-goggles | Comparing products | Claude Haiku 5.5 | 1 | Right | 1.9 s | Read |
| helmet-over-goggles | Comparing products | Claude Sonnet 5 | 1 | Right | 1.1 s | Read |
| helmet-over-goggles | Comparing products | Claude Sonnet 5.5 | 1 | Right | 1.0 s | Read |
| helmet-over-goggles | Comparing products | GPT-6.1 Sol | 2 | Right | 1.6 s | Read |
| helmet-over-goggles | Comparing products | GPT-6 Luna | 2 | Right | 4.6 s | Read |
| helmet-over-goggles | Comparing products | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| helmet-over-goggles | Comparing products | Claude Haiku 5.5 | 2 | Wrong | 2.0 s | Read |
| helmet-over-goggles | Comparing products | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| helmet-over-goggles | Comparing products | Claude Sonnet 5.5 | 2 | Right | 1.4 s | Read |
| helmet-over-goggles | Comparing products | GPT-6.1 Sol | 3 | Right | 1.6 s | Read |
| helmet-over-goggles | Comparing products | GPT-6 Luna | 3 | Right | 4.5 s | Read |
| helmet-over-goggles | Comparing products | GPT-6 Sol | 3 | Right | 1.0 s | Read |
| helmet-over-goggles | Comparing products | Claude Haiku 5.5 | 3 | Right | 2.3 s | Read |
| helmet-over-goggles | Comparing products | Claude Sonnet 5 | 3 | Right | 1.1 s | Read |
| helmet-over-goggles | Comparing products | Claude Sonnet 5.5 | 3 | Right | 1.9 s | Read |
| sentinel-vs-aurora | Comparing products | GPT-6.1 Sol | 1 | Right | 2.9 s | Read |
| sentinel-vs-aurora | Comparing products | GPT-6 Luna | 1 | Right | 1.1 s | Read |
| sentinel-vs-aurora | Comparing products | GPT-6 Sol | 1 | Right | 1.0 s | Read |
| sentinel-vs-aurora | Comparing products | Claude Haiku 5.5 | 1 | Wrong | 1.8 s | Read |
| sentinel-vs-aurora | Comparing products | Claude Sonnet 5 | 1 | Wrong | 1.2 s | Read |
| sentinel-vs-aurora | Comparing products | Claude Sonnet 5.5 | 1 | Wrong | 1.4 s | Read |
| sentinel-vs-aurora | Comparing products | GPT-6.1 Sol | 2 | Right | 1.4 s | Read |
| sentinel-vs-aurora | Comparing products | GPT-6 Luna | 2 | Wrong | 2.3 s | Read |
| sentinel-vs-aurora | Comparing products | GPT-6 Sol | 2 | Right | 1.0 s | Read |
| sentinel-vs-aurora | Comparing products | Claude Haiku 5.5 | 2 | Right | 2.1 s | Read |
| sentinel-vs-aurora | Comparing products | Claude Sonnet 5 | 2 | Right | 1.2 s | Read |
| sentinel-vs-aurora | Comparing products | Claude Sonnet 5.5 | 2 | Wrong | 2.2 s | Read |
| sentinel-vs-aurora | Comparing products | GPT-6.1 Sol | 3 | Right | 2.2 s | Read |
| sentinel-vs-aurora | Comparing products | GPT-6 Luna | 3 | Right | 2.7 s | Read |
| sentinel-vs-aurora | Comparing products | GPT-6 Sol | 3 | Right | 1.9 s | Read |
| sentinel-vs-aurora | Comparing products | Claude Haiku 5.5 | 3 | Right | 2.3 s | Read |
| sentinel-vs-aurora | Comparing products | Claude Sonnet 5 | 3 | Wrong | 1.3 s | Read |
| sentinel-vs-aurora | Comparing products | Claude Sonnet 5.5 | 3 | Right | 2.6 s | Read |
| ember-vs-vertex | Comparing products | GPT-6.1 Sol | 1 | Right | 3.1 s | Read |
| ember-vs-vertex | Comparing products | GPT-6 Luna | 1 | Wrong | 1.3 s | Read |
| ember-vs-vertex | Comparing products | GPT-6 Sol | 1 | Right | 1.6 s | Read |
| ember-vs-vertex | Comparing products | Claude Haiku 5.5 | 1 | Right | 2.3 s | Read |
| ember-vs-vertex | Comparing products | Claude Sonnet 5 | 1 | Right | 1.3 s | Read |
| ember-vs-vertex | Comparing products | Claude Sonnet 5.5 | 1 | Right | 1.4 s | Read |
| ember-vs-vertex | Comparing products | GPT-6.1 Sol | 2 | Right | 1.5 s | Read |
| ember-vs-vertex | Comparing products | GPT-6 Luna | 2 | Right | 1.2 s | Read |
| ember-vs-vertex | Comparing products | GPT-6 Sol | 2 | Right | 1.2 s | Read |
| ember-vs-vertex | Comparing products | Claude Haiku 5.5 | 2 | Right | 3.2 s | Read |
| ember-vs-vertex | Comparing products | Claude Sonnet 5 | 2 | Right | 1.3 s | Read |
| ember-vs-vertex | Comparing products | Claude Sonnet 5.5 | 2 | Right | 1.3 s | Read |
| ember-vs-vertex | Comparing products | GPT-6.1 Sol | 3 | Right | 1.9 s | Read |
| ember-vs-vertex | Comparing products | GPT-6 Luna | 3 | Right | 1.4 s | Read |
| ember-vs-vertex | Comparing products | GPT-6 Sol | 3 | Right | 1.7 s | Read |
| ember-vs-vertex | Comparing products | Claude Haiku 5.5 | 3 | Right | 1.8 s | Read |
| ember-vs-vertex | Comparing products | Claude Sonnet 5 | 3 | Right | 1.2 s | Read |
| ember-vs-vertex | Comparing products | Claude Sonnet 5.5 | 3 | Right | 1.5 s | Read |
| prism-vs-halo | Comparing products | GPT-6.1 Sol | 1 | Right | 1.8 s | Read |
| prism-vs-halo | Comparing products | GPT-6 Luna | 1 | Right | 2.5 s | Read |
| prism-vs-halo | Comparing products | GPT-6 Sol | 1 | Wrong | 1.1 s | Read |
| prism-vs-halo | Comparing products | Claude Haiku 5.5 | 1 | Right | 2.3 s | Read |
| prism-vs-halo | Comparing products | Claude Sonnet 5 | 1 | Right | 2.3 s | Read |
| prism-vs-halo | Comparing products | Claude Sonnet 5.5 | 1 | Right | 3.0 s | Read |
| prism-vs-halo | Comparing products | GPT-6.1 Sol | 2 | Right | 1.8 s | Read |
| prism-vs-halo | Comparing products | GPT-6 Luna | 2 | Right | 2.7 s | Read |
| prism-vs-halo | Comparing products | GPT-6 Sol | 2 | Right | 1.6 s | Read |
| prism-vs-halo | Comparing products | Claude Haiku 5.5 | 2 | Right | 2.3 s | Read |
| prism-vs-halo | Comparing products | Claude Sonnet 5 | 2 | Right | 2.9 s | Read |
| prism-vs-halo | Comparing products | Claude Sonnet 5.5 | 2 | Right | 2.0 s | Read |
| prism-vs-halo | Comparing products | GPT-6.1 Sol | 3 | Right | 1.4 s | Read |
| prism-vs-halo | Comparing products | GPT-6 Luna | 3 | Right | 2.5 s | Read |
| prism-vs-halo | Comparing products | GPT-6 Sol | 3 | Right | 1.4 s | Read |
| prism-vs-halo | Comparing products | Claude Haiku 5.5 | 3 | Right | 2.3 s | Read |
| prism-vs-halo | Comparing products | Claude Sonnet 5 | 3 | Right | 1.6 s | Read |
| prism-vs-halo | Comparing products | Claude Sonnet 5.5 | 3 | Right | 2.8 s | Read |
| complete-vs-compare | Comparing products | GPT-6.1 Sol | 1 | Right | 1.7 s | Read |
| complete-vs-compare | Comparing products | GPT-6 Luna | 1 | Wrong | 2.7 s | Read |
| complete-vs-compare | Comparing products | GPT-6 Sol | 1 | Right | 1.0 s | Read |
| complete-vs-compare | Comparing products | Claude Haiku 5.5 | 1 | Right | 3.3 s | Read |
| complete-vs-compare | Comparing products | Claude Sonnet 5 | 1 | Wrong | 3.1 s | Read |
| complete-vs-compare | Comparing products | Claude Sonnet 5.5 | 1 | Right | 2.7 s | Read |
| complete-vs-compare | Comparing products | GPT-6.1 Sol | 2 | Right | 1.5 s | Read |
| complete-vs-compare | Comparing products | GPT-6 Luna | 2 | Right | 2.3 s | Read |
| complete-vs-compare | Comparing products | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| complete-vs-compare | Comparing products | Claude Haiku 5.5 | 2 | Right | 3.9 s | Read |
| complete-vs-compare | Comparing products | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| complete-vs-compare | Comparing products | Claude Sonnet 5.5 | 2 | Right | 2.7 s | Read |
| complete-vs-compare | Comparing products | GPT-6.1 Sol | 3 | Right | 1.6 s | Read |
| complete-vs-compare | Comparing products | GPT-6 Luna | 3 | Right | 2.6 s | Read |
| complete-vs-compare | Comparing products | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| complete-vs-compare | Comparing products | Claude Haiku 5.5 | 3 | Wrong | 2.2 s | Read |
| complete-vs-compare | Comparing products | Claude Sonnet 5 | 3 | Wrong | 2.5 s | Read |
| complete-vs-compare | Comparing products | Claude Sonnet 5.5 | 3 | Wrong | 2.8 s | Read |
| wax-or-kit | Comparing products | GPT-6.1 Sol | 1 | Wrong | 1.7 s | Read |
| wax-or-kit | Comparing products | GPT-6 Luna | 1 | Wrong | 2.7 s | Read |
| wax-or-kit | Comparing products | GPT-6 Sol | 1 | Wrong | 1.2 s | Read |
| wax-or-kit | Comparing products | Claude Haiku 5.5 | 1 | Right | 1.0 s | Read |
| wax-or-kit | Comparing products | Claude Sonnet 5 | 1 | Wrong | 3.4 s | Read |
| wax-or-kit | Comparing products | Claude Sonnet 5.5 | 1 | Right | 1.5 s | Read |
| wax-or-kit | Comparing products | GPT-6.1 Sol | 2 | Right | 1.8 s | Read |
| wax-or-kit | Comparing products | GPT-6 Luna | 2 | Wrong | 2.5 s | Read |
| wax-or-kit | Comparing products | GPT-6 Sol | 2 | Wrong | 1.2 s | Read |
| wax-or-kit | Comparing products | Claude Haiku 5.5 | 2 | Right | 1.1 s | Read |
| wax-or-kit | Comparing products | Claude Sonnet 5 | 2 | Wrong | 3.6 s | Read |
| wax-or-kit | Comparing products | Claude Sonnet 5.5 | 2 | Wrong | 1.8 s | Read |
| wax-or-kit | Comparing products | GPT-6.1 Sol | 3 | Right | 1.8 s | Read |
| wax-or-kit | Comparing products | GPT-6 Luna | 3 | Wrong | 1.1 s | Read |
| wax-or-kit | Comparing products | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| wax-or-kit | Comparing products | Claude Haiku 5.5 | 3 | Right | 0.9 s | Read |
| wax-or-kit | Comparing products | Claude Sonnet 5 | 3 | Right | 1.7 s | Read |
| wax-or-kit | Comparing products | Claude Sonnet 5.5 | 3 | Right | 3.1 s | Read |
| board-size | Sizing and fit | GPT-6.1 Sol | 1 | Wrong | 1.7 s | Read |
| board-size | Sizing and fit | GPT-6 Luna | 1 | Right | 2.3 s | Read |
| board-size | Sizing and fit | GPT-6 Sol | 1 | Wrong | 1.2 s | Read |
| board-size | Sizing and fit | Claude Haiku 5.5 | 1 | Right | 3.3 s | Read |
| board-size | Sizing and fit | Claude Sonnet 5 | 1 | Right | 2.3 s | Read |
| board-size | Sizing and fit | Claude Sonnet 5.5 | 1 | Right | 2.7 s | Read |
| board-size | Sizing and fit | GPT-6.1 Sol | 2 | Wrong | 1.6 s | Read |
| board-size | Sizing and fit | GPT-6 Luna | 2 | Right | 2.3 s | Read |
| board-size | Sizing and fit | GPT-6 Sol | 2 | Wrong | 1.1 s | Read |
| board-size | Sizing and fit | Claude Haiku 5.5 | 2 | Right | 3.1 s | Read |
| board-size | Sizing and fit | Claude Sonnet 5 | 2 | Right | 5.0 s | Read |
| board-size | Sizing and fit | Claude Sonnet 5.5 | 2 | Right | 5.0 s | Read |
| board-size | Sizing and fit | GPT-6.1 Sol | 3 | Wrong | 2.6 s | Read |
| board-size | Sizing and fit | GPT-6 Luna | 3 | Wrong | 1.0 s | Read |
| board-size | Sizing and fit | GPT-6 Sol | 3 | Wrong | 1.1 s | Read |
| board-size | Sizing and fit | Claude Haiku 5.5 | 3 | Right | 3.3 s | Read |
| board-size | Sizing and fit | Claude Sonnet 5 | 3 | Right | 5.7 s | Read |
| board-size | Sizing and fit | Claude Sonnet 5.5 | 3 | Right | 3.1 s | Read |
| boots-true-to-size | Sizing and fit | GPT-6.1 Sol | 1 | Right | 1.7 s | Read |
| boots-true-to-size | Sizing and fit | GPT-6 Luna | 1 | Wrong | 4.4 s | Read |
| boots-true-to-size | Sizing and fit | GPT-6 Sol | 1 | Wrong | 1.1 s | Read |
| boots-true-to-size | Sizing and fit | Claude Haiku 5.5 | 1 | Right | 3.1 s | Read |
| boots-true-to-size | Sizing and fit | Claude Sonnet 5 | 1 | Wrong | 1.4 s | Read |
| boots-true-to-size | Sizing and fit | Claude Sonnet 5.5 | 1 | Right | 3.3 s | Read |
| boots-true-to-size | Sizing and fit | GPT-6.1 Sol | 2 | Right | 2.5 s | Read |
| boots-true-to-size | Sizing and fit | GPT-6 Luna | 2 | Wrong | 3.6 s | Read |
| boots-true-to-size | Sizing and fit | GPT-6 Sol | 2 | Right | 1.2 s | Read |
| boots-true-to-size | Sizing and fit | Claude Haiku 5.5 | 2 | Right | 2.6 s | Read |
| boots-true-to-size | Sizing and fit | Claude Sonnet 5 | 2 | Wrong | 3.1 s | Read |
| boots-true-to-size | Sizing and fit | Claude Sonnet 5.5 | 2 | Right | 3.4 s | Read |
| boots-true-to-size | Sizing and fit | GPT-6.1 Sol | 3 | Right | 1.6 s | Read |
| boots-true-to-size | Sizing and fit | GPT-6 Luna | 3 | Right | 2.8 s | Read |
| boots-true-to-size | Sizing and fit | GPT-6 Sol | 3 | Right | 1.0 s | Read |
| boots-true-to-size | Sizing and fit | Claude Haiku 5.5 | 3 | Right | 2.7 s | Read |
| boots-true-to-size | Sizing and fit | Claude Sonnet 5 | 3 | Wrong | 1.0 s | Read |
| boots-true-to-size | Sizing and fit | Claude Sonnet 5.5 | 3 | Right | 3.4 s | Read |
| glide-fits-ember | Sizing and fit | GPT-6.1 Sol | 1 | Right | 1.6 s | Read |
| glide-fits-ember | Sizing and fit | GPT-6 Luna | 1 | Wrong | 3.5 s | Read |
| glide-fits-ember | Sizing and fit | GPT-6 Sol | 1 | Right | 1.1 s | Read |
| glide-fits-ember | Sizing and fit | Claude Haiku 5.5 | 1 | Right | 1.8 s | Read |
| glide-fits-ember | Sizing and fit | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| glide-fits-ember | Sizing and fit | Claude Sonnet 5.5 | 1 | Wrong | 1.1 s | Read |
| glide-fits-ember | Sizing and fit | GPT-6.1 Sol | 2 | Right | 2.5 s | Read |
| glide-fits-ember | Sizing and fit | GPT-6 Luna | 2 | Wrong | 3.7 s | Read |
| glide-fits-ember | Sizing and fit | GPT-6 Sol | 2 | Right | 2.3 s | Read |
| glide-fits-ember | Sizing and fit | Claude Haiku 5.5 | 2 | Right | 3.8 s | Read |
| glide-fits-ember | Sizing and fit | Claude Sonnet 5 | 2 | Right | 1.6 s | Read |
| glide-fits-ember | Sizing and fit | Claude Sonnet 5.5 | 2 | Wrong | 1.4 s | Read |
| glide-fits-ember | Sizing and fit | GPT-6.1 Sol | 3 | Right | 1.4 s | Read |
| glide-fits-ember | Sizing and fit | GPT-6 Sol | 3 | Right | 1.0 s | Read |
| glide-fits-ember | Sizing and fit | Claude Haiku 5.5 | 3 | Right | 1.8 s | Read |
| glide-fits-ember | Sizing and fit | Claude Sonnet 5 | 3 | Right | 1.4 s | Read |
| glide-fits-ember | Sizing and fit | Claude Sonnet 5.5 | 3 | Wrong | 1.0 s | Read |
| jacket-sizes | Sizing and fit | GPT-6.1 Sol | 1 | Right | 1.6 s | Read |
| jacket-sizes | Sizing and fit | GPT-6 Luna | 1 | Right | 3.5 s | Read |
| jacket-sizes | Sizing and fit | GPT-6 Sol | 1 | Right | 1.4 s | Read |
| jacket-sizes | Sizing and fit | Claude Haiku 5.5 | 1 | Right | 1.6 s | Read |
| jacket-sizes | Sizing and fit | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| jacket-sizes | Sizing and fit | Claude Sonnet 5.5 | 1 | Right | 1.8 s | Read |
| jacket-sizes | Sizing and fit | GPT-6.1 Sol | 2 | Right | 1.5 s | Read |
| jacket-sizes | Sizing and fit | GPT-6 Luna | 2 | Right | 2.2 s | Read |
| jacket-sizes | Sizing and fit | GPT-6 Sol | 2 | Right | 1.2 s | Read |
| jacket-sizes | Sizing and fit | Claude Haiku 5.5 | 2 | Right | 1.0 s | Read |
| jacket-sizes | Sizing and fit | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| jacket-sizes | Sizing and fit | Claude Sonnet 5.5 | 2 | Right | 1.9 s | Read |
| jacket-sizes | Sizing and fit | GPT-6.1 Sol | 3 | Right | 1.7 s | Read |
| jacket-sizes | Sizing and fit | GPT-6 Luna | 3 | Right | 2.4 s | Read |
| jacket-sizes | Sizing and fit | GPT-6 Sol | 3 | Right | 1.2 s | Read |
| jacket-sizes | Sizing and fit | Claude Haiku 5.5 | 3 | Right | 0.9 s | Read |
| jacket-sizes | Sizing and fit | Claude Sonnet 5 | 3 | Right | 1.1 s | Read |
| jacket-sizes | Sizing and fit | Claude Sonnet 5.5 | 3 | Right | 1.2 s | Read |
| board-bag-fit | Sizing and fit | GPT-6.1 Sol | 1 | Right | 1.8 s | Read |
| board-bag-fit | Sizing and fit | GPT-6 Luna | 1 | Right | 3.0 s | Read |
| board-bag-fit | Sizing and fit | GPT-6 Sol | 1 | Right | 1.5 s | Read |
| board-bag-fit | Sizing and fit | Claude Haiku 5.5 | 1 | Right | 2.0 s | Read |
| board-bag-fit | Sizing and fit | Claude Sonnet 5 | 1 | Right | 1.1 s | Read |
| board-bag-fit | Sizing and fit | Claude Sonnet 5.5 | 1 | Right | 1.1 s | Read |
| board-bag-fit | Sizing and fit | GPT-6.1 Sol | 2 | Right | 1.6 s | Read |
| board-bag-fit | Sizing and fit | GPT-6 Luna | 2 | Right | 2.3 s | Read |
| board-bag-fit | Sizing and fit | GPT-6 Sol | 2 | Right | 1.0 s | Read |
| board-bag-fit | Sizing and fit | Claude Haiku 5.5 | 2 | Right | 1.5 s | Read |
| board-bag-fit | Sizing and fit | Claude Sonnet 5 | 2 | Right | 2.1 s | Read |
| board-bag-fit | Sizing and fit | Claude Sonnet 5.5 | 2 | Right | 1.4 s | Read |
| board-bag-fit | Sizing and fit | GPT-6.1 Sol | 3 | Right | 2.2 s | Read |
| board-bag-fit | Sizing and fit | GPT-6 Luna | 3 | Right | 3.1 s | Read |
| board-bag-fit | Sizing and fit | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| board-bag-fit | Sizing and fit | Claude Haiku 5.5 | 3 | Right | 2.6 s | Read |
| board-bag-fit | Sizing and fit | Claude Sonnet 5 | 3 | Right | 1.1 s | Read |
| board-bag-fit | Sizing and fit | Claude Sonnet 5.5 | 3 | Right | 1.4 s | Read |
| kids-helmet-age | Sizing and fit | GPT-6.1 Sol | 1 | Right | 1.5 s | Read |
| kids-helmet-age | Sizing and fit | GPT-6 Luna | 1 | Right | 5.3 s | Read |
| kids-helmet-age | Sizing and fit | GPT-6 Sol | 1 | Right | 1.8 s | Read |
| kids-helmet-age | Sizing and fit | Claude Haiku 5.5 | 1 | Right | 2.0 s | Read |
| kids-helmet-age | Sizing and fit | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| kids-helmet-age | Sizing and fit | Claude Sonnet 5.5 | 1 | Right | 4.6 s | Read |
| kids-helmet-age | Sizing and fit | GPT-6.1 Sol | 2 | Right | 1.6 s | Read |
| kids-helmet-age | Sizing and fit | GPT-6 Luna | 2 | Right | 4.3 s | Read |
| kids-helmet-age | Sizing and fit | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| kids-helmet-age | Sizing and fit | Claude Haiku 5.5 | 2 | Right | 2.5 s | Read |
| kids-helmet-age | Sizing and fit | Claude Sonnet 5 | 2 | Right | 1.3 s | Read |
| kids-helmet-age | Sizing and fit | Claude Sonnet 5.5 | 2 | Right | 3.0 s | Read |
| kids-helmet-age | Sizing and fit | GPT-6.1 Sol | 3 | Right | 1.9 s | Read |
| kids-helmet-age | Sizing and fit | GPT-6 Luna | 3 | Right | 4.0 s | Read |
| kids-helmet-age | Sizing and fit | GPT-6 Sol | 3 | Wrong | 1.9 s | Read |
| kids-helmet-age | Sizing and fit | Claude Haiku 5.5 | 3 | Right | 2.2 s | Read |
| kids-helmet-age | Sizing and fit | Claude Sonnet 5 | 3 | Right | 1.0 s | Read |
| kids-helmet-age | Sizing and fit | Claude Sonnet 5.5 | 3 | Right | 3.1 s | Read |
| gloves-xl | Sizing and fit | GPT-6.1 Sol | 1 | Right | 3.7 s | Read |
| gloves-xl | Sizing and fit | GPT-6 Luna | 1 | Right | 4.6 s | Read |
| gloves-xl | Sizing and fit | GPT-6 Sol | 1 | Right | 1.1 s | Read |
| gloves-xl | Sizing and fit | Claude Haiku 5.5 | 1 | Right | 2.2 s | Read |
| gloves-xl | Sizing and fit | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| gloves-xl | Sizing and fit | Claude Sonnet 5.5 | 1 | Right | 3.3 s | Read |
| gloves-xl | Sizing and fit | GPT-6.1 Sol | 2 | Right | 3.4 s | Read |
| gloves-xl | Sizing and fit | GPT-6 Luna | 2 | Right | 2.7 s | Read |
| gloves-xl | Sizing and fit | GPT-6 Sol | 2 | Right | 1.0 s | Read |
| gloves-xl | Sizing and fit | Claude Haiku 5.5 | 2 | Right | 2.7 s | Read |
| gloves-xl | Sizing and fit | Claude Sonnet 5 | 2 | Right | 2.6 s | Read |
| gloves-xl | Sizing and fit | Claude Sonnet 5.5 | 2 | Right | 2.0 s | Read |
| gloves-xl | Sizing and fit | GPT-6.1 Sol | 3 | Right | 4.0 s | Read |
| gloves-xl | Sizing and fit | GPT-6 Luna | 3 | Right | 3.8 s | Read |
| gloves-xl | Sizing and fit | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| gloves-xl | Sizing and fit | Claude Haiku 5.5 | 3 | Right | 2.1 s | Read |
| gloves-xl | Sizing and fit | Claude Sonnet 5 | 3 | Right | 2.7 s | Read |
| gloves-xl | Sizing and fit | Claude Sonnet 5.5 | 3 | Right | 2.5 s | Read |
| ember-us-12 | Sizing and fit | GPT-6.1 Sol | 1 | Right | 2.0 s | Read |
| ember-us-12 | Sizing and fit | GPT-6 Luna | 1 | Right | 4.0 s | Read |
| ember-us-12 | Sizing and fit | GPT-6 Sol | 1 | Right | 1.3 s | Read |
| ember-us-12 | Sizing and fit | Claude Haiku 5.5 | 1 | Right | 2.4 s | Read |
| ember-us-12 | Sizing and fit | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| ember-us-12 | Sizing and fit | Claude Sonnet 5.5 | 1 | Right | 3.4 s | Read |
| ember-us-12 | Sizing and fit | GPT-6.1 Sol | 2 | Right | 1.7 s | Read |
| ember-us-12 | Sizing and fit | GPT-6 Luna | 2 | Right | 3.5 s | Read |
| ember-us-12 | Sizing and fit | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| ember-us-12 | Sizing and fit | Claude Haiku 5.5 | 2 | Right | 2.7 s | Read |
| ember-us-12 | Sizing and fit | Claude Sonnet 5 | 2 | Right | 1.9 s | Read |
| ember-us-12 | Sizing and fit | Claude Sonnet 5.5 | 2 | Right | 2.4 s | Read |
| ember-us-12 | Sizing and fit | GPT-6.1 Sol | 3 | Right | 1.8 s | Read |
| ember-us-12 | Sizing and fit | GPT-6 Luna | 3 | Right | 3.5 s | Read |
| ember-us-12 | Sizing and fit | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| ember-us-12 | Sizing and fit | Claude Haiku 5.5 | 3 | Right | 4.3 s | Read |
| ember-us-12 | Sizing and fit | Claude Sonnet 5 | 3 | Right | 1.3 s | Read |
| ember-us-12 | Sizing and fit | Claude Sonnet 5.5 | 3 | Right | 3.4 s | Read |
| on-sale | Stock and price | GPT-6.1 Sol | 1 | Right | 3.2 s | Read |
| on-sale | Stock and price | GPT-6 Luna | 1 | Right | 2.4 s | Read |
| on-sale | Stock and price | GPT-6 Sol | 1 | Right | 2.4 s | Read |
| on-sale | Stock and price | Claude Haiku 5.5 | 1 | Wrong | 2.4 s | Read |
| on-sale | Stock and price | Claude Sonnet 5 | 1 | Right | 2.4 s | Read |
| on-sale | Stock and price | Claude Sonnet 5.5 | 1 | Right | 3.3 s | Read |
| on-sale | Stock and price | GPT-6.1 Sol | 2 | Right | 3.1 s | Read |
| on-sale | Stock and price | GPT-6 Luna | 2 | Right | 2.8 s | Read |
| on-sale | Stock and price | GPT-6 Sol | 2 | Right | 2.3 s | Read |
| on-sale | Stock and price | Claude Haiku 5.5 | 2 | Wrong | 2.0 s | Read |
| on-sale | Stock and price | Claude Sonnet 5 | 2 | Right | 2.7 s | Read |
| on-sale | Stock and price | Claude Sonnet 5.5 | 2 | Right | 3.7 s | Read |
| on-sale | Stock and price | GPT-6.1 Sol | 3 | Right | 4.5 s | Read |
| on-sale | Stock and price | GPT-6 Luna | 3 | Right | 2.2 s | Read |
| on-sale | Stock and price | GPT-6 Sol | 3 | Right | 2.8 s | Read |
| on-sale | Stock and price | Claude Haiku 5.5 | 3 | Right | 2.1 s | Read |
| on-sale | Stock and price | Claude Sonnet 5 | 3 | Right | 2.4 s | Read |
| on-sale | Stock and price | Claude Sonnet 5.5 | 3 | Right | 3.8 s | Read |
| beanie-on-sale | Stock and price | GPT-6.1 Sol | 1 | Right | 3.7 s | Read |
| beanie-on-sale | Stock and price | GPT-6 Luna | 1 | Right | 4.3 s | Read |
| beanie-on-sale | Stock and price | GPT-6 Sol | 1 | Right | 2.3 s | Read |
| beanie-on-sale | Stock and price | Claude Haiku 5.5 | 1 | Right | 2.1 s | Read |
| beanie-on-sale | Stock and price | Claude Sonnet 5 | 1 | Right | 1.3 s | Read |
| beanie-on-sale | Stock and price | Claude Sonnet 5.5 | 1 | Right | 3.9 s | Read |
| beanie-on-sale | Stock and price | GPT-6.1 Sol | 2 | Right | 3.9 s | Read |
| beanie-on-sale | Stock and price | GPT-6 Luna | 2 | Right | 2.8 s | Read |
| beanie-on-sale | Stock and price | GPT-6 Sol | 2 | Right | 2.5 s | Read |
| beanie-on-sale | Stock and price | Claude Haiku 5.5 | 2 | Right | 1.8 s | Read |
| beanie-on-sale | Stock and price | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| beanie-on-sale | Stock and price | Claude Sonnet 5.5 | 2 | Right | 3.5 s | Read |
| beanie-on-sale | Stock and price | GPT-6.1 Sol | 3 | Right | 3.8 s | Read |
| beanie-on-sale | Stock and price | GPT-6 Luna | 3 | Right | 2.7 s | Read |
| beanie-on-sale | Stock and price | GPT-6 Sol | 3 | Right | 2.6 s | Read |
| beanie-on-sale | Stock and price | Claude Haiku 5.5 | 3 | Right | 2.3 s | Read |
| beanie-on-sale | Stock and price | Claude Sonnet 5 | 3 | Right | 1.0 s | Read |
| beanie-on-sale | Stock and price | Claude Sonnet 5.5 | 3 | Right | 3.2 s | Read |
| halo-stock-count | Stock and price | GPT-6.1 Sol | 1 | Right | 4.1 s | Read |
| halo-stock-count | Stock and price | GPT-6 Luna | 1 | Right | 2.4 s | Read |
| halo-stock-count | Stock and price | GPT-6 Sol | 1 | Right | 2.4 s | Read |
| halo-stock-count | Stock and price | Claude Haiku 5.5 | 1 | Right | 2.9 s | Read |
| halo-stock-count | Stock and price | Claude Sonnet 5 | 1 | Right | 2.7 s | Read |
| halo-stock-count | Stock and price | Claude Sonnet 5.5 | 1 | Right | 2.8 s | Read |
| halo-stock-count | Stock and price | GPT-6.1 Sol | 2 | Right | 3.0 s | Read |
| halo-stock-count | Stock and price | GPT-6 Luna | 2 | Right | 2.1 s | Read |
| halo-stock-count | Stock and price | GPT-6 Sol | 2 | Right | 2.5 s | Read |
| halo-stock-count | Stock and price | Claude Haiku 5.5 | 2 | Right | 2.1 s | Read |
| halo-stock-count | Stock and price | Claude Sonnet 5 | 2 | Right | 2.6 s | Read |
| halo-stock-count | Stock and price | Claude Sonnet 5.5 | 2 | Right | 3.9 s | Read |
| halo-stock-count | Stock and price | GPT-6.1 Sol | 3 | Right | 3.7 s | Read |
| halo-stock-count | Stock and price | GPT-6 Luna | 3 | Right | 3.2 s | Read |
| halo-stock-count | Stock and price | GPT-6 Sol | 3 | Right | 2.2 s | Read |
| halo-stock-count | Stock and price | Claude Haiku 5.5 | 3 | Right | 2.8 s | Read |
| halo-stock-count | Stock and price | Claude Sonnet 5 | 3 | Right | 2.8 s | Read |
| halo-stock-count | Stock and price | Claude Sonnet 5.5 | 3 | Right | 3.4 s | Read |
| traverse-us-10 | Stock and price | GPT-6.1 Sol | 1 | Right | 3.7 s | Read |
| traverse-us-10 | Stock and price | GPT-6 Luna | 1 | Right | 3.8 s | Read |
| traverse-us-10 | Stock and price | GPT-6 Sol | 1 | Right | 3.9 s | Read |
| traverse-us-10 | Stock and price | Claude Haiku 5.5 | 1 | Right | 3.0 s | Read |
| traverse-us-10 | Stock and price | Claude Sonnet 5 | 1 | Right | 1.3 s | Read |
| traverse-us-10 | Stock and price | Claude Sonnet 5.5 | 1 | Right | 6.5 s | Read |
| traverse-us-10 | Stock and price | GPT-6.1 Sol | 2 | Right | 3.8 s | Read |
| traverse-us-10 | Stock and price | GPT-6 Luna | 2 | Right | 5.0 s | Read |
| traverse-us-10 | Stock and price | GPT-6 Sol | 2 | Right | 2.3 s | Read |
| traverse-us-10 | Stock and price | Claude Haiku 5.5 | 2 | Right | 2.5 s | Read |
| traverse-us-10 | Stock and price | Claude Sonnet 5 | 2 | Right | 2.0 s | Read |
| traverse-us-10 | Stock and price | Claude Sonnet 5.5 | 2 | Right | 9.7 s | Read |
| traverse-us-10 | Stock and price | GPT-6.1 Sol | 3 | Right | 3.6 s | Read |
| traverse-us-10 | Stock and price | GPT-6 Luna | 3 | Right | 5.7 s | Read |
| traverse-us-10 | Stock and price | GPT-6 Sol | 3 | Right | 3.7 s | Read |
| traverse-us-10 | Stock and price | Claude Haiku 5.5 | 3 | Right | 2.2 s | Read |
| traverse-us-10 | Stock and price | Claude Sonnet 5 | 3 | Right | 2.9 s | Read |
| traverse-us-10 | Stock and price | Claude Sonnet 5.5 | 3 | Right | 7.1 s | Read |
| drift-restock | Stock and price | GPT-6.1 Sol | 1 | Right | 5.4 s | Read |
| drift-restock | Stock and price | GPT-6 Luna | 1 | Right | 3.8 s | Read |
| drift-restock | Stock and price | GPT-6 Sol | 1 | Right | 3.7 s | Read |
| drift-restock | Stock and price | Claude Haiku 5.5 | 1 | Wrong | 4.8 s | Read |
| drift-restock | Stock and price | Claude Sonnet 5 | 1 | Wrong | 4.8 s | Read |
| drift-restock | Stock and price | Claude Sonnet 5.5 | 1 | Right | 7.1 s | Read |
| drift-restock | Stock and price | GPT-6.1 Sol | 2 | Right | 7.9 s | Read |
| drift-restock | Stock and price | GPT-6 Luna | 2 | Right | 4.6 s | Read |
| drift-restock | Stock and price | GPT-6 Sol | 2 | Right | 2.6 s | Read |
| drift-restock | Stock and price | Claude Haiku 5.5 | 2 | Right | 4.2 s | Read |
| drift-restock | Stock and price | Claude Sonnet 5 | 2 | Right | 5.3 s | Read |
| drift-restock | Stock and price | Claude Sonnet 5.5 | 2 | Right | 6.9 s | Read |
| drift-restock | Stock and price | GPT-6.1 Sol | 3 | Right | 7.0 s | Read |
| drift-restock | Stock and price | GPT-6 Luna | 3 | Wrong | 5.1 s | Read |
| drift-restock | Stock and price | GPT-6 Sol | 3 | Right | 4.9 s | Read |
| drift-restock | Stock and price | Claude Haiku 5.5 | 3 | Right | 3.1 s | Read |
| drift-restock | Stock and price | Claude Sonnet 5 | 3 | Right | 1.5 s | Read |
| drift-restock | Stock and price | Claude Sonnet 5.5 | 3 | Right | 6.3 s | Read |
| summit-price | Stock and price | GPT-6.1 Sol | 1 | Right | 1.7 s | Read |
| summit-price | Stock and price | GPT-6 Luna | 1 | Right | 2.3 s | Read |
| summit-price | Stock and price | GPT-6 Sol | 1 | Right | 2.5 s | Read |
| summit-price | Stock and price | Claude Haiku 5.5 | 1 | Right | 2.1 s | Read |
| summit-price | Stock and price | Claude Sonnet 5 | 1 | Right | 1.3 s | Read |
| summit-price | Stock and price | Claude Sonnet 5.5 | 1 | Right | 3.3 s | Read |
| summit-price | Stock and price | GPT-6.1 Sol | 2 | Right | 3.3 s | Read |
| summit-price | Stock and price | GPT-6 Luna | 2 | Right | 2.1 s | Read |
| summit-price | Stock and price | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| summit-price | Stock and price | Claude Haiku 5.5 | 2 | Right | 2.6 s | Read |
| summit-price | Stock and price | Claude Sonnet 5 | 2 | Right | 1.5 s | Read |
| summit-price | Stock and price | Claude Sonnet 5.5 | 2 | Right | 1.9 s | Read |
| summit-price | Stock and price | GPT-6.1 Sol | 3 | Right | 4.0 s | Read |
| summit-price | Stock and price | GPT-6 Luna | 3 | Right | 2.4 s | Read |
| summit-price | Stock and price | GPT-6 Sol | 3 | Right | 1.2 s | Read |
| summit-price | Stock and price | Claude Haiku 5.5 | 3 | Right | 2.5 s | Read |
| summit-price | Stock and price | Claude Sonnet 5 | 3 | Right | 1.2 s | Read |
| summit-price | Stock and price | Claude Sonnet 5.5 | 3 | Right | 1.7 s | Read |
| oxygen-price | Stock and price | GPT-6.1 Sol | 1 | Right | 2.9 s | Read |
| oxygen-price | Stock and price | GPT-6 Luna | 1 | Right | 2.1 s | Read |
| oxygen-price | Stock and price | GPT-6 Sol | 1 | Right | 1.2 s | Read |
| oxygen-price | Stock and price | Claude Haiku 5.5 | 1 | Right | 2.0 s | Read |
| oxygen-price | Stock and price | Claude Sonnet 5 | 1 | Right | 1.1 s | Read |
| oxygen-price | Stock and price | Claude Sonnet 5.5 | 1 | Right | 1.2 s | Read |
| oxygen-price | Stock and price | GPT-6.1 Sol | 2 | Right | 1.9 s | Read |
| oxygen-price | Stock and price | GPT-6 Luna | 2 | Right | 2.3 s | Read |
| oxygen-price | Stock and price | GPT-6 Sol | 2 | Right | 1.6 s | Read |
| oxygen-price | Stock and price | Claude Haiku 5.5 | 2 | Right | 2.3 s | Read |
| oxygen-price | Stock and price | Claude Sonnet 5 | 2 | Right | 1.3 s | Read |
| oxygen-price | Stock and price | Claude Sonnet 5.5 | 2 | Right | 1.8 s | Read |
| oxygen-price | Stock and price | GPT-6.1 Sol | 3 | Right | 1.8 s | Read |
| oxygen-price | Stock and price | GPT-6 Luna | 3 | Right | 3.5 s | Read |
| oxygen-price | Stock and price | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| oxygen-price | Stock and price | Claude Haiku 5.5 | 3 | Right | 2.1 s | Read |
| oxygen-price | Stock and price | Claude Sonnet 5 | 3 | Right | 1.3 s | Read |
| oxygen-price | Stock and price | Claude Sonnet 5.5 | 3 | Right | 3.0 s | Read |
| most-expensive-board | Stock and price | GPT-6.1 Sol | 1 | Wrong | 2.6 s | Read |
| most-expensive-board | Stock and price | GPT-6 Luna | 1 | Wrong | 2.3 s | Read |
| most-expensive-board | Stock and price | GPT-6 Sol | 1 | Wrong | 2.1 s | Read |
| most-expensive-board | Stock and price | Claude Haiku 5.5 | 1 | Wrong | 2.4 s | Read |
| most-expensive-board | Stock and price | Claude Sonnet 5 | 1 | Wrong | 1.3 s | Read |
| most-expensive-board | Stock and price | Claude Sonnet 5.5 | 1 | Right | 5.6 s | Read |
| most-expensive-board | Stock and price | GPT-6.1 Sol | 2 | Wrong | 2.9 s | Read |
| most-expensive-board | Stock and price | GPT-6 Luna | 2 | Wrong | 2.2 s | Read |
| most-expensive-board | Stock and price | GPT-6 Sol | 2 | Wrong | 1.5 s | Read |
| most-expensive-board | Stock and price | Claude Haiku 5.5 | 2 | Wrong | 2.0 s | Read |
| most-expensive-board | Stock and price | Claude Sonnet 5 | 2 | Wrong | 1.1 s | Read |
| most-expensive-board | Stock and price | Claude Sonnet 5.5 | 2 | Right | 5.3 s | Read |
| most-expensive-board | Stock and price | GPT-6.1 Sol | 3 | Wrong | 1.7 s | Read |
| most-expensive-board | Stock and price | GPT-6 Luna | 3 | Right | 4.6 s | Read |
| most-expensive-board | Stock and price | GPT-6 Sol | 3 | Wrong | 1.2 s | Read |
| most-expensive-board | Stock and price | Claude Haiku 5.5 | 3 | Wrong | 2.8 s | Read |
| most-expensive-board | Stock and price | Claude Sonnet 5 | 3 | Wrong | 1.1 s | Read |
| most-expensive-board | Stock and price | Claude Sonnet 5.5 | 3 | Right | 4.7 s | Read |
| cheapest-goggles | Stock and price | GPT-6.1 Sol | 1 | Right | 2.0 s | Read |
| cheapest-goggles | Stock and price | GPT-6 Luna | 1 | Right | 2.8 s | Read |
| cheapest-goggles | Stock and price | GPT-6 Sol | 1 | Right | 1.1 s | Read |
| cheapest-goggles | Stock and price | Claude Haiku 5.5 | 1 | Right | 2.3 s | Read |
| cheapest-goggles | Stock and price | Claude Sonnet 5 | 1 | Right | 1.3 s | Read |
| cheapest-goggles | Stock and price | Claude Sonnet 5.5 | 1 | Right | 1.1 s | Read |
| cheapest-goggles | Stock and price | GPT-6.1 Sol | 2 | Right | 1.6 s | Read |
| cheapest-goggles | Stock and price | GPT-6 Luna | 2 | Right | 4.1 s | Read |
| cheapest-goggles | Stock and price | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| cheapest-goggles | Stock and price | Claude Haiku 5.5 | 2 | Right | 2.6 s | Read |
| cheapest-goggles | Stock and price | Claude Sonnet 5 | 2 | Right | 1.2 s | Read |
| cheapest-goggles | Stock and price | Claude Sonnet 5.5 | 2 | Right | 1.2 s | Read |
| cheapest-goggles | Stock and price | GPT-6.1 Sol | 3 | Right | 1.9 s | Read |
| cheapest-goggles | Stock and price | GPT-6 Luna | 3 | Right | 3.8 s | Read |
| cheapest-goggles | Stock and price | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| cheapest-goggles | Stock and price | Claude Haiku 5.5 | 3 | Right | 5.0 s | Read |
| cheapest-goggles | Stock and price | Claude Sonnet 5 | 3 | Right | 1.2 s | Read |
| cheapest-goggles | Stock and price | Claude Sonnet 5.5 | 3 | Right | 2.9 s | Read |
| complete-colours | Stock and price | GPT-6.1 Sol | 1 | Right | 2.2 s | Read |
| complete-colours | Stock and price | GPT-6 Luna | 1 | Right | 2.8 s | Read |
| complete-colours | Stock and price | GPT-6 Sol | 1 | Right | 0.9 s | Read |
| complete-colours | Stock and price | Claude Haiku 5.5 | 1 | Right | 1.0 s | Read |
| complete-colours | Stock and price | Claude Sonnet 5 | 1 | Right | 1.6 s | Read |
| complete-colours | Stock and price | Claude Sonnet 5.5 | 1 | Right | 1.4 s | Read |
| complete-colours | Stock and price | GPT-6.1 Sol | 2 | Right | 1.7 s | Read |
| complete-colours | Stock and price | GPT-6 Luna | 2 | Right | 2.1 s | Read |
| complete-colours | Stock and price | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| complete-colours | Stock and price | Claude Haiku 5.5 | 2 | Right | 1.0 s | Read |
| complete-colours | Stock and price | Claude Sonnet 5 | 2 | Right | 1.4 s | Read |
| complete-colours | Stock and price | Claude Sonnet 5.5 | 2 | Right | 1.0 s | Read |
| complete-colours | Stock and price | GPT-6.1 Sol | 3 | Right | 2.3 s | Read |
| complete-colours | Stock and price | GPT-6 Luna | 3 | Right | 2.6 s | Read |
| complete-colours | Stock and price | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| complete-colours | Stock and price | Claude Haiku 5.5 | 3 | Right | 2.1 s | Read |
| complete-colours | Stock and price | Claude Sonnet 5 | 3 | Right | 1.3 s | Read |
| complete-colours | Stock and price | Claude Sonnet 5.5 | 3 | Right | 1.0 s | Read |
| gift-cards | Stock and price | GPT-6.1 Sol | 1 | Right | 3.6 s | Read |
| gift-cards | Stock and price | GPT-6 Luna | 1 | Right | 3.8 s | Read |
| gift-cards | Stock and price | GPT-6 Sol | 1 | Right | 2.8 s | Read |
| gift-cards | Stock and price | Claude Haiku 5.5 | 1 | Right | 2.4 s | Read |
| gift-cards | Stock and price | Claude Sonnet 5 | 1 | Right | 2.4 s | Read |
| gift-cards | Stock and price | Claude Sonnet 5.5 | 1 | Right | 5.1 s | Read |
| gift-cards | Stock and price | GPT-6.1 Sol | 2 | Right | 3.1 s | Read |
| gift-cards | Stock and price | GPT-6 Luna | 2 | Right | 2.3 s | Read |
| gift-cards | Stock and price | GPT-6 Sol | 2 | Right | 2.6 s | Read |
| gift-cards | Stock and price | Claude Haiku 5.5 | 2 | Right | 4.1 s | Read |
| gift-cards | Stock and price | Claude Sonnet 5 | 2 | Right | 4.8 s | Read |
| gift-cards | Stock and price | Claude Sonnet 5.5 | 2 | Right | 4.9 s | Read |
| gift-cards | Stock and price | GPT-6.1 Sol | 3 | Right | 3.5 s | Read |
| gift-cards | Stock and price | GPT-6 Luna | 3 | Right | 3.7 s | Read |
| gift-cards | Stock and price | GPT-6 Sol | 3 | Right | 2.5 s | Read |
| gift-cards | Stock and price | Claude Haiku 5.5 | 3 | Right | 2.3 s | Read |
| gift-cards | Stock and price | Claude Sonnet 5 | 3 | Right | 5.7 s | Read |
| gift-cards | Stock and price | Claude Sonnet 5.5 | 3 | Right | 5.2 s | Read |
| shipping-time | Shipping and returns | GPT-6.1 Sol | 1 | Right | 1.7 s | Read |
| shipping-time | Shipping and returns | GPT-6 Luna | 1 | Right | 1.3 s | Read |
| shipping-time | Shipping and returns | GPT-6 Sol | 1 | Right | 1.2 s | Read |
| shipping-time | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 0.8 s | Read |
| shipping-time | Shipping and returns | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| shipping-time | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 1.5 s | Read |
| shipping-time | Shipping and returns | GPT-6.1 Sol | 2 | Right | 1.2 s | Read |
| shipping-time | Shipping and returns | GPT-6 Luna | 2 | Right | 1.1 s | Read |
| shipping-time | Shipping and returns | GPT-6 Sol | 2 | Right | 1.0 s | Read |
| shipping-time | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 0.9 s | Read |
| shipping-time | Shipping and returns | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| shipping-time | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 1.3 s | Read |
| shipping-time | Shipping and returns | GPT-6.1 Sol | 3 | Right | 1.5 s | Read |
| shipping-time | Shipping and returns | GPT-6 Luna | 3 | Right | 2.5 s | Read |
| shipping-time | Shipping and returns | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| shipping-time | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 0.9 s | Read |
| shipping-time | Shipping and returns | Claude Sonnet 5 | 3 | Right | 1.0 s | Read |
| shipping-time | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 1.6 s | Read |
| ship-to-germany | Shipping and returns | GPT-6.1 Sol | 1 | Right | 1.5 s | Read |
| ship-to-germany | Shipping and returns | GPT-6 Luna | 1 | Wrong | 2.2 s | Read |
| ship-to-germany | Shipping and returns | GPT-6 Sol | 1 | Wrong | 1.1 s | Read |
| ship-to-germany | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 1.6 s | Read |
| ship-to-germany | Shipping and returns | Claude Sonnet 5 | 1 | Right | 2.5 s | Read |
| ship-to-germany | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 2.1 s | Read |
| ship-to-germany | Shipping and returns | GPT-6.1 Sol | 2 | Wrong | 1.9 s | Read |
| ship-to-germany | Shipping and returns | GPT-6 Luna | 2 | Wrong | 2.3 s | Read |
| ship-to-germany | Shipping and returns | GPT-6 Sol | 2 | Wrong | 0.9 s | Read |
| ship-to-germany | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 1.7 s | Read |
| ship-to-germany | Shipping and returns | Claude Sonnet 5 | 2 | Wrong | 1.0 s | Read |
| ship-to-germany | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 1.8 s | Read |
| ship-to-germany | Shipping and returns | GPT-6.1 Sol | 3 | Right | 2.4 s | Read |
| ship-to-germany | Shipping and returns | GPT-6 Luna | 3 | Wrong | 2.5 s | Read |
| ship-to-germany | Shipping and returns | GPT-6 Sol | 3 | Wrong | 1.3 s | Read |
| ship-to-germany | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 1.7 s | Read |
| ship-to-germany | Shipping and returns | Claude Sonnet 5 | 3 | Right | 1.2 s | Read |
| ship-to-germany | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 2.1 s | Read |
| return-window | Shipping and returns | GPT-6.1 Sol | 1 | Right | 1.6 s | Read |
| return-window | Shipping and returns | GPT-6 Luna | 1 | Right | 2.5 s | Read |
| return-window | Shipping and returns | GPT-6 Sol | 1 | Right | 2.0 s | Read |
| return-window | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 1.2 s | Read |
| return-window | Shipping and returns | Claude Sonnet 5 | 1 | Right | 1.0 s | Read |
| return-window | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 0.8 s | Read |
| return-window | Shipping and returns | GPT-6.1 Sol | 2 | Right | 1.6 s | Read |
| return-window | Shipping and returns | GPT-6 Luna | 2 | Right | 2.6 s | Read |
| return-window | Shipping and returns | GPT-6 Sol | 2 | Right | 0.9 s | Read |
| return-window | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 2.6 s | Read |
| return-window | Shipping and returns | Claude Sonnet 5 | 2 | Right | 1.0 s | Read |
| return-window | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 1.7 s | Read |
| return-window | Shipping and returns | GPT-6.1 Sol | 3 | Right | 2.5 s | Read |
| return-window | Shipping and returns | GPT-6 Luna | 3 | Right | 4.5 s | Read |
| return-window | Shipping and returns | GPT-6 Sol | 3 | Right | 1.5 s | Read |
| return-window | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 0.9 s | Read |
| return-window | Shipping and returns | Claude Sonnet 5 | 3 | Right | 1.0 s | Read |
| return-window | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 1.3 s | Read |
| return-ridden-board | Shipping and returns | GPT-6.1 Sol | 1 | Right | 1.3 s | Read |
| return-ridden-board | Shipping and returns | GPT-6 Luna | 1 | Right | 1.1 s | Read |
| return-ridden-board | Shipping and returns | GPT-6 Sol | 1 | Right | 1.1 s | Read |
| return-ridden-board | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 1.2 s | Read |
| return-ridden-board | Shipping and returns | Claude Sonnet 5 | 1 | Right | 1.3 s | Read |
| return-ridden-board | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 1.5 s | Read |
| return-ridden-board | Shipping and returns | GPT-6.1 Sol | 2 | Right | 1.5 s | Read |
| return-ridden-board | Shipping and returns | GPT-6 Luna | 2 | Right | 1.9 s | Read |
| return-ridden-board | Shipping and returns | GPT-6 Sol | 2 | Right | 1.2 s | Read |
| return-ridden-board | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 2.1 s | Read |
| return-ridden-board | Shipping and returns | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| return-ridden-board | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 1.2 s | Read |
| return-ridden-board | Shipping and returns | GPT-6.1 Sol | 3 | Right | 1.7 s | Read |
| return-ridden-board | Shipping and returns | GPT-6 Luna | 3 | Right | 1.0 s | Read |
| return-ridden-board | Shipping and returns | GPT-6 Sol | 3 | Right | 1.1 s | Read |
| return-ridden-board | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 1.1 s | Read |
| return-ridden-board | Shipping and returns | Claude Sonnet 5 | 3 | Right | 1.3 s | Read |
| return-ridden-board | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 4.2 s | Read |
| start-return | Shipping and returns | GPT-6.1 Sol | 1 | Right | 2.9 s | Read |
| start-return | Shipping and returns | GPT-6 Luna | 1 | Right | 2.0 s | Read |
| start-return | Shipping and returns | GPT-6 Sol | 1 | Right | 1.1 s | Read |
| start-return | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 0.9 s | Read |
| start-return | Shipping and returns | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| start-return | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 1.3 s | Read |
| start-return | Shipping and returns | GPT-6.1 Sol | 2 | Right | 1.5 s | Read |
| start-return | Shipping and returns | GPT-6 Luna | 2 | Right | 0.9 s | Read |
| start-return | Shipping and returns | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| start-return | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 1.0 s | Read |
| start-return | Shipping and returns | Claude Sonnet 5 | 2 | Right | 0.9 s | Read |
| start-return | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 1.0 s | Read |
| start-return | Shipping and returns | GPT-6.1 Sol | 3 | Right | 1.4 s | Read |
| start-return | Shipping and returns | GPT-6 Luna | 3 | Right | 2.0 s | Read |
| start-return | Shipping and returns | GPT-6 Sol | 3 | Right | 1.0 s | Read |
| start-return | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 1.0 s | Read |
| start-return | Shipping and returns | Claude Sonnet 5 | 3 | Right | 0.9 s | Read |
| start-return | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 1.6 s | Read |
| refund-time | Shipping and returns | GPT-6.1 Sol | 1 | Right | 1.6 s | Read |
| refund-time | Shipping and returns | GPT-6 Luna | 1 | Right | 1.3 s | Read |
| refund-time | Shipping and returns | GPT-6 Sol | 1 | Right | 1.7 s | Read |
| refund-time | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 0.8 s | Read |
| refund-time | Shipping and returns | Claude Sonnet 5 | 1 | Right | 1.1 s | Read |
| refund-time | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 1.0 s | Read |
| refund-time | Shipping and returns | GPT-6.1 Sol | 2 | Right | 1.3 s | Read |
| refund-time | Shipping and returns | GPT-6 Luna | 2 | Right | 1.0 s | Read |
| refund-time | Shipping and returns | GPT-6 Sol | 2 | Right | 0.8 s | Read |
| refund-time | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 0.7 s | Read |
| refund-time | Shipping and returns | Claude Sonnet 5 | 2 | Right | 0.9 s | Read |
| refund-time | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 1.1 s | Read |
| refund-time | Shipping and returns | GPT-6.1 Sol | 3 | Right | 1.5 s | Read |
| refund-time | Shipping and returns | GPT-6 Luna | 3 | Right | 0.8 s | Read |
| refund-time | Shipping and returns | GPT-6 Sol | 3 | Right | 1.0 s | Read |
| refund-time | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 0.8 s | Read |
| refund-time | Shipping and returns | Claude Sonnet 5 | 3 | Right | 1.0 s | Read |
| refund-time | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 1.3 s | Read |
| cancel-order | Shipping and returns | GPT-6.1 Sol | 1 | Wrong | 3.5 s | Read |
| cancel-order | Shipping and returns | GPT-6 Luna | 1 | Wrong | 1.9 s | Read |
| cancel-order | Shipping and returns | GPT-6 Sol | 1 | Right | 1.9 s | Read |
| cancel-order | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 1.4 s | Read |
| cancel-order | Shipping and returns | Claude Sonnet 5 | 1 | Wrong | 2.2 s | Read |
| cancel-order | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 2.6 s | Read |
| cancel-order | Shipping and returns | GPT-6.1 Sol | 2 | Wrong | 3.1 s | Read |
| cancel-order | Shipping and returns | GPT-6 Luna | 2 | Wrong | 2.2 s | Read |
| cancel-order | Shipping and returns | GPT-6 Sol | 2 | Wrong | 1.8 s | Read |
| cancel-order | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 1.5 s | Read |
| cancel-order | Shipping and returns | Claude Sonnet 5 | 2 | Wrong | 2.2 s | Read |
| cancel-order | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 2.1 s | Read |
| cancel-order | Shipping and returns | GPT-6.1 Sol | 3 | Wrong | 2.7 s | Read |
| cancel-order | Shipping and returns | GPT-6 Luna | 3 | Wrong | 1.8 s | Read |
| cancel-order | Shipping and returns | GPT-6 Sol | 3 | Right | 1.9 s | Read |
| cancel-order | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 1.9 s | Read |
| cancel-order | Shipping and returns | Claude Sonnet 5 | 3 | Wrong | 1.3 s | Read |
| cancel-order | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 3.1 s | Read |
| order-status-no-email | Shipping and returns | GPT-6.1 Sol | 1 | Right | 2.7 s | Read |
| order-status-no-email | Shipping and returns | GPT-6 Luna | 1 | Right | 2.3 s | Read |
| order-status-no-email | Shipping and returns | GPT-6 Sol | 1 | Right | 2.6 s | Read |
| order-status-no-email | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 1.8 s | Read |
| order-status-no-email | Shipping and returns | Claude Sonnet 5 | 1 | Right | 2.7 s | Read |
| order-status-no-email | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 2.5 s | Read |
| order-status-no-email | Shipping and returns | GPT-6.1 Sol | 2 | Right | 4.0 s | Read |
| order-status-no-email | Shipping and returns | GPT-6 Luna | 2 | Right | 2.1 s | Read |
| order-status-no-email | Shipping and returns | GPT-6 Sol | 2 | Right | 2.2 s | Read |
| order-status-no-email | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 1.9 s | Read |
| order-status-no-email | Shipping and returns | Claude Sonnet 5 | 2 | Right | 2.4 s | Read |
| order-status-no-email | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 2.7 s | Read |
| order-status-no-email | Shipping and returns | GPT-6.1 Sol | 3 | Right | 3.2 s | Read |
| order-status-no-email | Shipping and returns | GPT-6 Luna | 3 | Right | 2.0 s | Read |
| order-status-no-email | Shipping and returns | GPT-6 Sol | 3 | Right | 1.9 s | Read |
| order-status-no-email | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 1.8 s | Read |
| order-status-no-email | Shipping and returns | Claude Sonnet 5 | 3 | Right | 2.5 s | Read |
| order-status-no-email | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 3.2 s | Read |
| paypal | Shipping and returns | GPT-6.1 Sol | 1 | Wrong | 1.3 s | Read |
| paypal | Shipping and returns | GPT-6 Luna | 1 | Wrong | 2.7 s | Read |
| paypal | Shipping and returns | GPT-6 Sol | 1 | Wrong | 1.3 s | Read |
| paypal | Shipping and returns | Claude Haiku 5.5 | 1 | Wrong | 1.4 s | Read |
| paypal | Shipping and returns | Claude Sonnet 5 | 1 | Wrong | 1.0 s | Read |
| paypal | Shipping and returns | Claude Sonnet 5.5 | 1 | Wrong | 1.1 s | Read |
| paypal | Shipping and returns | GPT-6.1 Sol | 2 | Wrong | 2.1 s | Read |
| paypal | Shipping and returns | GPT-6 Luna | 2 | Wrong | 2.3 s | Read |
| paypal | Shipping and returns | GPT-6 Sol | 2 | Wrong | 0.9 s | Read |
| paypal | Shipping and returns | Claude Haiku 5.5 | 2 | Wrong | 1.3 s | Read |
| paypal | Shipping and returns | Claude Sonnet 5 | 2 | Wrong | 0.7 s | Read |
| paypal | Shipping and returns | Claude Sonnet 5.5 | 2 | Wrong | 1.1 s | Read |
| paypal | Shipping and returns | GPT-6.1 Sol | 3 | Wrong | 1.5 s | Read |
| paypal | Shipping and returns | GPT-6 Luna | 3 | Wrong | 1.0 s | Read |
| paypal | Shipping and returns | GPT-6 Sol | 3 | Wrong | 1.2 s | Read |
| paypal | Shipping and returns | Claude Haiku 5.5 | 3 | Wrong | 1.9 s | Read |
| paypal | Shipping and returns | Claude Sonnet 5 | 3 | Wrong | 0.8 s | Read |
| paypal | Shipping and returns | Claude Sonnet 5.5 | 3 | Wrong | 1.2 s | Read |
| warranty | Shipping and returns | GPT-6.1 Sol | 1 | Right | 1.4 s | Read |
| warranty | Shipping and returns | GPT-6 Luna | 1 | Right | 2.0 s | Read |
| warranty | Shipping and returns | GPT-6 Sol | 1 | Right | 1.1 s | Read |
| warranty | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 1.7 s | Read |
| warranty | Shipping and returns | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| warranty | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 2.5 s | Read |
| warranty | Shipping and returns | GPT-6.1 Sol | 2 | Right | 3.3 s | Read |
| warranty | Shipping and returns | GPT-6 Luna | 2 | Right | 2.1 s | Read |
| warranty | Shipping and returns | GPT-6 Sol | 2 | Right | 1.1 s | Read |
| warranty | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 2.5 s | Read |
| warranty | Shipping and returns | Claude Sonnet 5 | 2 | Right | 1.2 s | Read |
| warranty | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 2.2 s | Read |
| warranty | Shipping and returns | GPT-6.1 Sol | 3 | Right | 1.5 s | Read |
| warranty | Shipping and returns | GPT-6 Luna | 3 | Right | 2.4 s | Read |
| warranty | Shipping and returns | GPT-6 Sol | 3 | Right | 1.2 s | Read |
| warranty | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 2.0 s | Read |
| warranty | Shipping and returns | Claude Sonnet 5 | 3 | Right | 1.1 s | Read |
| warranty | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 2.3 s | Read |
| phone-number | Shipping and returns | GPT-6.1 Sol | 1 | Right | 1.3 s | Read |
| phone-number | Shipping and returns | GPT-6 Luna | 1 | Right | 3.2 s | Read |
| phone-number | Shipping and returns | GPT-6 Sol | 1 | Right | 1.0 s | Read |
| phone-number | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 0.7 s | Read |
| phone-number | Shipping and returns | Claude Sonnet 5 | 1 | Right | 0.9 s | Read |
| phone-number | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 2.0 s | Read |
| phone-number | Shipping and returns | GPT-6.1 Sol | 2 | Right | 1.3 s | Read |
| phone-number | Shipping and returns | GPT-6 Luna | 2 | Right | 2.1 s | Read |
| phone-number | Shipping and returns | GPT-6 Sol | 2 | Right | 1.0 s | Read |
| phone-number | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 1.4 s | Read |
| phone-number | Shipping and returns | Claude Sonnet 5 | 2 | Right | 0.9 s | Read |
| phone-number | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 2.5 s | Read |
| phone-number | Shipping and returns | GPT-6.1 Sol | 3 | Right | 1.3 s | Read |
| phone-number | Shipping and returns | GPT-6 Luna | 3 | Right | 1.8 s | Read |
| phone-number | Shipping and returns | GPT-6 Sol | 3 | Right | 0.9 s | Read |
| phone-number | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 1.1 s | Read |
| phone-number | Shipping and returns | Claude Sonnet 5 | 3 | Right | 0.9 s | Read |
| phone-number | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 2.3 s | Read |
| wax-how-often | Shipping and returns | GPT-6.1 Sol | 1 | Right | 1.9 s | Read |
| wax-how-often | Shipping and returns | GPT-6 Luna | 1 | Right | 1.2 s | Read |
| wax-how-often | Shipping and returns | GPT-6 Sol | 1 | Right | 1.1 s | Read |
| wax-how-often | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 1.9 s | Read |
| wax-how-often | Shipping and returns | Claude Sonnet 5 | 1 | Right | 1.2 s | Read |
| wax-how-often | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 1.8 s | Read |
| wax-how-often | Shipping and returns | GPT-6.1 Sol | 2 | Right | 2.6 s | Read |
| wax-how-often | Shipping and returns | GPT-6 Luna | 2 | Right | 1.2 s | Read |
| wax-how-often | Shipping and returns | GPT-6 Sol | 2 | Right | 1.0 s | Read |
| wax-how-often | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 1.7 s | Read |
| wax-how-often | Shipping and returns | Claude Sonnet 5 | 2 | Right | 1.0 s | Read |
| wax-how-often | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 1.3 s | Read |
| wax-how-often | Shipping and returns | GPT-6.1 Sol | 3 | Right | 1.2 s | Read |
| wax-how-often | Shipping and returns | GPT-6 Luna | 3 | Right | 1.1 s | Read |
| wax-how-often | Shipping and returns | GPT-6 Sol | 3 | Right | 2.1 s | Read |
| wax-how-often | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 1.6 s | Read |
| wax-how-often | Shipping and returns | Claude Sonnet 5 | 3 | Right | 2.0 s | Read |
| wax-how-often | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 1.1 s | Read |
| off-season-storage | Shipping and returns | GPT-6.1 Sol | 1 | Right | 1.7 s | Read |
| off-season-storage | Shipping and returns | GPT-6 Luna | 1 | Right | 1.1 s | Read |
| off-season-storage | Shipping and returns | GPT-6 Sol | 1 | Right | 1.8 s | Read |
| off-season-storage | Shipping and returns | Claude Haiku 5.5 | 1 | Right | 1.7 s | Read |
| off-season-storage | Shipping and returns | Claude Sonnet 5 | 1 | Right | 1.0 s | Read |
| off-season-storage | Shipping and returns | Claude Sonnet 5.5 | 1 | Right | 1.5 s | Read |
| off-season-storage | Shipping and returns | GPT-6.1 Sol | 2 | Right | 1.6 s | Read |
| off-season-storage | Shipping and returns | GPT-6 Luna | 2 | Right | 2.3 s | Read |
| off-season-storage | Shipping and returns | GPT-6 Sol | 2 | Right | 1.0 s | Read |
| off-season-storage | Shipping and returns | Claude Haiku 5.5 | 2 | Right | 1.9 s | Read |
| off-season-storage | Shipping and returns | Claude Sonnet 5 | 2 | Right | 1.2 s | Read |
| off-season-storage | Shipping and returns | Claude Sonnet 5.5 | 2 | Right | 1.1 s | Read |
| off-season-storage | Shipping and returns | GPT-6.1 Sol | 3 | Right | 2.6 s | Read |
| off-season-storage | Shipping and returns | GPT-6 Luna | 3 | Right | 1.2 s | Read |
| off-season-storage | Shipping and returns | GPT-6 Sol | 3 | Right | 1.0 s | Read |
| off-season-storage | Shipping and returns | Claude Haiku 5.5 | 3 | Right | 1.9 s | Read |
| off-season-storage | Shipping and returns | Claude Sonnet 5 | 3 | Right | 1.1 s | Read |
| off-season-storage | Shipping and returns | Claude Sonnet 5.5 | 3 | Right | 1.4 s | Read |
| add-wax-to-cart | Cart | GPT-6.1 Sol | 1 | Right | 3.3 s | Read |
| add-wax-to-cart | Cart | GPT-6 Luna | 1 | Right | 2.7 s | Read |
| add-wax-to-cart | Cart | GPT-6 Sol | 1 | Right | 4.8 s | Read |
| add-wax-to-cart | Cart | Claude Haiku 5.5 | 1 | Right | 3.0 s | Read |
| add-wax-to-cart | Cart | Claude Sonnet 5 | 1 | Right | 2.9 s | Read |
| add-wax-to-cart | Cart | Claude Sonnet 5.5 | 1 | Right | 3.7 s | Read |
| add-wax-to-cart | Cart | GPT-6.1 Sol | 2 | Right | 3.9 s | Read |
| add-wax-to-cart | Cart | GPT-6 Luna | 2 | Right | 3.6 s | Read |
| add-wax-to-cart | Cart | GPT-6 Sol | 2 | Right | 3.8 s | Read |
| add-wax-to-cart | Cart | Claude Haiku 5.5 | 2 | Right | 2.7 s | Read |
| add-wax-to-cart | Cart | Claude Sonnet 5 | 2 | Right | 3.1 s | Read |
| add-wax-to-cart | Cart | Claude Sonnet 5.5 | 2 | Right | 3.6 s | Read |
| add-wax-to-cart | Cart | GPT-6.1 Sol | 3 | Right | 5.1 s | Read |
| add-wax-to-cart | Cart | GPT-6 Luna | 3 | Right | 3.6 s | Read |
| add-wax-to-cart | Cart | GPT-6 Sol | 3 | Right | 3.1 s | Read |
| add-wax-to-cart | Cart | Claude Haiku 5.5 | 3 | Right | 3.0 s | Read |
| add-wax-to-cart | Cart | Claude Sonnet 5 | 3 | Right | 3.7 s | Read |
| add-wax-to-cart | Cart | Claude Sonnet 5.5 | 3 | Right | 4.3 s | Read |
| add-ember-us-9 | Cart | GPT-6.1 Sol | 1 | Right | 3.9 s | Read |
| add-ember-us-9 | Cart | GPT-6 Luna | 1 | Right | 2.9 s | Read |
| add-ember-us-9 | Cart | GPT-6 Sol | 1 | Right | 2.6 s | Read |
| add-ember-us-9 | Cart | Claude Haiku 5.5 | 1 | Right | 2.8 s | Read |
| add-ember-us-9 | Cart | Claude Sonnet 5 | 1 | Right | 2.9 s | Read |
| add-ember-us-9 | Cart | Claude Sonnet 5.5 | 1 | Right | 3.2 s | Read |
| add-ember-us-9 | Cart | GPT-6.1 Sol | 2 | Right | 3.7 s | Read |
| add-ember-us-9 | Cart | GPT-6 Luna | 2 | Right | 2.8 s | Read |
| add-ember-us-9 | Cart | GPT-6 Sol | 2 | Right | 3.0 s | Read |
| add-ember-us-9 | Cart | Claude Haiku 5.5 | 2 | Right | 2.7 s | Read |
| add-ember-us-9 | Cart | Claude Sonnet 5 | 2 | Right | 3.3 s | Read |
| add-ember-us-9 | Cart | Claude Sonnet 5.5 | 2 | Right | 3.4 s | Read |
| add-ember-us-9 | Cart | GPT-6.1 Sol | 3 | Right | 4.7 s | Read |
| add-ember-us-9 | Cart | GPT-6 Luna | 3 | Right | 3.6 s | Read |
| add-ember-us-9 | Cart | GPT-6 Sol | 3 | Right | 3.6 s | Read |
| add-ember-us-9 | Cart | Claude Haiku 5.5 | 3 | Right | 2.3 s | Read |
| add-ember-us-9 | Cart | Claude Sonnet 5 | 3 | Right | 4.1 s | Read |
| add-ember-us-9 | Cart | Claude Sonnet 5.5 | 3 | Right | 3.5 s | Read |
| add-two-socks | Cart | GPT-6.1 Sol | 1 | Right | 3.8 s | Read |
| add-two-socks | Cart | GPT-6 Luna | 1 | Right | 3.0 s | Read |
| add-two-socks | Cart | GPT-6 Sol | 1 | Right | 3.8 s | Read |
| add-two-socks | Cart | Claude Haiku 5.5 | 1 | Right | 3.7 s | Read |
| add-two-socks | Cart | Claude Sonnet 5 | 1 | Right | 3.5 s | Read |
| add-two-socks | Cart | Claude Sonnet 5.5 | 1 | Right | 3.4 s | Read |
| add-two-socks | Cart | GPT-6.1 Sol | 2 | Right | 5.2 s | Read |
| add-two-socks | Cart | GPT-6 Luna | 2 | Right | 2.9 s | Read |
| add-two-socks | Cart | GPT-6 Sol | 2 | Right | 2.7 s | Read |
| add-two-socks | Cart | Claude Haiku 5.5 | 2 | Right | 3.2 s | Read |
| add-two-socks | Cart | Claude Sonnet 5 | 2 | Right | 3.7 s | Read |
| add-two-socks | Cart | Claude Sonnet 5.5 | 2 | Right | 4.8 s | Read |
| add-two-socks | Cart | GPT-6.1 Sol | 3 | Right | 3.8 s | Read |
| add-two-socks | Cart | GPT-6 Luna | 3 | Right | 3.0 s | Read |
| add-two-socks | Cart | GPT-6 Sol | 3 | Right | 2.3 s | Read |
| add-two-socks | Cart | Claude Haiku 5.5 | 3 | Right | 3.7 s | Read |
| add-two-socks | Cart | Claude Sonnet 5 | 3 | Right | 3.2 s | Read |
| add-two-socks | Cart | Claude Sonnet 5.5 | 3 | Right | 3.7 s | Read |
| add-sold-out | Cart | GPT-6.1 Sol | 1 | Right | 4.6 s | Read |
| add-sold-out | Cart | GPT-6 Luna | 1 | Right | 5.7 s | Read |
| add-sold-out | Cart | GPT-6 Sol | 1 | Right | 2.4 s | Read |
| add-sold-out | Cart | Claude Haiku 5.5 | 1 | Right | 3.0 s | Read |
| add-sold-out | Cart | Claude Sonnet 5 | 1 | Right | 1.0 s | Read |
| add-sold-out | Cart | Claude Sonnet 5.5 | 1 | Right | 7.8 s | Read |
| add-sold-out | Cart | GPT-6.1 Sol | 2 | Right | 3.7 s | Read |
| add-sold-out | Cart | GPT-6 Luna | 2 | Right | 3.8 s | Read |
| add-sold-out | Cart | GPT-6 Sol | 2 | Right | 3.0 s | Read |
| add-sold-out | Cart | Claude Haiku 5.5 | 2 | Right | 4.0 s | Read |
| add-sold-out | Cart | Claude Sonnet 5 | 2 | Right | 1.7 s | Read |
| add-sold-out | Cart | Claude Sonnet 5.5 | 2 | Right | 7.6 s | Read |
| add-sold-out | Cart | GPT-6.1 Sol | 3 | Right | 3.6 s | Read |
| add-sold-out | Cart | GPT-6 Luna | 3 | Right | 2.8 s | Read |
| add-sold-out | Cart | GPT-6 Sol | 3 | Wrong | 2.5 s | Read |
| add-sold-out | Cart | Claude Haiku 5.5 | 3 | Right | 3.3 s | Read |
| add-sold-out | Cart | Claude Sonnet 5 | 3 | Right | 1.2 s | Read |
| add-sold-out | Cart | Claude Sonnet 5.5 | 3 | Right | 6.9 s | Read |
| real-person | Tone and tricky | GPT-6.1 Sol | 1 | Wrong | 1.5 s | Read |
| real-person | Tone and tricky | GPT-6 Luna | 1 | Wrong | 0.8 s | Read |
| real-person | Tone and tricky | GPT-6 Sol | 1 | Right | 0.9 s | Read |
| real-person | Tone and tricky | Claude Haiku 5.5 | 1 | Right | 0.8 s | Read |
| real-person | Tone and tricky | Claude Sonnet 5 | 1 | Right | 0.9 s | Read |
| real-person | Tone and tricky | Claude Sonnet 5.5 | 1 | Right | 1.1 s | Read |
| real-person | Tone and tricky | GPT-6.1 Sol | 2 | Wrong | 1.7 s | Read |
| real-person | Tone and tricky | GPT-6 Luna | 2 | Wrong | 0.9 s | Read |
| real-person | Tone and tricky | GPT-6 Sol | 2 | Right | 1.0 s | Read |
| real-person | Tone and tricky | Claude Haiku 5.5 | 2 | Right | 0.9 s | Read |
| real-person | Tone and tricky | Claude Sonnet 5 | 2 | Right | 0.9 s | Read |
| real-person | Tone and tricky | Claude Sonnet 5.5 | 2 | Right | 1.7 s | Read |
| real-person | Tone and tricky | GPT-6.1 Sol | 3 | Wrong | 1.3 s | Read |
| real-person | Tone and tricky | GPT-6 Luna | 3 | Wrong | 0.8 s | Read |
| real-person | Tone and tricky | GPT-6 Sol | 3 | Right | 0.8 s | Read |
| real-person | Tone and tricky | Claude Haiku 5.5 | 3 | Right | 0.7 s | Read |
| real-person | Tone and tricky | Claude Sonnet 5 | 3 | Right | 0.8 s | Read |
| real-person | Tone and tricky | Claude Sonnet 5.5 | 3 | Right | 0.9 s | Read |
| broken-zip | Tone and tricky | GPT-6.1 Sol | 1 | Right | 4.0 s | Read |
| broken-zip | Tone and tricky | GPT-6 Luna | 1 | Right | 2.0 s | Read |
| broken-zip | Tone and tricky | GPT-6 Sol | 1 | Right | 2.6 s | Read |
| broken-zip | Tone and tricky | Claude Haiku 5.5 | 1 | Right | 2.9 s | Read |
| broken-zip | Tone and tricky | Claude Sonnet 5 | 1 | Right | 1.6 s | Read |
| broken-zip | Tone and tricky | Claude Sonnet 5.5 | 1 | Right | 4.9 s | Read |
| broken-zip | Tone and tricky | GPT-6.1 Sol | 2 | Right | 4.6 s | Read |
| broken-zip | Tone and tricky | GPT-6 Luna | 2 | Wrong | 2.6 s | Read |
| broken-zip | Tone and tricky | GPT-6 Sol | 2 | Right | 3.0 s | Read |
| broken-zip | Tone and tricky | Claude Haiku 5.5 | 2 | Right | 3.0 s | Read |
| broken-zip | Tone and tricky | Claude Sonnet 5 | 2 | Right | 2.3 s | Read |
| broken-zip | Tone and tricky | Claude Sonnet 5.5 | 2 | Right | 2.8 s | Read |
| broken-zip | Tone and tricky | GPT-6.1 Sol | 3 | Right | 4.1 s | Read |
| broken-zip | Tone and tricky | GPT-6 Luna | 3 | Right | 2.3 s | Read |
| broken-zip | Tone and tricky | GPT-6 Sol | 3 | Right | 2.4 s | Read |
| broken-zip | Tone and tricky | Claude Haiku 5.5 | 3 | Right | 2.8 s | Read |
| broken-zip | Tone and tricky | Claude Sonnet 5 | 3 | Right | 1.9 s | Read |
| broken-zip | Tone and tricky | Claude Sonnet 5.5 | 3 | Right | 4.1 s | Read |
| bulgarian-kids-helmet | Tone and tricky | GPT-6.1 Sol | 1 | Right | 3.2 s | Read |
| bulgarian-kids-helmet | Tone and tricky | GPT-6 Luna | 1 | Right | 2.1 s | Read |
| bulgarian-kids-helmet | Tone and tricky | GPT-6 Sol | 1 | Right | 1.8 s | Read |
| bulgarian-kids-helmet | Tone and tricky | Claude Haiku 5.5 | 1 | Right | 2.1 s | Read |
| bulgarian-kids-helmet | Tone and tricky | Claude Sonnet 5 | 1 | Right | 1.0 s | Read |
| bulgarian-kids-helmet | Tone and tricky | Claude Sonnet 5.5 | 1 | Right | 3.9 s | Read |
| bulgarian-kids-helmet | Tone and tricky | GPT-6.1 Sol | 2 | Right | 3.2 s | Read |
| bulgarian-kids-helmet | Tone and tricky | GPT-6 Luna | 2 | Right | 2.7 s | Read |
| bulgarian-kids-helmet | Tone and tricky | GPT-6 Sol | 2 | Right | 1.3 s | Read |
| bulgarian-kids-helmet | Tone and tricky | Claude Haiku 5.5 | 2 | Right | 2.8 s | Read |
| bulgarian-kids-helmet | Tone and tricky | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| bulgarian-kids-helmet | Tone and tricky | Claude Sonnet 5.5 | 2 | Right | 3.3 s | Read |
| bulgarian-kids-helmet | Tone and tricky | GPT-6.1 Sol | 3 | Right | 3.2 s | Read |
| bulgarian-kids-helmet | Tone and tricky | GPT-6 Luna | 3 | Right | 2.6 s | Read |
| bulgarian-kids-helmet | Tone and tricky | GPT-6 Sol | 3 | Right | 2.1 s | Read |
| bulgarian-kids-helmet | Tone and tricky | Claude Haiku 5.5 | 3 | Right | 3.1 s | Read |
| bulgarian-kids-helmet | Tone and tricky | Claude Sonnet 5 | 3 | Right | 2.1 s | Read |
| bulgarian-kids-helmet | Tone and tricky | Claude Sonnet 5.5 | 3 | Right | 2.7 s | Read |
| competitor-goggles | Tone and tricky | GPT-6.1 Sol | 1 | Right | 2.5 s | Read |
| competitor-goggles | Tone and tricky | GPT-6 Luna | 1 | Right | 2.6 s | Read |
| competitor-goggles | Tone and tricky | GPT-6 Sol | 1 | Right | 1.0 s | Read |
| competitor-goggles | Tone and tricky | Claude Haiku 5.5 | 1 | Right | 3.0 s | Read |
| competitor-goggles | Tone and tricky | Claude Sonnet 5 | 1 | Right | 1.0 s | Read |
| competitor-goggles | Tone and tricky | Claude Sonnet 5.5 | 1 | Right | 2.2 s | Read |
| competitor-goggles | Tone and tricky | GPT-6.1 Sol | 2 | Wrong | 1.6 s | Read |
| competitor-goggles | Tone and tricky | GPT-6 Luna | 2 | Right | 1.0 s | Read |
| competitor-goggles | Tone and tricky | GPT-6 Sol | 2 | Wrong | 1.1 s | Read |
| competitor-goggles | Tone and tricky | Claude Haiku 5.5 | 2 | Right | 2.3 s | Read |
| competitor-goggles | Tone and tricky | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| competitor-goggles | Tone and tricky | Claude Sonnet 5.5 | 2 | Right | 1.1 s | Read |
| competitor-goggles | Tone and tricky | GPT-6.1 Sol | 3 | Right | 1.9 s | Read |
| competitor-goggles | Tone and tricky | GPT-6 Luna | 3 | Right | 3.1 s | Read |
| competitor-goggles | Tone and tricky | GPT-6 Sol | 3 | Right | 1.3 s | Read |
| competitor-goggles | Tone and tricky | Claude Haiku 5.5 | 3 | Right | 2.3 s | Read |
| competitor-goggles | Tone and tricky | Claude Sonnet 5 | 3 | Right | 1.5 s | Read |
| competitor-goggles | Tone and tricky | Claude Sonnet 5.5 | 3 | Right | 1.7 s | Read |
| whistler-weather | Tone and tricky | GPT-6.1 Sol | 1 | Right | 2.4 s | Read |
| whistler-weather | Tone and tricky | GPT-6 Luna | 1 | Wrong | 2.4 s | Read |
| whistler-weather | Tone and tricky | GPT-6 Sol | 1 | Right | 1.2 s | Read |
| whistler-weather | Tone and tricky | Claude Haiku 5.5 | 1 | Right | 2.5 s | Read |
| whistler-weather | Tone and tricky | Claude Sonnet 5 | 1 | Right | 1.3 s | Read |
| whistler-weather | Tone and tricky | Claude Sonnet 5.5 | 1 | Right | 1.9 s | Read |
| whistler-weather | Tone and tricky | GPT-6.1 Sol | 2 | Wrong | 1.6 s | Read |
| whistler-weather | Tone and tricky | GPT-6 Luna | 2 | Wrong | 2.5 s | Read |
| whistler-weather | Tone and tricky | GPT-6 Sol | 2 | Right | 1.5 s | Read |
| whistler-weather | Tone and tricky | Claude Haiku 5.5 | 2 | Right | 1.4 s | Read |
| whistler-weather | Tone and tricky | Claude Sonnet 5 | 2 | Right | 1.1 s | Read |
| whistler-weather | Tone and tricky | Claude Sonnet 5.5 | 2 | Right | 1.7 s | Read |
| whistler-weather | Tone and tricky | GPT-6.1 Sol | 3 | Right | 1.7 s | Read |
| whistler-weather | Tone and tricky | GPT-6 Luna | 3 | Wrong | 2.6 s | Read |
| whistler-weather | Tone and tricky | GPT-6 Sol | 3 | Right | 1.9 s | Read |
| whistler-weather | Tone and tricky | Claude Haiku 5.5 | 3 | Right | 2.3 s | Read |
| whistler-weather | Tone and tricky | Claude Sonnet 5 | 3 | Right | 1.2 s | Read |
| whistler-weather | Tone and tricky | Claude Sonnet 5.5 | 3 | Right | 2.0 s | Read |
| talk-to-a-human | Tone and tricky | GPT-6.1 Sol | 1 | Right | 3.8 s | Read |
| talk-to-a-human | Tone and tricky | GPT-6 Luna | 1 | Right | 1.6 s | Read |
| talk-to-a-human | Tone and tricky | GPT-6 Sol | 1 | Right | 2.4 s | Read |
| talk-to-a-human | Tone and tricky | Claude Haiku 5.5 | 1 | Right | 1.7 s | Read |
| talk-to-a-human | Tone and tricky | Claude Sonnet 5 | 1 | Right | 2.2 s | Read |
| talk-to-a-human | Tone and tricky | Claude Sonnet 5.5 | 1 | Right | 2.7 s | Read |
| talk-to-a-human | Tone and tricky | GPT-6.1 Sol | 2 | Right | 3.0 s | Read |
| talk-to-a-human | Tone and tricky | GPT-6 Luna | 2 | Right | 1.8 s | Read |
| talk-to-a-human | Tone and tricky | GPT-6 Sol | 2 | Right | 1.8 s | Read |
| talk-to-a-human | Tone and tricky | Claude Haiku 5.5 | 2 | Right | 2.6 s | Read |
| talk-to-a-human | Tone and tricky | Claude Sonnet 5 | 2 | Right | 2.2 s | Read |
| talk-to-a-human | Tone and tricky | Claude Sonnet 5.5 | 2 | Right | 2.4 s | Read |
| talk-to-a-human | Tone and tricky | GPT-6.1 Sol | 3 | Right | 2.6 s | Read |
| talk-to-a-human | Tone and tricky | GPT-6 Luna | 3 | Right | 1.9 s | Read |
| talk-to-a-human | Tone and tricky | GPT-6 Sol | 3 | Right | 2.0 s | Read |
| talk-to-a-human | Tone and tricky | Claude Haiku 5.5 | 3 | Right | 1.5 s | Read |
| talk-to-a-human | Tone and tricky | Claude Sonnet 5 | 3 | Right | 3.2 s | Read |
| talk-to-a-human | Tone and tricky | Claude Sonnet 5.5 | 3 | Right | 3.0 s | Read |
What the file looks like
One row per reply. The first three rows, as a spreadsheet opens them:
| edition | tested_on | question_id | group | question | right_answer_rule | dont_have_that | model | made_by | run | verdict | judge_anthropic | judge_openai | our_reading | wrong_fact | first_word_s | full_reply_s | words | reply | products_shown | added_to_cart | form_shown | chat_url |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2026-10-18 | first-board | Finding gear | I'm a complete beginner, what snowboard should I get? | Recommends one or two snowboards the store sells (a product card counts as a recommendation) and asks about, or accounts for, something a … | no | Claude Sonnet 5.5 | Anthropic | 1 | right | pass | pass | 7.5 | 8.9 | 103 | For a complete beginner, I'd start with The Collection Snowboard: Hydrogen. It has full rocker, so the edges rarely catch. It's also a so … | The Collection Snowboard: Hydrogen $600.00; The Multi-managed Snowboard $629.95 | https://bountful.com/research/shop-assistant-test/2026-10/chats/first-board | ||||
| 1 | 2026-10-18 | first-board | Finding gear | I'm a complete beginner, what snowboard should I get? | Recommends one or two snowboards the store sells (a product card counts as a recommendation) and asks about, or accounts for, something a … | no | Claude Sonnet 5.5 | Anthropic | 2 | right | pass | pass | 2.2 | 3.5 | 96 | For a first board, I'd start with The Collection Snowboard: Hydrogen. It has full rocker, so the edges rarely catch. The soft flex is for … | The Collection Snowboard: Hydrogen $600.00; The Multi-managed Snowboard $629.95 | https://bountful.com/research/shop-assistant-test/2026-10/chats/first-board | ||||
| 1 | 2026-10-18 | first-board | Finding gear | I'm a complete beginner, what snowboard should I get? | Recommends one or two snowboards the store sells (a product card counts as a recommendation) and asks about, or accounts for, something a … | no | Claude Sonnet 5.5 | Anthropic | 3 | right | pass | pass | 2.1 | 3.3 | 81 | For a first-timer, I'd go with The Collection Snowboard: Hydrogen. It's soft and forgiving, with full rocker so the edges rarely catch. I … | The Collection Snowboard: Hydrogen $600.00; The Multi-managed Snowboard $629.95 | https://bountful.com/research/shop-assistant-test/2026-10/chats/first-board |
What each column means
- edition, tested_on
- The edition the row belongs to, and the day the reply was made.
- question_id, group, question
- The question’s short name, its group and the exact words the shopper sent.
- right_answer_rule
- What a right answer must do. Both judges read this rule.
- dont_have_that
- yes when the right answer is that the store doesn’t sell it, or doesn’t say.
- model, made_by, run
- The AI model that answered, its maker, and which run this was. Each model answered every question three times.
- verdict
- right when both judges passed the reply, or when they disagreed and we read it as right. Otherwise wrong.
- judge_anthropic, judge_openai
- Each judge’s pass or fail, with its reason when it failed the reply.
- our_reading
- When the judges disagreed: right or wrong, as we read the reply against the rule. Empty when they agreed.
- wrong_fact
- What the reply got wrong, when it stated a price, size, stock state or product detail the store contradicts. We checked each one against the store.
- first_word_s, full_reply_s
- Seconds from the question to the first word, and to the whole reply.
- words
- The reply’s length in words.
- reply
- The reply’s full text.
- products_shown
- The product cards shown under the reply, with the price each card showed.
- added_to_cart, form_shown
- What the reply put in the cart, and the form that showed under it: the contact form, the back-in-stock form or the cancel form.
- chat_url
- The page where the reply can be read.
Cite this data
Bountful Research (2026). The shop assistant test, edition 1 [Data set]. https://bountful.com/research/shop-assistant-test/data