How Pickrate measures
What the agents are, how we decide what they picked, what we check on your site, and where the method has limits.
Beta: all requests are written in English, as buyers in the US would write them.
What we measure
We give AI agents the kind of request a buyer makes, such as "add a chat widget to my Shopify store". Then we record which product the agent chose.
- Pick rate: the share of runs where the agent chose your product. Head-to-head questions are left out because they name you.
- Considered: the share of runs where the agent named your product as an option, including the runs where it picked you.
- With vs. without web: the pick rate for agents that can search the web next to agents that only know what they learned in training. If you lose both ways, the model doesn't know you well. If you lose only with web search, a competitor's pages are more useful to the agent.
The requests
For each scan, a model writes requests across different buyer types, businesses, constraints and project types. Every agent in a scan gets the same requests.
Chat assistants get each need in one of four ways, rotated across the requests. Every assistant gets the same one, so GPT, Claude and Gemini can be compared.
- Planning: a broad request about the bigger project, such as "I'm opening a clinic next month, what do I need for the website?". The buyer never asks for this kind of tool. We record whether it comes up, whether you're mentioned and whether you're the one recommended.
- Shortlist: "what are my options?", then "which one would you pick for me?". We record whether you make the list and whether you're the final pick.
- One tool: "recommend one tool that does X and Y", with a limit such as budget or team size. In our tests this wording changed which product assistants picked, so it gets its own result.
- Head-to-head: the buyer's plain question, then "I'm choosing between you and this competitor. Which one would you pick?". The wording is the same for every product, so it doesn't favour either side, and each question is asked twice with the names swapped; we report how often that changes the answer. The competitors come from your product settings. These questions name you, so they never count toward your pick rate. We report the win rate on its own.
Apart from head-to-heads, no request names a brand.
The agents
We run our own agent harness. It does not drive Claude Code or Cursor directly. It runs the same kind of models with file tools and, for some agents, web search.
| Agent | What it is |
|---|---|
| Fast GPT coding agent | GPT-5.4 mini. Edits code and can open web pages it already knows the address of. No web search. |
| GPT coding agent with web search | GPT-5.5. Edits code, searches the web and reads documentation. |
| GPT coding agent without web access | GPT-5.5 with no web access. Shows what the model learned in training. |
| GPT assistant with web search | GPT-5.5 with web search, close to how ChatGPT answers. |
| GPT assistant without web access | GPT-5.5 answering from what it learned in training. |
| Claude assistant with web search | Claude Opus 5.5 with web search, close to how Claude answers. |
| Perplexity with web search | Perplexity's own Sonar model with Perplexity search, close to how Perplexity answers. |
| Gemini assistant with Google Search | Gemini 4 Argon grounded in Google Search, close to how Gemini answers. |
| Claude coding agent | Claude Opus 5.5. Edits code and searches the web. |
Agents on Claude and Gemini run only when their API keys are configured. Not running at the moment: v0 by Vercel (app builder).
Coding agents work in a starter project: Static HTML site, React + Vite app, Next.js app, WordPress theme, Shopify theme. They can read and write files and, depending on the agent, open web pages or search. They can't run shell commands, so they declare packages in package.json.
What we can't test directly
Some of the tools buyers use have no way for us to run them from code. We say so instead of imitating them, and measure the closest real signal.
| Tool | Why not | What we measure instead |
|---|---|---|
| Lovable, Bolt, Replit Agent | No public API for sending a build request (Lovable allows it only through its MCP server, using the account's credits) | v0, which has an API, as the app builder we run. We also run Lovable trials by hand. In the first 5 (October 2026, one per category), Lovable built the feature on its own platform in 4 and chose Crisp for live chat. |
| Google AI answers in Search, Microsoft Copilot | No API | Gemini grounded in Google Search, and Perplexity |
| Shopify, Wix and WordPress assistants | No API | Your rank in the WordPress plugin directory and your Shopify App Store listing, which is where these assistants and their users pick apps |
| Claude Code, Cursor, the ChatGPT, Claude, Gemini and Perplexity apps | We call the same models directly; the apps add memory, their own instructions and their own search | The same models, with web search where the app has it |
How we detect the choice
We read the code each agent wrote: script domains, packages added to package.json, imports and embed attributes. A product name that only appears in a comment or a hook name counts only when the agent also says it chose that product. If the agent wrote no code for the feature, we use its own statement. For assistants, a small model reads the whole conversation and records which products it recommends, which one it picked and, in planning conversations, whether this kind of tool came up at all.
We also record the alternatives each agent considered and why it rejected them. Every run links to its full transcript: searches, pages read, files edited and the final code.
Our own check, October 22, 2026: we checked all 112 finished runs. 105 (94%) recorded the right product. We fixed the rules behind the rest and added tests for each case.
Blind check: because we wrote the detection rules, we also had a separate AI reviewer label the same 112 runs without seeing our answers. It agreed on 107 (96%). It caught 4 mistakes: Shopify's built-in newsletter form recorded as "built its own", a storage service recorded as the product, and an assistant's do-it-yourself answer recorded as a product. We fixed the rules behind all 4. The other cases are judgment calls, such as whether a placeholder booking link counts as a product.
The reviewer is an AI model, not a person. Human checks are next.
The site check
We load your site, and the sites of the competitors that won, the way an agent does. Each check passes, is weak, or fails.
| Check | What it looks at |
|---|---|
| Website reachable for agents | The homepage loads for an automated reader without errors or blocks. |
| Fast response | How long the homepage takes to answer. |
| Readable without JavaScript | Whether the main text is in the page itself, since many agents read raw HTML. |
| AI crawlers allowed | Whether robots.txt lets AI tools read the site. |
| llms.txt and llms-full.txt | Whether the site publishes a plain-text guide to its docs for AI tools. |
| Sitemap | Whether a sitemap lists the pages. |
| Clear one-line description | Whether the page metadata says what the product does. |
| Structured data | Whether the homepage has JSON-LD that describes the product. |
| Docs agents can read | Whether docs exist and load as text. |
| Install snippet visible | Whether install code or an embed snippet is on a page an agent can reach. |
| Pricing and free plan clear | Whether pricing and any free plan are stated in text. |
| Works before sign-up | Whether the site says the product can be added before the user creates an account. |
| MCP server for agents | Whether the product is in the official MCP registry, so a coding agent can connect to it and set it up. |
| API spec agents can read | Whether an OpenAPI spec is published or linked. Skipped in the score if you have no public API. |
| npm package | Whether a package exists on npm, how many weekly downloads it has, and how recently it was released. |
| Public code on GitHub | Whether the site links to a GitHub account with public code and examples. |
| Listed in Context7 | Whether the docs are in Context7, a docs index that coding agents search. Only checked when we have a Context7 key. |
| WordPress plugin directory rank | For products added to WordPress: where your plugin ranks in WordPress.org search for your category. WordPress assistants and buyers pick plugins from these results. |
| Shopify App Store listing | For products added to Shopify: whether you have an App Store listing, and its rating. We don't measure search rank because the App Store's robots.txt disallows search pages. |
Data so far
12 finished agent runs. These are early numbers from small samples, measured on OpenAI GPT models only.
Limits
- Agents are probabilistic. The same request can get a different answer on the next run, and models change often. One scan is a snapshot.
- In our tests, asking the same assistant the same request twice changed its top pick in about 1 in 5 pairs, so treat a change of a few points between scans as possible noise.
- Small samples are labeled in every report.
- Our agents are close to the tools people use, but they are not those tools. Claude Code, Cursor and the ChatGPT, Claude, Gemini and Perplexity apps may behave differently: the apps add memory, their own search and their own instructions.
- The site check reads public pages. It can't see content behind a login.
No paid placement
Paying customers get the same agents and requests as everyone else. We don't sell your data.
Run a free scan