Skip to content

robots.txt.liquid · AI bots

GPTBot, ClaudeBot, PerplexityBot, Google-Extended on Shopify

AI bots crawl Shopify stores by default. Per Shopify's verbatim Crawling-your-store doc: "Your store can be indexed by search engines and large language models (LLMs) without signatures"2. The only Shopify-native lever for changing this is robots.txt.liquid1. Four tokens come up in almost every block-AI guide: GPTBot (OpenAI training), ClaudeBot (Anthropic training), PerplexityBot (Perplexity search and citations), and Google-Extended (Gemini training and grounding, not Search). The search and user bots next to them, OpenAI's OAI-SearchBot and ChatGPT-User and Anthropic's Claude-SearchBot and Claude-User, are the ones that put a store into AI answers. This article names what blocking each one costs.

Published Verified 2026-09-16

The default: AI bots crawl unless you block them

Shopify's default robots.txt does not block AI training bots. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and similar user-agents are allowed by default the same way bingbot and Googlebot are. The platform's stance is opt-out, not opt-in. To block any AI bot, the merchant must create robots.txt.liquid and add an explicit per-user-agent Disallow rule. This is consistent with Shopify's broader 'platform handles the safe defaults, merchant handles the strategic decisions' philosophy.

The implication for stores that have not touched robots.txt.liquid: every AI bot that respects robots.txt is currently crawling your storefront. Every product page, collection page, blog post, and policy page is being read by AI training crawlers and citation crawlers. For most Shopify stores in 2026 this is the desired outcome — AI shopping channels are the new traffic surface, and visibility there depends partly on the AI vendors having crawled the storefront.

The four AI crawler user-agents that matter in 2026

Four tokens come up in almost every guide. (1) GPTBot — OpenAI's training crawler. Distinct from OAI-SearchBot (which powers ChatGPT search) and ChatGPT-User (which fetches pages for certain user actions in ChatGPT and Custom GPTs). Blocking GPTBot does not block ChatGPT's ability to cite your store via OAI-SearchBot. (2) ClaudeBot — Anthropic's training crawler. Honours robots.txt. Anthropic runs two more: Claude-SearchBot, which indexes pages for Claude's search results, and Claude-User, which fetches a page when a Claude user asks about it. Blocking ClaudeBot touches neither. (3) PerplexityBot — Perplexity's search crawler. It surfaces and links sites in Perplexity's results, and Perplexity says it isn't used to train models. (4) Google-Extended — a robots.txt token, not a separate crawler, that controls whether Google can use content it already crawls to train Gemini models and to ground Gemini answers. Google says it doesn't affect inclusion or ranking in Google Search.

Worth understanding the OpenAI distinction in particular: OpenAI runs three crawlers relevant here. GPTBot is the training crawler (the one most "block AI training" guides target). OAI-SearchBot finds pages for ChatGPT's search features. ChatGPT-User fetches pages for certain user actions in ChatGPT and Custom GPTs, and because a user started the request, OpenAI says robots.txt rules may not apply to it. OpenAI treats each setting independently, so blocking GPTBot but allowing OAI-SearchBot is the common compromise: opt out of training but stay visible in ChatGPT search3. Anthropic's current page lists the same split: ClaudeBot for training, Claude-SearchBot and Claude-User for search and user requests4.

The decision matrix — block, allow, or selective

Three positions cover most stores. (1) Allow all — the default. Best for stores chasing AI shopping visibility (most Shopify stores in 2026). Catalog eligibility plus AI bot access maximizes the chance products surface in ChatGPT, Perplexity, Gemini, and Copilot. (2) Block training, allow shopping — block GPTBot, ClaudeBot, Google-Extended (training-focused user-agents) while leaving OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, Googlebot, and PerplexityBot (search and citation) allowed. Useful for brands sensitive to AI training but still wanting AI shopping visibility. (3) Block all — block every named AI user-agent. Almost never the right call on an ecommerce Shopify store; sometimes justified on editorial brands with proprietary content.

The honest assessment: position (1) is correct for ~95% of Shopify stores. The AI shopping channels are an emerging traffic stream the merchant cannot afford to opt out of, and the marginal damage of being in OpenAI's training corpus is small for ecommerce content (product descriptions, specifications, policies). Position (2) is a middle ground for brands that publicly advocate for AI opt-out as a brand stance. Position (3) is almost always the wrong call — it forfeits AI shopping visibility without meaningful upside.

How to add per-bot rules in robots.txt.liquid

Add per-bot Disallow rules below Shopify's default-groups loop in robots.txt.liquid. Never replace the default loop — that disconnects the store from Shopify's automatic default-rule updates. The pattern: keep the default loop intact, then append User-agent / Disallow blocks for each bot you want to block. Validate the rendered /robots.txt in a private browser window before pushing to live.

liquid robots.txt.liquid with custom AI-bot blocking rules 
{% for group in robots.default_groups -%}
{{ group.user_agent }}
{% for rule in group.rules -%}
{{ rule }}
{% endfor -%}
{% if group.sitemap != blank -%}
{{ group.sitemap }}
{% endif %}
{% endfor -%}

# Custom rules below the default loop
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Per Shopify Dev Docs, the default-groups loop is the recommended starting point because "the default rules are updated regularly to ensure that SEO best practices are always applied." Layering custom rules below the loop preserves the update path; replacing the loop forfeits it. Keep the three group.sitemap lines, or robots.txt stops pointing crawlers at your sitemap. The loop above prints each line with its own line break, because live stores show Shopify's objects can otherwise run two lines together (a rule and a Sitemap line fused into one).

The AI shopping cost of blocking

Blocking AI bots has a real, asymmetric cost in 2026. Your products reach AI channels via two paths: (1) Shopify Catalog, which syndicates product data to agentic storefronts such as ChatGPT and Microsoft Copilot independently of robots.txt, and (2) the AI bots' independent crawl of your storefront for context enrichment. Blocking the bots breaks path (2) but not path (1) — your products still appear in AI shopping recommendations if eligible, but the AI engines have less context (no policy data, no editorial copy, no FAQ pages) to reason about them. The result: fewer citations, weaker product summaries, and reduced placement in conversational shopping flows.

The strategic frame: AI shopping visibility is not just about Catalog eligibility. The AI engines need to understand your store to recommend it well, and understanding comes from crawling the storefront content. Blocking the crawlers limits understanding without saving the products from the recommendation surface entirely. It's an asymmetric loss.

For the GEO-side complement to this article — what each AI bot does after crawling and how to influence the resulting citations — see /shopify-ai-search/ai-crawlers/. For the broader robots.txt.liquid cluster, see the hub.

Keep reading