GPTBot, ClaudeBot, PerplexityBot: Which AI Crawlers to Allow and Which to Block

Here is a story we see during AI visibility audits all the time. A company spends a year investing in content: expert articles, updated service pages, a proper FAQ. Yet it never shows up in ChatGPT or Perplexity answers. We open robots.txt and find a rule blocking every "unknown" bot, added years ago to stop scrapers. Or robots.txt is fine, but the CDN's bot protection or a geo-blocking rule rejects requests from overseas data centers, so OpenAI's crawlers get turned away. The website exists, the content exists, but for several AI assistants it simply is not there.

The opposite happens too. An owner learns that GPTBot is downloading their articles for model training and, in frustration, blocks every AI crawler at once, including the ones that send customers. Let's break down which AI crawlers exist, how a training crawler differs from a search crawler and how to set up access so you lose neither your content nor your customers.

Why AI needs to visit your website

When someone asks ChatGPT, Claude, Gemini or Perplexity "who can audit the books for a small ecommerce business", they get an AI answer: a ready-made text with recommendations, company names and links to source websites. In chat assistants, the answer arrives in the conversation with citations; in Google, it appears as an AI Overview above the regular results. People read the answer and often choose straight from it.

To write that answer, the AI has to read pages. Pages are fetched by programs called crawlers, or bots. Each crawler identifies itself to the server with a name, the user agent string, and that name is what robots.txt rules use to allow or block it. If the right crawler cannot open a page, the AI cannot cite it.

Three kinds of AI crawlers: training, search and user requests

The most important thing to understand: "the AI bot" is not a single bot. Major AI companies run several, and each has a different job.

• Training crawlers collect text that may be used to train future models. Blocking them keeps your content out of the model's "memory", but it usually does not affect real-time search.
• Search crawlers index pages so the AI can find them and cite them in answers. Block these and your website drops out of answers with sources.
• User-initiated fetchers open a specific page when a person in the chat asks "take a look at this website", or when the assistant needs to check a page while answering. This is not bulk crawling but a one-off action on behalf of a user.

For a business website, the logic is simple. Search crawlers and user-initiated fetchers should almost always be allowed: they bring mentions and visits. Training crawlers can be allowed or blocked; that is a company policy decision, not a marketing one.

AI crawlers list: who is who

Crawler names and roles change from time to time: companies add new bots and clarify what existing ones do. Below is the picture at the time of writing. Before you configure anything, check the documentation for website owners that each company publishes.

A note on Google. Google-Extended is not a bot that visits your site; it is a token you use in robots.txt. Googlebot still does the crawling, and the token only tells Google whether your content may be used for Gemini training and grounding. It does not affect Google Search. AI Overviews are built on the Google Search index, so you cannot opt out of them with Google-Extended, and blocking Googlebot means leaving Google Search entirely.

A note on user-initiated fetchers. AI companies sometimes describe bots that open a page at a person's direct request as an action by the user rather than automated crawling, and robots.txt rules may apply to them differently than to regular crawlers. Perplexity-User is one example. If you specifically need to restrict this kind of access, check the current documentation of that company and consider server-level rules.

GPTBot: block it or not?

GPTBot is the question we hear most, because it is the best-known AI crawler. The key point: GPTBot handles training, not answers with sources in ChatGPT. Search is handled by OAI-SearchBot, and opening pages at a user's request by ChatGPT-User. We cover the rest of the ChatGPT picture in our guide to getting your business recommended in ChatGPT.

That leaves three deliberate options:

• Allow everyone. Right for most service businesses and online stores: the more AI knows about your company, the better. Such websites rarely hold unique data worth protecting.
• Block training, allow search. Suits publishers, creators of paid content and owners of unique databases or methods. GPTBot, ClaudeBot and Google-Extended are blocked; search crawlers and user-initiated fetchers are allowed.
• Block everything except traditional search. Can make sense for private corporate portals. For a website that needs customers, it is almost always a mistake.

In your server logs, the GPTBot user agent is a string that contains the word GPTBot and a link to a page describing the crawler. Any script can fake that string, so in doubtful cases verify requests against the published IP ranges. OpenAI and other AI companies publish them.

How to block AI crawlers in robots.txt: examples

In each example, example.com is a placeholder; replace the utility sections (customer account, cart, site search) with your own.

Example 1. Allow everyone, block utility sections

User-agent: *
Disallow: /account/
Disallow: /cart/
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

The simplest option. There are no separate groups for AI crawlers, so they all follow the general group marked with an asterisk.

Example 2. Block training, allow search and user requests

# Training crawlers: blocked
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Search crawlers and user-initiated fetchers: allowed, except utility sections
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /account/
Disallow: /cart/
Disallow: /search/

# Everyone else, including Googlebot, Bingbot and YandexBot
User-agent: *
Disallow: /account/
Disallow: /cart/
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

There is a technical detail here that people get wrong most often. A crawler looks for a group with its own name in robots.txt, and if it finds one, it ignores the general asterisk group. That is why utility section rules must be repeated in every named group. Otherwise the account area is closed to everyone except OAI-SearchBot.

Example 3. A common mistake

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

This is what "block everyone except Google" looks like. The website stays in Google, but it is completely closed to the crawlers behind ChatGPT, Claude and Perplexity, and to Bingbot, which means Copilot too. We come across files like this regularly, and owners usually do not even remember they exist.

What about llms.txt in robots.txt?

There is no robots.txt directive for llms.txt, so you do not need to reference it: the file is found at its standard address, example.com/llms.txt. Just make sure robots.txt does not block access to it. What to write in the file itself is covered in our complete llms.txt guide.

Firewalls, CDNs and bot protection: the invisible block

robots.txt is a request, not a lock. The real lock sits on your server or in front of it, and it often stops the very bots you wanted to let in. The most common culprits:

• "Block AI bots" toggles in your CDN. Some CDNs and security services offer a one-click setting that blocks known AI crawlers, and it may be switched on by default for new projects. Such toggles do not always distinguish training crawlers from search crawlers.
• DDoS protection and browser challenges. A crawler cannot pass a JavaScript challenge or a CAPTCHA and gets a challenge page instead of your content.
• Geo-blocking. Businesses that only serve one country sometimes block traffic from everywhere else. AI crawlers typically run from data centers in other countries and get rejected too.
• Rate limiting. A crawler that requests several pages in a row hits the limit and receives a 429 status code.
• Web application firewalls and security plugins. Security modules in WordPress and other CMSs, as well as hosting-level firewalls, block "suspicious" user agents based on their own lists.

The fix is not to switch protection off, but to add the right crawlers to the allow list, ideally verified by their published IP ranges rather than by name alone. That way the real crawler gets through, and a bot pretending to be GPTBot does not.

How to check your server logs: who visits and what they get

The only reliable way to know the truth is your server access log. It records every request: URL, status code and user agent. Here are a few commands for a Linux server running nginx. The log path on your server may differ.

# How many requests GPTBot made
grep -ci "GPTBot" /var/log/nginx/access.log

# Which status codes OAI-SearchBot receives
grep -i "OAI-SearchBot" /var/log/nginx/access.log | cut -d' ' -f9 | sort | uniq -c

# Which pages PerplexityBot requested
grep -i "PerplexityBot" /var/log/nginx/access.log | cut -d' ' -f7 | sort | uniq -c | sort -rn | head -20

How to read the results:

• No requests at all: the crawler has not visited yet, or it is being blocked before it reaches your server, at the CDN level. Check your CDN logs.
• 200 status codes: all good, the crawler is getting your pages.
• 403, 429 or 503 status codes: the crawler arrives but gets rejected. Look for the cause in your protection settings.
• Lots of 301 and 404 codes: the crawler wastes time on redirects and broken links. Time to clean up.

If you have no server access, for example on a hosted website builder, crawl reports in Google Search Console and Bing Webmaster Tools help partially, but they tell you nothing about AI crawlers. In that case, ask your platform's support team.

Step-by-step access setup

Crawler access is the first section of our AI search optimization checklist. Without it, nothing else on the list matters.

The result: your website is open to the bots that bring customers

Once access is set up, the picture becomes manageable. Search crawlers receive pages with a 200 status code, utility sections are closed to everyone, the decision about model training is deliberate and recorded in robots.txt, and your protection lets verified crawlers through while stopping fake ones. This does not guarantee a spot in AI answers, but it removes the most frustrating reason for being absent from them: "they just could not get in".

How we handle AI crawler access and everything else

AI Search Optimization has been doing SEO since 2002. We have 100+ clients around the world, Google and Yandex certifications, and one team that handles your website and your AI visibility end to end. We optimize for Google AI Overviews and AI Mode, ChatGPT, Gemini, Claude, Perplexity, Microsoft Copilot and other AI assistants, for businesses in New York City and in international markets.

Setting up AI crawler access is part of technical fixes on the website. The full service includes:

• an AI visibility audit: robots.txt, server protection, logs, and what AI assistants say about you and your competitors;
• prompt research: the questions your customers ask AI assistants;
• technical fixes on the website: AI crawler access, llms.txt, schema markup, speed, errors;
• new content built for AI answers;
• brand mentions on third-party sites;
• publishing articles in external sources to build the authority of your company and its experts;
• ongoing AI visibility tracking.

AI evaluates your website as a whole, so we take care of the entire site from day one, from technical fixes to GEO. We guarantee delivery of GEO tasks, error fixes and fast adaptation of your website to changing AI search requirements, including new crawlers as they appear.

The monthly fee is $290 per month, including the first 10 hours of work. Every additional hour is $45. GEO work is billed on a time and materials basis: we invoice only the work completed and delivered that month. No discounts or free months.

Book a call. We will find out which crawlers your website is turning away right now and open it to the AI assistants your customers actually use.