What Is GPTBot, and Should You Let AI Crawlers on Your Site?
Sergey Carp from Pasay, Co-founder / CTO
GPTBot is the web crawler run by OpenAI, the company behind ChatGPT. It visits websites and collects content that may be used to train OpenAI's models. It identifies itself with the GPTBot user agent, which is the name you use to allow or block it in robots.txt and the name you will see in your server logs.
Key takeaway
GPTBot is OpenAI's training crawler, separate from OAI-SearchBot and ChatGPT-User. Allow search bots, set a policy for training bots, and check your CDN.
OpenAI has three bots, not one
- GPTBot collects content for model training;
- OAI-SearchBot finds pages for search results in ChatGPT;
- ChatGPT-User opens a page when a user asks ChatGPT to look at it.
Blocking GPTBot does not remove you from ChatGPT search, and allowing it does not put you there. Each bot is controlled separately.
Other AI crawlers
- Google: Google-Extended controls whether your content is used for Gemini training and grounding; it does not affect Google Search.
- Anthropic: ClaudeBot, Claude-SearchBot and Claude-User.
- Perplexity: PerplexityBot and Perplexity-User.
- Microsoft: Copilot relies on Bingbot.
For example, to keep your content out of OpenAI training while staying visible in ChatGPT search, robots.txt could include:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: /
Should you let AI crawlers on your site?
- Search and answer bots are usually worth allowing on a business website. Otherwise AI assistants cannot cite or link to your pages.
- Training bots can be allowed or restricted according to your company policy.
- Private areas such as account pages, carts and admin sections should be closed to all bots.
Do not stop at robots.txt. Hosting firewalls and CDN bot protection sometimes block AI crawlers without the site owner knowing. Server logs show whether these bots are actually getting through. For a full breakdown, see which AI crawlers to allow and which to block.
At AI Search Optimization, configuring AI crawler access is part of our technical fixes. Book a call and we will check which bots your site is turning away right now.