Check robots.txt for GPTBot and AI robots
Enter your site’s address: the tool reads your robots.txt and tells you, robot by robot, whether ChatGPT, Perplexity, Claude or Google can read your pages.
Which AI robots should you allow?
OAI-SearchBot and ChatGPT-User
They let ChatGPT read and cite your pages when it answers. Blocking them means disappearing from its answers.
GPTBot
It is used to train models. Blocking it does not stop you being cited in searches, but makes it less likely that ChatGPT knows your brand.
PerplexityBot and Claude-SearchBot
They feed Perplexity’s and Claude’s answers. Allow them if you want to be cited there.
Bingbot
ChatGPT relies heavily on Bing’s index. Blocking Bingbot cuts you off from part of the citations.
Google-Extended
It controls how Google’s AI (Gemini) uses your pages, with no effect on Google Search ranking.
Only if you refuse to let your content train models. To be cited in ChatGPT’s answers, it is OAI-SearchBot and ChatGPT-User you need to allow.
Yes: without a file, everything is allowed. A robots.txt is still useful to declare your sitemap.
Your host’s firewall or bot protection may block these robots without going through robots.txt. Also check that your content shows without JavaScript.
Go further with the full audit
The free audit also checks mobile speed, authority, the tags of 30 pages and broken links.