Skip to main content

robots.txt generator for AI robots

Choose which AI robots can read your site, add the folders to exclude and your sitemap: your robots.txt is written as you go, ready to copy.

Starting point

AI robots

RobotUsed byRoleAllowed
OAI-SearchBotChatGPTAnswers and search
ChatGPT-UserChatGPTAnswers and search
PerplexityBotPerplexityAnswers and search
Perplexity-UserPerplexityAnswers and search
Claude-SearchBotClaudeAnswers and search
Claude-UserClaudeAnswers and search
GPTBotChatGPTTraining
ClaudeBotClaudeTraining
Google-ExtendedGeminiTraining
Applebot-ExtendedApple IntelligenceTraining
Meta-ExternalAgentMeta AITraining
CCBotCommon CrawlTraining
BytespiderTikTokTraining

All robots

Example: /admin/ or /cart/

Example: https://mysite.com/sitemap.xml

Only for a test site: Google and AI will read nothing.

Your robots.txt

Save it at the root of your site, at /robots.txt.

# robots.txt built with the free TadaSEO generator

# All robots
User-agent: *
Allow: /

# AI robots: answers and search
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# AI robots: model training
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: CCBot
Allow: /

User-agent: Bytespider
Allow: /

Which AI robots should you allow?

  • Answers and search

    OAI-SearchBot, ChatGPT-User, PerplexityBot or Claude-SearchBot read your page when an AI prepares its answer. Blocking them means never being cited.

  • Training

    GPTBot, ClaudeBot, Google-Extended or CCBot collect pages to train models. Blocking them does not stop you from being cited, but makes your brand less known to AI.

  • One group per robot

    A robot with its own group ignores the rules of the “all robots” group. The generator therefore copies your excluded folders into each group.

  • The sitemap

    The Sitemap line lets every search engine find your pages. Use the full URL, in https.

  • At the root of the site: mysite.com/robots.txt. On WordPress, an SEO plugin or your host lets you edit it.

  • No. It asks robots not to read, without blocking anything. Protect private pages with a password.

  • Only if you refuse that your content is used for training. To be cited in ChatGPT, OAI-SearchBot and ChatGPT-User are what count.

Go further with the full audit

The free audit also checks mobile speed, authority, the tags of 30 pages and broken links.

  • No credit card
  • Results in 30 seconds
  • 5 free credits

Other free tools

Audit my site