Robots.txt Generator for AI Crawlers

Your robots.txt is the first file every AI crawler reads. If GPTBot, ClaudeBot or PerplexityBot are blocked there, your business cannot appear in ChatGPT, Claude or Perplexity answers, no matter how good your content is. This free generator lets you pick a sensible preset or decide bot by bot, and gives you a ready-to-upload file.

Robots.txt Generator

Your robots.txtis the first file an AI crawler reads. It tells bots like GPTBot, ClaudeBot and PerplexityBot whether they may read your site. Allow the right ones and you become visible in ChatGPT, Claude, Perplexity and AI search; block the rest to keep scrapers out. Pick a preset or fine-tune each bot below, then copy the file to your site's root (/robots.txt).

Quick presets

Crawlers (20 of 25 allowed)

GPTBotOpenAI

Crawls pages to train OpenAI's models (ChatGPT).

OAI-SearchBotOpenAI

Indexes pages so they can appear in ChatGPT Search.

ChatGPT-UserOpenAI

Fetches a page live when a ChatGPT user opens its link.

ClaudeBotAnthropic

Crawls pages to train Anthropic's Claude models.

Claude-SearchBotAnthropic

Indexes pages for Claude's search features.

Claude-UserAnthropic

Fetches a page live when a Claude user opens its link.

GooglebotGoogle

Google's core search crawler (also feeds AI Overviews).

Google-ExtendedGoogle

Controls whether your content trains Gemini / Vertex AI.

Google-CloudVertexBotGoogle

Fetches content for Vertex AI agents on demand.

BingbotMicrosoft

Bing's crawler (powers Copilot answers).

PerplexityBotPerplexity

Indexes pages so they can be cited in Perplexity answers.

Perplexity-UserPerplexity

Fetches a page live when a Perplexity user asks about it.

ApplebotApple

Apple's crawler for Siri, Spotlight and Apple Intelligence.

Applebot-ExtendedApple

Controls whether your content trains Apple's AI models.

AmazonbotAmazon

Crawls for Amazon and Alexa answers.

Meta-ExternalAgentMeta

Crawls pages to train Meta AI models.

Meta-ExternalFetcherMeta

Fetches links for Meta AI assistants.

FacebookBotMeta

Meta's crawler for link previews and AI.

MistralAI-UserMistral

Fetches pages for Mistral's Le Chat assistant.

DuckAssistBotDuckDuckGo

Fetches pages for DuckDuckGo's AI-assisted answers.

CCBotCommon Crawl

Common Crawl; its open archive is used to train many models.

BytespiderByteDance

ByteDance / TikTok crawler. Known to be aggressive.

PetalBotHuawei

Huawei's crawler. Often blocked as low-value traffic.

cohere-aiCohere

Cohere's crawler for model training.

DiffbotDiffbot

Commercial scraper that resells extracted web data.

Your robots.txt

# robots.txt - generated with BotLens (botlens.app)
# Controls which AI crawlers and assistants may access your site.

# OpenAI - Crawls pages to train OpenAI's models (ChatGPT).
User-agent: GPTBot
Allow: /

# OpenAI - Indexes pages so they can appear in ChatGPT Search.
User-agent: OAI-SearchBot
Allow: /

# OpenAI - Fetches a page live when a ChatGPT user opens its link.
User-agent: ChatGPT-User
Allow: /

# Anthropic - Crawls pages to train Anthropic's Claude models.
User-agent: ClaudeBot
Allow: /

# Anthropic - Indexes pages for Claude's search features.
User-agent: Claude-SearchBot
Allow: /

# Anthropic - Fetches a page live when a Claude user opens its link.
User-agent: Claude-User
Allow: /

# Google - Google's core search crawler (also feeds AI Overviews).
User-agent: Googlebot
Allow: /

# Google - Controls whether your content trains Gemini / Vertex AI.
User-agent: Google-Extended
Allow: /

# Google - Fetches content for Vertex AI agents on demand.
User-agent: Google-CloudVertexBot
Allow: /

# Microsoft - Bing's crawler (powers Copilot answers).
User-agent: Bingbot
Allow: /

# Perplexity - Indexes pages so they can be cited in Perplexity answers.
User-agent: PerplexityBot
Allow: /

# Perplexity - Fetches a page live when a Perplexity user asks about it.
User-agent: Perplexity-User
Allow: /

# Apple - Apple's crawler for Siri, Spotlight and Apple Intelligence.
User-agent: Applebot
Allow: /

# Apple - Controls whether your content trains Apple's AI models.
User-agent: Applebot-Extended
Allow: /

# Amazon - Crawls for Amazon and Alexa answers.
User-agent: Amazonbot
Allow: /

# Meta - Crawls pages to train Meta AI models.
User-agent: Meta-ExternalAgent
Allow: /

# Meta - Fetches links for Meta AI assistants.
User-agent: Meta-ExternalFetcher
Allow: /

# Meta - Meta's crawler for link previews and AI.
User-agent: FacebookBot
Allow: /

# Mistral - Fetches pages for Mistral's Le Chat assistant.
User-agent: MistralAI-User
Allow: /

# DuckDuckGo - Fetches pages for DuckDuckGo's AI-assisted answers.
User-agent: DuckAssistBot
Allow: /

# Common Crawl - Common Crawl; its open archive is used to train many models.
User-agent: CCBot
Disallow: /

# ByteDance - ByteDance / TikTok crawler. Known to be aggressive.
User-agent: Bytespider
Disallow: /

# Huawei - Huawei's crawler. Often blocked as low-value traffic.
User-agent: PetalBot
Disallow: /

# Cohere - Cohere's crawler for model training.
User-agent: cohere-ai
Disallow: /

# Diffbot - Commercial scraper that resells extracted web data.
User-agent: Diffbot
Disallow: /

# Everyone else: allowed by default
User-agent: *
Disallow:

# Tip: point bots to your sitemap
Sitemap: https://your-domain.com/sitemap.xml

Where to put this file

Save the text above as a file named robots.txt and upload it to the root of your domain, so it loads at https://your-domain.com/robots.txt. It has to sit at the root, not in a subfolder.

Netlify / Cloudflare Pages / Vercel: drop it in your public (or site root) folder and redeploy.
WordPress:use your SEO plugin's robots.txt editor (Yoast, Rank Math), or upload the file to the site root via FTP.
Webflow / Wix / Squarespace:paste it into the platform's robots.txt / SEO settings.

Check it worked by opening https://your-domain.com/robots.txt in your browser.

How long until the changes are visible

For a BotLens scan: right away. BotLens reads your live robots.txt every time you run a Visibility Analysis, so a fresh scan reflects your changes as soon as your site serves the new file (after your deploy or CDN cache clears, usually a few minutes).

For the AI crawlers themselves: up to about 24 hours, sometimes longer. Bots like GPTBot, ClaudeBot and PerplexityBot do not re-read robots.txt on every visit. They cache it so they do not hammer your server, and only refresh it periodically (Google refreshes roughly daily; others vary). So a bot you just allowed may take a day or more to start crawling, and a bot you just blocked may make a few more requests before it notices the change.

Example output

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Frequently asked questions

What is robots.txt and why does it matter for AI visibility?

robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may read. AI assistants rely on their crawlers (GPTBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity) to learn about your business. If those bots are blocked, your site is invisible to AI answers.

Which AI crawlers should I allow?

Most businesses that want to be found in AI answers should allow GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User and Google-Extended. Bots used purely for model training with no answer attribution, like CCBot, are a judgment call and the generator explains each one.

Where do I upload the file?

Upload it to the root of your domain so it loads at https://your-domain.com/robots.txt. On most platforms (WordPress, Webflow, Wix, Squarespace) there is a robots.txt editor in the SEO settings; on custom sites drop it in the public folder and redeploy.

How long until changes take effect?

AI crawlers cache robots.txt and refresh it periodically, so expect up to about 24 hours, sometimes longer. A BotLens Visibility Scan reads your live file immediately, so you can verify the change right away.

Is your site visible to AI at all?

This tool fixes one signal. The free BotLens Visibility Scan checks 13, from crawler access to schema markup to answer-ready content, and scores your site the way AI assistants see it. No signup, no credit card.

Run the free scan