Home › GEO › Which AI crawlers to allow in robots.txt
GEO
Which AI crawlers to allow in robots.txt
Allow the crawlers of the assistants you want to be recommended by: GPTBot and OAI-SearchBot for ChatGPT, Google-Extended for Gemini, PerplexityBot for Perplexity, and ClaudeBot for Claude. Bing’s crawler covers Copilot. Many sites block these by accident through a plugin or a copied robots.txt, and then wonder why the assistants never name them.
The crawlers by name
- GPTBot: OpenAI, used to read pages for ChatGPT. OAI-SearchBot: OpenAI, used for ChatGPT search results.
- Google-Extended: the control for whether Google may use your content in Gemini. Googlebot still indexes you either way.
- PerplexityBot: Perplexity.
- ClaudeBot: Anthropic, for Claude.
- Bingbot: Microsoft, which also feeds Copilot.
- Applebot and Applebot-Extended: Apple, for Siri and Apple Intelligence.
What to write
The simplest correct robots.txt for a business that wants to be found allows everything and points to the sitemap. If you must block something, block by path, not by crawler. A file that says User-agent: * with Disallow: for nothing, then Sitemap: with your sitemap address, is right for almost every local business.
How to check yours
Open yourdomain.com/robots.txt in a browser. Look for any line that names one of the crawlers above with a Disallow: / under it. Also check for a blanket Disallow: / under User-agent: *, which blocks everyone. Some security and SEO plugins add these lines with a checkbox; turn it off.
The trade-off
Blocking AI crawlers keeps your content out of training data. It also keeps your business out of the answers. For a local service business the answers are where customers are, and the content at stake is your hours and your prices. Allow them.
Questions people ask
Will allowing GPTBot slow my site down?
No. These crawlers visit at a low rate, far below what Googlebot does every day.
I allowed the crawlers but still am not named. Why?
Access is the door, not the reason. The assistants still need clear facts on the site, agreement with your listings, and reviews that mention what you do.
Does llms.txt replace robots.txt?
No. llms.txt is an optional summary file for assistants. robots.txt is what decides whether they may read the site at all.
Related
Want this done for you?
This is the work we do for every client: search, answer engines, and AI assistants, on one plan with one monthly report. Tell us about your business and we will tell you what we would do first.
