Which AI crawlers to allow and how to check your robots.txt

Which AI crawlers there are and what each one feeds

Each company that runs an assistant has published the names of its crawlers and what they do. This part is documented and not a matter of testing.

The distinction that matters is between search crawlers and training crawlers. OAI-SearchBot and PerplexityBot fetch pages so an assistant can cite them in an answer. GPTBot and Google-Extended govern whether your content is used to train future models, and blocking them has no effect on whether you appear in ChatGPT search or in Google’s AI answers. Bingbot and Googlebot are the ordinary search crawlers, and they matter most of all, because every assistant reads one of the two indexes they build. The noindex and robots.txt page covers the rest of the file.

What a blocked robots.txt looks like on a dental office site

Teton Peaks Dental is a two-dentist general practice in Rexburg with about 1,900 active patients, open four days a week, taking Delta Dental, Blue Cross of Idaho and most PPO plans. The site was rebuilt in 2025 and the web designer installed a security plugin that offered to block AI bots with one click. The office manager clicked it.

The plugin wrote a robots.txt that disallows GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, CCBot and anthropic-ai. It did what it said. Six months later a new resident asks ChatGPT which dentists in Rexburg take Delta Dental and are accepting new patients. The answer names three offices. Teton Peaks is not one of them. The same question in Perplexity gives the same three. Gemini, which reads Google’s index and was never blocked, names Teton Peaks first.

Nothing on the site is wrong. The pages state the insurance plans, the hours, and the new-patient offer in plain sentences. OAI-SearchBot and PerplexityBot are simply not allowed to read them, so ChatGPT and Perplexity are answering from Bing’s cached view of the site and from a Healthgrades listing that shows the practice as not accepting new patients, which stopped being true in 2024.

A worked example: the Rexburg robots.txt before and after

Here is the file the plugin wrote, as it stood at tetonpeaksdental.com/robots.txt.

User-agent: GPTBot

Disallow: /

User-agent: OAI-SearchBot

Disallow: /

User-agent: ChatGPT-User

Disallow: /

User-agent: PerplexityBot

Disallow: /

User-agent: Google-Extended

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: CCBot

Disallow: /

User-agent: *

Disallow: /wp-admin/

Sitemap: https://tetonpeaksdental.com/sitemap_index.xml

The office decided it wanted to be found by every assistant and did not mind its public pages being used for training, since every sentence on the site was written to be read. The plugin setting was turned off and the file was rewritten by hand.

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Disallow: /patient-portal/

Sitemap: https://tetonpeaksdental.com/sitemap_index.xml

The after file allows every crawler, blocks the WordPress admin and the patient portal, and points at the sitemap. A practice that wanted to keep its content out of training while staying in search would keep the file above and add two blocks, one disallowing GPTBot and one disallowing Google-Extended, and nothing else.

The change was made on a Tuesday. OAI-SearchBot first appeared in the server log the following Monday, and PerplexityBot the day after. Three weeks later the ChatGPT prompt named Teton Peaks Dental with the correct insurance list and the new-patient offer. The Healthgrades listing was also corrected, which is a separate job covered under entity clarity. In our testing, a blocked search crawler is the one GEO problem with a same-day fix and no judgment call.

How to check your own robots.txt for AI crawlers

1. Open yourdomain.com/robots.txt in a browser. It is a plain text file. If the address returns a 404, you have no file and every crawler is allowed by default, which is fine.
2. Read every line that begins User-agent. For each one, look at the Disallow lines below it. A Disallow: / under OAI-SearchBot, PerplexityBot, bingbot or Googlebot is blocking that assistant from your whole site.
3. Check the User-agent: * block, which applies to every crawler not named separately. It should disallow only admin areas and private pages, never /.
4. If your site runs WordPress, open the settings of your security or SEO plugin and look for an option labeled block AI bots, block AI crawlers or similar. Wordfence, All in One SEO, Rank Math and several others have one. Turn it off, then reload robots.txt to confirm the lines are gone.
5. Ask your host whether bot protection at the server or CDN level is filtering crawlers. Cloudflare has a setting called AI Scrapers and Crawlers that blocks OAI-SearchBot and PerplexityBot when switched on, and it does not show up in robots.txt at all.
6. Test with Google’s robots.txt report in Search Console and with Bing Webmaster Tools’ robots.txt tester, which both show whether a given URL is fetchable by their crawler.

What we do about AI crawlers and robots.txt

You will get a reading of your robots.txt, your plugin settings and your host’s bot protection, with each crawler listed and marked as allowed or blocked. You choose whether to allow the training crawlers, GPTBot and Google-Extended, and the file is rewritten to reflect that choice while allowing every search crawler. The server log is then checked to confirm OAI-SearchBot, PerplexityBot and bingbot have actually visited. Each month the file is read again, because plugin updates have been observed to add the blocks back. This is part of every site audit, which is where the work starts.

Questions people ask about GEO: which AI crawlers to allow and how to check your robots.txt