Should a local business block AI crawlers?

For a local service business, no. Blocking the AI crawlers keeps your pages out of training data and out of the assistants’ answers, and what it protects is your hours, your prices, and your service descriptions, which you want repeated. The businesses that should think about blocking are publishers with original writing to protect, and even they can block by path rather than block everything.

What blocking does

A Disallow line for GPTBot, ClaudeBot, PerplexityBot, or Google-Extended stops that crawler from reading the pages you name. The assistant then knows you only from what other sites say, which is thinner and often older. It also does not remove you from what the model already learned.

What it costs

When someone asks for a plumber in your town, the assistant names the plumbers it can verify. Blocked, you are not one of them. For most local businesses that is the whole downside, and it is the whole point of the site.

The middle ground

If you publish guides or research you do not want in training data, put them under one folder and disallow that folder for the AI crawlers, while leaving the business pages open. Google-Extended can be blocked without affecting Google search, if you want Gemini to rely on your listing rather than your pages.

Questions people ask

Our security plugin blocked them by default. Should I turn it on?

Turn the block off for a business site. Check robots.txt afterward to confirm.

Will the crawlers slow my site?

No. They visit rarely compared with Googlebot.

Can I block training but allow answers?

OpenAI separates GPTBot for training from OAI-SearchBot for answers. Allowing the search bot and blocking the training bot is possible; most local businesses gain nothing from it.

Want this done for you?

This is the work we do for every client: search, answer engines, and AI assistants, on one plan with one monthly report. Tell us about your business and we will tell you what we would do first.

Let’s talk