The robots.txt mistake that's blocking ChatGPT from your site.

A line in your robots.txt file may be blocking ChatGPT from reading your site — and most owners never know it's there. The file is a plain-text instruction sheet that lives at yoursite.com/robots.txt. It tells crawlers what they can and cannot fetch. If it names an AI crawler and tells it to stay out, that crawler stays out. Your pages never enter the training and retrieval systems that answer questions for millions of people.

This is a foundation problem, not a content problem. You can write the best answer on the internet. If the door is locked, the robot never reads it. Let's look at how the door gets locked, how to check yours, and how to decide who you let in.

What robots.txt actually does

Robots.txt is a small text file that tells automated visitors which parts of your site they may read. It sits at the root of your domain. Every well-behaved crawler checks it before it fetches anything else. Google checks it. Bing checks it. And the AI crawlers — the ones that feed ChatGPT, Perplexity, and Google's AI answers — check it too.

The file works on an honor system. It cannot physically stop a bad actor. But the major crawlers respect it, and the ones you want reading your site are the polite ones. So the file matters a great deal for the traffic and visibility you care about.

A typical entry looks like this:

User-agent: GPTBot
Disallow: /

The first line names the crawler. The second line tells it what it cannot read. A single forward slash means the whole site. Two lines, and every page you own is off-limits to that bot. No warning. No email. It just goes quiet.

The mistake that blocks ChatGPT

The mistake is a Disallow: / rule aimed at an AI crawler — often added by a plugin, a template, or a well-meaning developer who read a scary headline. Somewhere in the last two years, a wave of advice told site owners to block AI bots to protect their content. Plenty of hosting platforms and security plugins now ship a default that does exactly that.

The crawlers most people accidentally block:

• GPTBot — OpenAI's crawler. This is the one that reads pages for ChatGPT.
• OAI-SearchBot — OpenAI's search crawler, which powers ChatGPT's live web answers.
• PerplexityBot — the crawler behind Perplexity.
• Google-Extended — Google's control for whether your content trains its AI and appears in AI Overviews.
• ClaudeBot and anthropic-ai — Anthropic's crawlers.

Here is how the block usually shows up. A developer copies a robots.txt "best practice" file from a forum. The file includes a block on GPTBot because the author decided AI companies should not use their content for free. That opinion travels with the file. Now a San Diego family-law firm has the same block — even though it wants to be the answer when someone types "can I change a custody agreement in California" into ChatGPT.

The firm never made that choice. It inherited it. And it costs them every day the file stays that way.

Why blocking AI crawlers costs you clients

Blocking AI crawlers removes your business from the answers people now use to make decisions. Search is no longer one blue-link page. A growing share of people ask a question, read the AI's summary, and act on it. If your site is not in the crawler's index, you are not in the summary. You are not a source. You are not a link the reader clicks.

Think about how your clients actually search now. Someone hurt in a car accident asks ChatGPT what to do first. Someone with a leaking roof asks which repairs count as emergencies. Someone starting a business asks whether they need an operating agreement. The AI answers. If it pulls from three firms and yours is walled off, those three firms get named and you do not.

This is the same problem as a slow site or a broken page, dressed in new clothes. The buyer is looking. The technology that connects them to you cannot reach your pages. We wrote about the offline version of this in why your slow site is a sales problem, not an IT problem — the lesson holds here. A technical setting quietly decides whether a ready buyer ever sees you.

There is a real debate about whether AI companies should pay for content. That is a fair argument for a national publisher with millions of articles. It is the wrong argument for a five-person law firm or a two-truck HVAC company. You are not selling articles. You are trying to get found by people who need your service. Being in the answer is the win.

How to check your robots.txt in two minutes

You can check your file right now by typing your domain followed by /robots.txt into a browser. Go to yoursite.com/robots.txt and read what loads. That is the exact file every crawler sees. No login. No tool. Just look.

Here is what to do, step by step:

1. Open the file. Type your-domain.com/robots.txt into the address bar and press enter. If nothing loads, you may not have a file at all — which is fine, because no file means no blocks.
2. Search for the AI crawler names. Use your browser's find function (Ctrl+F or Cmd+F). Search for GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended, ClaudeBot, and anthropic-ai one at a time.
3. Read the line below each match. If you see Disallow: / under any of those names, that crawler is blocked from your whole site.
4. Check for a blanket block. Look for User-agent: * followed by Disallow: /. That blocks every crawler, including Google. If you see that, you have a bigger problem than AI — your site is invisible everywhere.
5. Note what you find. Write down which bots are blocked. You will want that list for the fix.

Most owners find one of two things. Either the file is clean and no AI bots are named — good. Or a plugin has quietly added a block they never asked for. The second one is common and takes minutes to fix.

How to fix it and who to let in

The fix is to remove the Disallow: / lines under the AI crawlers you want to allow. You edit the robots.txt file, delete the blocking rules, and save. If you run WordPress with an SEO plugin, the file is usually editable inside the plugin settings. If your developer manages the site, send them the list of bots you found blocked and ask them to remove those blocks.

A clean, open robots.txt for a business that wants AI visibility can be short. It allows the crawlers, points them to your sitemap, and gets out of the way. You do not need a complicated file. You need one that does not lock the doors you want open.

A word on choosing who to allow. You do not have to allow everyone. Some owners are fine with ChatGPT and Google but want to block scrapers that offer nothing back. That is a reasonable position. The point is to make the choice on purpose, not inherit it from a forum post. Decide who you want reading your site, then write the file to match.

One caution. Robots.txt is easy to break. A stray character, a wrong slash, a misplaced line — and you can block Google by accident while trying to allow ChatGPT. If you are not sure, have someone check the file after you edit it. Then re-open your-domain.com/robots.txt in a browser and confirm it reads the way you meant.

This kind of quiet technical setting is exactly what our Search Foundations work exists to catch. We check the file, the crawl rules, and the load speed before anyone touches content — because a locked door makes the best content worthless. When we audited McShanes Solicitors, fixing the foundation is what let the good work show up in search at all.

What this won't fix

Opening your robots.txt to ChatGPT will not put you in AI answers by itself. It removes the block. That is necessary, not sufficient. The crawler still has to find pages worth citing — clear answers, real expertise, a site that loads and reads well. If your pages are thin or slow, an open door only means the robot arrives, looks, and leaves. Speed is part of that story; Core Web Vitals explains the numbers crawlers weigh. Robots.txt gets you in the room. The rest of the work keeps you there.

Check your file today. It takes two minutes and it might be the cheapest visibility fix you make all year. Foundations first.