Can AI crawlers read your site? How to check it in five minutes
An overly strict setting at your host or in robots.txt keeps the crawlers of ChatGPT, Claude and Perplexity out, without you noticing. How to check it and set it right.
New to this topic? Start with What Is GEO? Generative Engine Optimization Explained
The short answer
AI assistants read your site with their own crawlers, such as GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic and PerplexityBot. If your robots.txt or your host blocks them, they cannot read your pages and you do not get mentioned. Check your robots.txt for rules that exclude these crawlers, see whether your host or security service stops bots, and test with the GEO check in dGENIX. If you want AI assistants to mention you, allow the search crawlers.
Why this often goes wrong
Most businesses never consciously chose to keep AI crawlers out. It happens by accident:
- A default setting at the host or a security service that blocks "unknown bots". AI crawlers sometimes fall under that.
- A robots.txt from a template or plugin, with rules someone once copied.
- A decision from 2023, when many sites blocked all AI crawlers as a precaution. What made sense then now costs mentions in AI answers.
You do not notice, because your site works fine for normal visitors. Only the assistants see a closed door.
Two kinds of AI crawlers
Not every AI crawler does the same. Broadly there are two kinds:
| Kind | What it does | Examples |
|---|---|---|
| Search crawlers | Read your page to answer a user's question now, with you as a source | OAI-SearchBot, PerplexityBot, Claude-SearchBot |
| Training crawlers | Collect text to train future models | GPTBot, ClaudeBot, Google-Extended |
If you want to be mentioned in answers, at least the search crawlers need access. Whether you allow the training crawlers is a separate choice. For most small businesses allowing them makes sense: the more the models know about you, the more likely they know you.
The exact names change now and then. When in doubt, check the provider's documentation.
Step 1: look at your robots.txt
Go to yoursite.com/robots.txt. You see rules like:
User-agent: GPTBot Disallow: /
That means: GPTBot may not read anything. A rule with "User-agent: *" and "Disallow: /" excludes all crawlers, including Google. If there is nothing about AI crawlers, they may read everything not excluded elsewhere.
Change rules you did not consciously want, or have your web developer do it.
Step 2: check your hosting and security
Robots.txt is a request, not a lock. Some hosting plans and security services block bots in another way, before robots.txt is even read. Look in your host's or security service's control panel for settings such as "bot protection", "block AI scrapers" or "block AI bots". If it is on while you want to be mentioned, turn it off or make an exception for the search crawlers.
Step 3: test it
With the GEO audit in dGENIX, GENI checks whether the main AI crawlers can reach your site, and which rules in your robots.txt stop them. Ask:
"Check whether AI crawlers may read my site, and say which rule or setting stops them."
You get an answer per crawler, and what to change. The GEO Engine also does a broader check: how citable your pages are and whether you have an llms.txt. What an llms.txt is, is explained in llms.txt explained.
Step 4: keep an eye on it
An update at your host, a new plugin or a change by your web developer can quietly undo the setting. Have GENI repeat the check every month with a scheduled task. How that works is in scheduled tasks explained.
What you should keep closed
Access for crawlers does not mean everything must be open. Exclude what does not need to be found:
- login pages and customer areas;
- search results within your own site;
- test pages and drafts;
- files with personal data.
Those rules then apply to all crawlers, AI or not.
Next: being read is not yet being mentioned
Access is the first step. Whether an assistant also mentions you depends on how well your pages answer a question. More on that in writing so AI cites you and getting found in Gemini and Copilot.
After access: what the assistant gets to read
Once the crawlers are allowed in, what they find counts. A site whose text is only built in the browser, with lots of JavaScript, can still be empty to a crawler. Check that your main pages have their text in the HTML. The SEO Engine looks at that too, along with speed, titles and the structure of your pages.
Also make sure of a clear structure: headings that answer questions, short paragraphs, a frequently asked questions block. An assistant reading your page looks for a passage it can quote. The easier that is to find, the more likely you are the source.
A quick checklist
- Robots.txt checked for rules about GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot.
- Bot protection settings at your host or security service reviewed.
- A test run with the GEO check.
- A monthly repeat scheduled.
- Login and customer pages deliberately excluded.
Frequently asked questions
Does allowing AI crawlers cost bandwidth?
A little. For a normal business site that is negligible. If you do notice a burden, limit the training crawlers and allow the search crawlers.
Can I see whether AI crawlers visit my site?
Your host's log files show the visits with the crawler's name. Visits that arrive via an AI assistant often show in your analytics as a referral.
What if my web developer manages the settings?
Send them the outcome of the check, with the rules that need changing.
Where do I start?
dGENIX is in early access. Join the waitlist, or first read what is GEO.
