AI search glossary
What is an AI crawler?
An AI crawler is software that AI companies use to visit and read websites, such as OpenAI's GPTBot, Anthropic's ClaudeBot or PerplexityBot. If these crawlers cannot read your website, AI tools may know little about you, or rely on what other sites say.
Also called: AI bot, GPTBot, ClaudeBot, PerplexityBot
Can my website block AI crawlers?
Yes, sometimes by accident. A setting in your website’s robots.txt file, a security service, or a site that needs JavaScript to show its text can all stop AI crawlers reading your pages.
An example
A firm’s services page loads its content with JavaScript. A person sees it fine, but an AI crawler sees an almost empty page. The AI then tells customers the firm does not offer that service.
What to do
Check that your robots.txt does not block the main AI crawlers, unless you have chosen to. Make sure your important content is in the page as plain text. Our audit includes this check.
Find out what AI is telling your customers.
Get a free AI visibility review from our team. We check what ChatGPT, Gemini, Perplexity and Claude say about your business, then talk you through what it means.
