AI crawler
An automated bot run by an AI company that fetches web pages to train models, build a search index, or answer a user's question in real time.
AI crawlers are how answer engines read the web. Most AI vendors now run more than one, and each does a different job. Treat them as one group in robots.txt and you'll eventually block the bot you actually wanted.
Three jobs, three kinds of bot
- Training crawlers collect content that may be used to train future models. Examples:
GPTBotfrom OpenAI andClaudeBotfrom Anthropic. - Search crawlers build the index an assistant searches when it answers. Examples:
OAI-SearchBot,PerplexityBotandClaude-SearchBot. - User-triggered fetchers visit a page because someone asked about it. Examples:
ChatGPT-User,Perplexity-UserandClaude-User. OpenAI and Perplexity say robots.txt may not apply to theirs, because a person started the request.
Google is the odd one out. Its AI features in Search rely on the regular Googlebot crawl, and Google-Extended is a robots.txt token rather than a separate crawler.
Why AI crawlers matter for AEO
An engine can't cite a page its search crawler never fetched. Blocking training bots is a legitimate business choice. Blocking search bots by accident is a visibility leak, and it's more common than it should be. The usual culprit isn't robots.txt at all. It's a firewall or CDN rule that challenges any bot it doesn't recognize.
Somewhere, right now, an AI is recommending your client's competitor.
Elite Answers is invite-only. Request access through Elite Digital and the team will set you up.