AI crawler

An AI crawler is a bot that reads web pages for an AI company, either to train models or to answer users' questions in real time.

By the SmooxSEO team. Updated on 16 November 2026.

In more detail

Companies run different bots for different jobs. OpenAI uses GPTBot for training, OAI-SearchBot for search and ChatGPT-User for live requests; Anthropic uses ClaudeBot, Claude-SearchBot and Claude-User; Perplexity uses PerplexityBot and Perplexity-User.

You can allow or block each one in robots.txt: blocking training bots is a choice, while blocking search bots keeps you out of AI answers.

Related problems SmooxSEO checks

Related terms