AI robots stopped by a firewall
Short answer
AI robots stopped by a firewall means: asked with their real names, these robots got an error or a challenge page while a normal browser got the site. Often Cloudflare's "Block AI bots" setting or a bot-protection rule. Tested from Smoox's server: real robots come from their own addresses, so treat it as a strong hint. To fix it: in Cloudflare (Security > Bots) or your firewall, allow the AI search robots you want to be quoted by. Training robots can stay blocked.
By the SmooxSEO team. Updated on 16 November 2026.
What it means and why it matters
Asked with their real names, these robots got an error or a challenge page while a normal browser got the site. Often Cloudflare's "Block AI bots" setting or a bot-protection rule. Tested from Smoox's server: real robots come from their own addresses, so treat it as a strong hint.
How to fix it
In Cloudflare (Security > Bots) or your firewall, allow the AI search robots you want to be quoted by. Training robots can stay blocked.
Common causes
- A CDN or firewall "bot fight" mode that blocks or challenges unknown user agents.
- Rate limits that stop crawlers after a few requests.
- A CAPTCHA or JavaScript challenge shown to every non-browser visitor.
How to spot it
robots.txt may allow AI crawlers while your firewall still blocks them. Check your CDN or firewall settings for bot protection, and its logs for blocked AI user agents. SmooxSEO sends real requests with each AI crawler's user agent and compares the answers with a normal browser's.
Mistakes to avoid
- Allowing the bots in robots.txt and assuming they get in: the firewall decides first.
- Allowing user agents by name only: some firewalls also verify the crawler's IP ranges, which the AI providers publish.
How serious is it?
The page still ranks, but less well, or a real opportunity is lost. Fix it when you can.
About ai search checks
Whether AI assistants (ChatGPT, Claude, Perplexity, Gemini, Copilot) can read and quote your site: AI crawler access, llms.txt and answer-ready content.
Find this problem on your site
SmooxSEO checks your whole site for this problem and 311 others, lists the details, and tells you what to fix first.