robots.txt blocks the whole site
Short answer
robots.txt blocks the whole site means: "Disallow: /" stops Google from crawling every page. Often left over from development. To fix it: remove "Disallow: /" from the rules for Googlebot and User-agent: *.
By the SmooxSEO team. Updated on 16 November 2026.
What it means and why it matters
"Disallow: /" stops Google from crawling every page. Often left over from development.
How to fix it
Remove "Disallow: /" from the rules for Googlebot and User-agent: *.
Common causes
- A "Disallow: /" left in robots.txt from development.
- A CMS setting or plugin that writes a blocking robots.txt.
- A robots.txt copied from a staging server during deployment.
How to spot it
Open yourdomain.com/robots.txt: "User-agent: *" followed by "Disallow: /" blocks everything. Search Console's robots.txt report shows what Google read. SmooxSEO reads your robots.txt and tests your pages against it.
Example
User-agent: *
Disallow: /User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlMistakes to avoid
- Expecting robots.txt to remove pages from Google: it stops crawling, not indexing. Use noindex to keep a page out.
- Blocking the site while you "prepare the launch" and forgetting to remove it afterwards.
How to fix it on your platform
- WordPress
- Turn off Settings > Reading > "Discourage search engines", and check for a physical robots.txt file or an SEO plugin's robots.txt editor.
- Shopify
- Shopify generates robots.txt; a robots.txt.liquid template in your theme can override it. Remove any Disallow: / you added there.
Questions people ask
How long does it take Google to see a fixed robots.txt?
Google usually refreshes robots.txt within a day. In Search Console's robots.txt report, you can ask it to fetch the file again.
How serious is it?
The site, or its key pages, can drop out of Google or can't be reached. Fix it first.
About robots.txt checks
The robots.txt file tells crawlers which parts of the site they may visit. A wrong rule can hide a whole site from search engines.
Find this problem on your site
SmooxSEO checks your whole site for this problem and 311 others, lists the details, and tells you what to fix first.