robots.txt

robots.txt is a text file at the root of a site (example.com/robots.txt) that tells crawlers which parts they may or may not visit.

By the SmooxSEO team. Updated on 16 November 2026.

In more detail

Each block names a user agent (a crawler) and lists Allow or Disallow rules for paths. It can also point to your XML sitemap.

robots.txt controls crawling, not indexing: a blocked page can still appear in results if other pages link to it. To keep a page out of results, use noindex instead, and don't block it.

Related problems SmooxSEO checks

Related terms