robots.txt
robots.txt is a text file at the root of a site (example.com/robots.txt) that tells crawlers which parts they may or may not visit.
By the SmooxSEO team. Updated on 16 November 2026.
In more detail
Each block names a user agent (a crawler) and lists Allow or Disallow rules for paths. It can also point to your XML sitemap.
robots.txt controls crawling, not indexing: a blocked page can still appear in results if other pages link to it. To keep a page out of results, use noindex instead, and don't block it.