Crawling
Crawling is how search engines discover pages: a program (a crawler or bot) downloads pages and follows their links to find more.
By the SmooxSEO team. Updated on 16 November 2026.
In more detail
Googlebot and other crawlers start from known addresses and sitemaps, read each page, and add the links they find to their list. Pages that no link points to, or that are blocked by robots.txt, are hard or impossible to discover.
Crawlers only follow real links written as <a href>. Buttons and JavaScript click handlers usually aren't followed.