robots.txt

robots.txt is the file every well-behaved crawler reads before requesting any other URL on a host. Each record names one or more `User-agent` lines and a set of `Allow` and `Disallow` rules. `Disallow: /admin/` asks crawlers to stay out of that subtree; `Disallow:` (empty) allows everything. The standard is advisory, so it stops Googlebot and friends but not malicious scrapers. robots.txt does not prevent indexing of URLs that are linked from elsewhere; it only blocks the crawl. To keep a page out of search results, use a `noindex` meta tag or `X-Robots-Tag` header instead.