Robots.txt Checker

robots.txt is a plain text file located at the root of a website — for example, https://example.com/robots.txt. It uses the Robots Exclusion Protocol, standardized in RFC 9309 (published June 2022 by the IETF), to tell web crawlers which parts of your site they are and are not allowed to access. The protocol was originally proposed by Martijn Koster in 1994 and remained an informal convention for nearly three decades before being formally standardized.

Search engines like Google, Bing, and Yandex check robots.txt before crawling any page on your domain. According to Google's documentation, if the file returns HTTP 200, the rules within it are enforced. A 404 means the entire site is considered crawlable. A 5xx response causes Google to temporarily stop crawling — and after 30 days of consecutive 5xx responses, Google treats the last cached version of robots.txt as authoritative (RFC 9309 §2.3).

A misconfigured robots.txt can have serious consequences. Accidentally blocking Disallow: / on your wildcard user-agent will cause search engine bots to stop crawling your entire site, and pages can disappear from search results within days. Similarly, blocking CSS and JavaScript resources prevents search engines from rendering your pages, which can harm indexing and rankings even if the HTML itself is crawlable.

It's important to understand that robots.txt is an advisory protocol, not a security mechanism. It does not prevent access — anyone can view your robots.txt file, and malicious crawlers will ignore it entirely. For pages that must not be indexed, use the noindex meta tag or X-Robots-Tag HTTP header instead.

This tool fetches your robots.txt file server-side and performs a comprehensive analysis based on RFC 9309 and search engine best practices:

1. HTTP status check. The file must return HTTP 200. A 404 means no robots.txt is present — crawlers will treat the entire site as allowed. A 5xx means the file is temporarily inaccessible, which may cause crawlers to pause (RFC 9309 §2.3).
2. Content-Type validation. The server must return Content-Type: text/plain. Per RFC 9309 §2.2, crawlers should parse the file as UTF-8 encoded text. If it returns HTML, most bots will fail to parse it correctly.
3. HTML detection. Some servers return a 200 with an HTML error page instead of a proper robots.txt. This tool detects and flags that condition.
4. Directive parsing. The file is parsed into User-agent blocks, each with its Disallow, Allow, and Crawl-delay rules. Sitemap and Host declarations are also extracted. Duplicate rules, empty directives, and invalid sitemap URLs are flagged.
5. Sitemap reachability. Each declared Sitemap URL is checked with an HTTP HEAD request to verify it responds successfully. This confirms the URL is reachable — it does not validate the XML contents of the sitemap itself.
6. CMS detection. Common path patterns are analyzed to detect your CMS platform (WordPress, Next.js, Shopify, Drupal, etc.) and provide platform-specific recommendations for optimizing your robots.txt configuration.

You can also drag and drop a local robots.txt file or paste content from your clipboard to analyze it offline without using any credits. Local analysis performs all checks except HTTP status, Content-Type, and sitemap reachability.

With the rise of large language models (LLMs), a new generation of web crawlers has emerged. Most vendors run two or three separate bots for different jobs: a training crawler that learns from your content (GPTBot, ClaudeBot, Meta-ExternalAgent), a citation crawler that powers AI-search results and sends real referral traffic (OAI-SearchBot, PerplexityBot, Claude-SearchBot), and a user-triggered fetcher invoked only when a person clicks a citation or asks the model to read a URL (ChatGPT-User, Claude-User, Perplexity-User). Google and Apple add a third pattern: a robots.txt token (Google-Extended, Applebot-Extended) that is not a crawler at all, just an opt-out switch for model training.

Most respect robots.txt. You can block specific bots with a User-agent directive, for example User-agent: GPTBot followed by Disallow: /. The Generator tab includes a "Block AI Training Crawlers" preset that blocks the training bots and opt-out tokens while leaving citation and user-triggered bots allowed, so AI-search referrals still work. For the full implementation guide across robots.txt, Nginx, Apache, Cloudflare, and WordPress, see How to Block AI Bots. Before you decide whether you even should block them, Should You Block AI Bots? walks the strategic side.

Two distinctions matter. Training versus citation: blocking GPTBot has no effect on OAI-SearchBot, because they are different bots from the same company. If you want AI Overviews citations or ChatGPT search referrals, leave the citation crawlers allowed. Crawler versus token: Google-Extended and Applebot-Extended are not crawlers. Disallowing them opts your content out of Gemini and Apple Intelligence training without blocking Googlebot or Applebot. These are the only two bots where "block" means "do not train on" rather than "do not crawl".

Two crawlers do not reliably honor robots.txt. Bytespider (ByteDance / TikTok) is the documented non-compliant case and has been observed fetching paths it was told to skip; pair the robots.txt rule with a server- or CDN-level block. Perplexity-User may also ignore robots.txt because the fetch is user-initiated; if you want a hard block on user-triggered Perplexity fetches, enforce it at the web server.

The table below lists the major AI crawlers and tokens active as of 2026, their User-agent strings, role, and whether they respect robots.txt.