Robots.txt Checker

robots.txt is a plain text file located at the root of a website — for example, https://example.com/robots.txt. It uses the Robots Exclusion Protocol, standardized in RFC 9309 (published June 2022 by the IETF), to tell web crawlers which parts of your site they are and are not allowed to access. The protocol was originally proposed by Martijn Koster in 1994 and remained an informal convention for nearly three decades before being formally standardized.

Search engines like Google, Bing, and Yandex check robots.txt before crawling any page on your domain. According to Google's documentation, if the file returns HTTP 200, the rules within it are enforced. A 404 means the entire site is considered crawlable. A 5xx response causes Google to temporarily stop crawling — and after 30 days of consecutive 5xx responses, Google treats the last cached version of robots.txt as authoritative (RFC 9309 §2.3).

A misconfigured robots.txt can have serious consequences. Accidentally blocking Disallow: / on your wildcard user-agent will cause search engine bots to stop crawling your entire site, and pages can disappear from search results within days. Similarly, blocking CSS and JavaScript resources prevents search engines from rendering your pages, which can harm indexing and rankings even if the HTML itself is crawlable.

It's important to understand that robots.txt is an advisory protocol, not a security mechanism. It does not prevent access — anyone can view your robots.txt file, and malicious crawlers will ignore it entirely. For pages that must not be indexed, use the noindex meta tag or X-Robots-Tag HTTP header instead.

This tool fetches your robots.txt file server-side and performs a comprehensive analysis based on RFC 9309 and search engine best practices:

1. HTTP status check. The file must return HTTP 200. A 404 means no robots.txt is present — crawlers will treat the entire site as allowed. A 5xx means the file is temporarily inaccessible, which may cause crawlers to pause (RFC 9309 §2.3).
2. Content-Type validation. The server must return Content-Type: text/plain. Per RFC 9309 §2.2, crawlers should parse the file as UTF-8 encoded text. If it returns HTML, most bots will fail to parse it correctly.
3. HTML detection. Some servers return a 200 with an HTML error page instead of a proper robots.txt. This tool detects and flags that condition.
4. Directive parsing. The file is parsed into User-agent blocks, each with its Disallow, Allow, and Crawl-delay rules. Sitemap and Host declarations are also extracted. Duplicate rules, empty directives, and invalid sitemap URLs are flagged.
5. Sitemap reachability. Each declared Sitemap URL is checked with an HTTP HEAD request to verify it responds successfully. This confirms the URL is reachable — it does not validate the XML contents of the sitemap itself.
6. CMS detection. Common path patterns are analyzed to detect your CMS platform (WordPress, Next.js, Shopify, Drupal, etc.) and provide platform-specific recommendations for optimizing your robots.txt configuration.

You can also drag and drop a local robots.txt file or paste content from your clipboard to analyze it offline without using any credits. Local analysis performs all checks except HTTP status, Content-Type, and sitemap reachability.

With the rise of large language models (LLMs), a new generation of web crawlers has emerged. Companies like OpenAI (GPTBot, ChatGPT-User), Anthropic (ClaudeBot, anthropic-ai), Google (Google-Extended), Meta (FacebookBot for AI training), and others now crawl the web to build training datasets and power AI-powered search features.

These crawlers generally respect robots.txt rules. You can block them using specific User-agent directives — for example, User-agent: GPTBot followed by Disallow: / prevents OpenAI from crawling your site. The Generator tab in this tool includes presets for blocking all known AI crawlers.

Note that blocking AI crawlers is separate from blocking search engine crawlers. You can allow Googlebot to index your site for search results while simultaneously blocking GPTBot from using your content for AI training. Each bot uses its own User-agent string and follows its own group in robots.txt.

The table below lists the major AI crawlers active as of 2026, their User-agent strings, and whether they respect robots.txt directives.