How to Identify and Manage Web Crawlers: A Sysadmin's Guide to robots.txt, AI Bots, and SEO Crawlers

Last month I got a frantic message from a colleague: "Someone is hammering our production server, thousands of requests per minute from a single IP. Should I file an abuse report?" I looked up the IP with the IP Location tool, checked the access logs, and found the User-Agent string: Mozilla/5.0 (compatible; Screaming Frog SEO Spider/21.0). The marketing team had kicked off a full-site audit without telling IT.

This happens more than you would think. Sysadmins see an IP generating massive traffic, look it up, and immediately reach for the abuse report template from my complete guide to reporting IP abuse. But not every heavy hitter is malicious. Web crawlers, from SEO audit tools to AI training bots to search engine indexers, can generate traffic patterns that look almost identical to a denial-of-service attack. Filing an abuse report against Googlebot or your own company's SEO tool is not just embarrassing; it wastes the ISP's abuse desk time and your own.

This guide covers how to identify what is actually crawling your server, the full landscape of crawlers you will encounter in 2025, how to manage them with robots.txt and server-level controls, and a decision framework for when to block, allow, or escalate to an abuse report.

How to Identify What Is Crawling Your Server

The single most important step before taking any action is checking the User-Agent string in your access logs. Every legitimate crawler identifies itself (or at least attempts to), and the User-Agent tells you exactly who is responsible.

Extracting User-Agents from Access Logs

Nginx/Apache combined log format:

# List all unique User-Agents, sorted by request count
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -30

# Find all requests from a specific IP and show their User-Agents
awk -v ip="198.51.100.47" '$1 == ip {split($0, a, "\""); print a[6]}' /var/log/nginx/access.log | sort -u

# Count requests per IP per hour (find the heavy hitters)
awk '{print $1, substr($4,2,14)}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

Quick one-liner to find bot traffic:

# Extract all requests with "bot" or "crawler" or "spider" in User-Agent
grep -iE '(bot|crawler|spider|scraper|fetcher)' /var/log/nginx/access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

Verifying Crawler Identity with IP Lookup

User-Agent strings can be spoofed. Anyone can send a request claiming to be Googlebot. To verify a crawler is genuine:

1. Look up the IP with DNS Checker's IP Location tool, check if the owning organization matches the claimed crawler. A "Googlebot" request from an IP owned by a residential ISP in Romania is fake.

2. Reverse DNS verification: legitimate search engine crawlers have verifiable PTR records:

# Step 1: Reverse DNS lookup
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

# Step 2: Forward DNS to confirm
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1
#   ← the address matches the one you started with, so this is genuine
#     Googlebot. If it does not match, or the PTR is missing, it is not.

# Combined verification script
dig -x 198.51.100.47 +short | xargs -I{} dig {} +short

This two-step verification (reverse lookup, then forward lookup to confirm the IP matches) is the gold standard for confirming Google, Bing, and other major search engine crawlers.

SEO Crawlers: The Usual Suspects

SEO crawlers are the most common source of "is this an attack?" false alarms. They hit your server hard because they are designed to crawl entire sites quickly, sometimes thousands of pages in minutes.

How to Tell Internal vs. External Crawlers

The critical distinction is whether someone in your organization triggered the crawl:

• Screaming Frog, Sitebulb: these are desktop applications. If the source IP is your office network or a VPN exit node, someone on your team is running it. Talk to your marketing or SEO team before filing anything.
• AhrefsBot, SemrushBot, MJ12bot: these are third-party services crawling the open web. They typically respect robots.txt and have published IP ranges. If their crawl rate is causing problems, robots.txt or rate limiting is the appropriate response, not an abuse report.
• Botify, Lumar: enterprise tools that may be run by an agency working for your company. Check with whoever manages your SEO.

AI Crawlers: The New Heavy Hitters

AI crawlers are the fastest-growing category of bot traffic and the most likely to trigger abuse report instincts. They hit harder than traditional search engine crawlers because many operate at aggressive rates, lack politeness delays, and some ignore robots.txt entirely.

Two entries in that table need a caveat, because they behave differently from the rest.

Google-Extended and Applebot-Extended are robots.txt control tokens, not crawlers. Google is explicit that Google-Extended "doesn't have a separate HTTP request user agent string" and that crawling is done with existing Google user agent strings, with the robots.txt token used purely in a control capacity. So you will never find Google-Extended in an access log, and grepping for it is wasted effort. You put it in robots.txt to opt out of Gemini training, and you verify the opt-out by the absence of AI use, not by log analysis.

OAI-SearchBot is a search crawler, not a training crawler. It controls whether your content can be discovered and included in ChatGPT search results and summaries. GPTBot is the one to disallow if your goal is opting out of model training. Blocking OAI-SearchBot is a visibility decision, closer to blocking Googlebot than to blocking a scraper, so decide on it separately.

Why AI Crawlers Hit Harder Than Search Engines

Traditional search engine crawlers like Googlebot are designed for long-term relationships with websites. They implement polite crawl rates, back off when servers return 429 or 503 status codes, and adjust their crawl budget based on server responsiveness.

Many AI crawlers operate differently:

• Aggressive crawl rates: some AI crawlers send 50-100+ requests per second without built-in politeness delays
• Full-site scraping: instead of indexing a few pages, they try to download everything for training data
• Ignoring signals: some crawlers do not respect Crawl-delay directives or slow down in response to HTTP 429 responses
• Bandwidth impact: a single AI crawler can consume 10-50 GB of bandwidth per day on a medium-sized site, compared to 1-5 GB for Googlebot

I have seen AI crawlers generate enough traffic to cause visible CPU spikes on production servers. But this is not malicious behavior. It is aggressive but legitimate crawling. The correct response is robots.txt or server-level rate limiting, not an abuse report. If a crawler continues after you have blocked it in robots.txt and at the server level, that is a different story, and that does warrant an abuse report.

Search Engine Crawlers (Do Not Block These)

Search engine crawlers drive your organic traffic. Blocking them means your site disappears from search results.

Verification Technique

Always verify before trusting a search engine User-Agent string. The reverse-then-forward DNS technique catches impersonators:

# Verify Googlebot
ip="66.249.66.1"
ptr=$(dig -x $ip +short)
echo "PTR: $ptr"
# Should be: crawl-66-249-66-1.googlebot.com.

fwd=$(dig $ptr +short)
echo "Forward: $fwd"
# Should match the original IP: 66.249.66.1

# If forward DNS matches original IP → genuine
# If it doesn't match or PTR doesn't resolve → fake

If you are seeing high traffic from a "Googlebot" that fails this verification, that is an impersonator and worth investigating further. Fake Googlebot traffic can indicate scraping, competitive intelligence gathering, or actual malicious activity.

How to Manage Crawlers with robots.txt

The robots.txt file is the standard mechanism for telling crawlers what they can and cannot access on your site. It lives at the root of your domain (e.g., https://example.com/robots.txt) and uses a simple directive syntax.

robots.txt Basics

# Allow all crawlers to access everything (default if no robots.txt exists)
User-agent: *
Allow: /

# Block a specific crawler from everything
User-agent: GPTBot
Disallow: /

# Block a crawler from specific paths
User-agent: SemrushBot
Disallow: /admin/
Disallow: /api/
Allow: /

# Set a crawl rate (seconds between requests)
User-agent: AhrefsBot
Crawl-delay: 10

Key directives:

• User-agent: which crawler this rule applies to (* means all)
• Disallow, paths the crawler should not access
• Allow, paths the crawler can access (overrides Disallow for more specific paths)
• Crawl-delay, seconds to wait between requests (not supported by all crawlers)
• Sitemap, URL of your XML sitemap

Block All AI Crawlers

If you want to opt out of AI training while keeping search engines and SEO tools working. Note that this block goes further than a training opt-out: ChatGPT-User handles user-triggered browsing and OAI-SearchBot governs inclusion in ChatGPT search, so disallowing them costs you AI-surface visibility rather than training exposure. Drop those two lines if you only want out of training:

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

# Below this line is a visibility choice, not a training opt-out:
# ChatGPT-User = user-triggered browsing
# OAI-SearchBot = discovery/inclusion in ChatGPT search
User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: Omgilibot
Disallow: /

# Allow search engines (default: everything allowed)
User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

Sitemap: https://example.com/sitemap.xml

Rate-Limit Aggressive SEO Crawlers

If you do not want to block SEO crawlers entirely but need them to slow down:

User-agent: AhrefsBot
Crawl-delay: 10

User-agent: SemrushBot
Crawl-delay: 10

User-agent: MJ12bot
Crawl-delay: 30

User-agent: DotBot
Crawl-delay: 10

Crawl-delay: 10 asks the crawler to wait 10 seconds between requests. Support varies, and the two big search crawlers differ:

• Bingbot honours it. Microsoft documents this explicitly, and defines it as a time window of 1 to 30 seconds during which Bingbot will crawl your site only once, rather than as a rate. A delay of 5 means Bing slices the day into five-second windows and takes at most one page from each.
• Googlebot ignores it. There is also no longer a Search Console control to fall back on: Google deprecated the Crawl Rate Limiter tool on 8 January 2024. Googlebot now backs off automatically in response to your server, slowing down when it sees persistent HTTP 500s or rising response times. If you need to throttle it, return 429, 500, or 503 for a period, and use Google's reporting form for genuinely unmanageable crawling.

Limitations of robots.txt

This is the most important thing to understand: robots.txt is voluntary. It is a convention, not a technical enforcement mechanism. Well-behaved crawlers obey it; malicious scrapers and some aggressive bots ignore it entirely.

robots.txt does not:

• Prevent access (it is publicly readable at /robots.txt, so it can actually reveal paths you might prefer to keep less visible)
• Enforce rate limits (a crawler that ignores the file will ignore Crawl-delay too)
• Block direct URL requests (users and bots can still access Disallowed URLs directly)

When robots.txt is not enough, you need server-level controls.

Beyond robots.txt: Server-Level Controls

When a crawler ignores your robots.txt or you need hard enforcement, server-level controls are the answer.

Nginx Rate Limiting by User-Agent

The trick here is that Nginx does not account requests whose zone key evaluates to an empty string. So map the User-Agent straight to the key: bots get their address as the key and are counted, everyone else gets "" and passes through untouched.

# Bots get a non-empty key (their address) and are rate limited.
# Everyone else maps to "" and is not accounted at all.
map $http_user_agent $bot_limit_key {
    default             "";
    "~*AhrefsBot"       $binary_remote_addr;
    "~*SemrushBot"      $binary_remote_addr;
    "~*MJ12bot"         $binary_remote_addr;
    "~*GPTBot"          $binary_remote_addr;
    "~*ClaudeBot"       $binary_remote_addr;
    "~*Bytespider"      $binary_remote_addr;
    "~*CCBot"           $binary_remote_addr;
    "~*PerplexityBot"   $binary_remote_addr;
}

limit_req_zone $bot_limit_key zone=bot_limit:10m rate=1r/s;

server {
    location / {
        limit_req zone=bot_limit burst=5 nodelay;
        # ... your normal config
    }
}

Do not key the zone on $binary_remote_addr directly and then try to gate it with an if ($is_bot) block, which is a pattern you will see in the wild. Every request has a non-empty $binary_remote_addr, so the zone counts everyone and you end up rate-limiting your actual users at 1 request per second. The if block never enters into it, because the zone key was already decided.

Apache Rate Limiting

# Block specific crawlers entirely
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|Bytespider|CCBot) [NC]
RewriteRule .* - [F,L]

# Or return 429 Too Many Requests
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|Bytespider|CCBot) [NC]
RewriteRule .* - [R=429,L]

Cloudflare Bot Management

If you use Cloudflare, their Bot Management and WAF rules provide the easiest server-level controls:

1. Super Bot Fight Mode (free plans), automatically challenges or blocks bots categorized as "automated"
2. WAF Custom Rules: create rules that match specific User-Agent strings and block, challenge, or rate-limit them
3. AI Bot blocking: Cloudflare added a one-click toggle to block known AI crawlers in 2024

Example Cloudflare WAF rule expression:

(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Bytespider") or
(http.user_agent contains "CCBot")

Set the action to "Block" or "Managed Challenge" depending on how aggressively you want to handle these.

Firewall Rules for Known CIDR Ranges

Some AI crawlers publish their IP ranges, which lets you block at the network level regardless of User-Agent:

# OpenAI published ranges (check their documentation for current ranges)
# Block GPTBot at the firewall
sudo ufw deny from 20.15.240.0/24 comment "OpenAI GPTBot"
sudo ufw deny from 20.15.241.0/24 comment "OpenAI GPTBot"

# Or with iptables
iptables -A INPUT -s 20.15.240.0/24 -j DROP -m comment --comment "OpenAI GPTBot"

When to use firewall-level blocks:

• The crawler ignores robots.txt
• The crawler spoofs its User-Agent to bypass User-Agent-based rules
• You need to block before the request even reaches your web server (saves CPU)
• The crawler is generating enough traffic to impact server performance

The Decision Framework: Block, Allow, or Report?

Here is the framework I use when I see an IP generating heavy traffic on a server:

Step 1: Identify the Source

Check the User-Agent in your access logs and look up the IP with the IP Location tool. This immediately tells you if it is a known crawler, which organization owns the IP, and whether the User-Agent matches the IP owner.

Step 2: Classify the Crawler

Based on what you find:

Known search engine (Googlebot, Bingbot, etc.) → Allow. These drive your traffic. If they are crawling too aggressively, throttle Bingbot with Crawl-delay or Bing Webmaster Tools' crawl control. For Googlebot there is no crawl-rate setting any more (Search Console's limiter was retired in January 2024), so slow it by returning 429/503 responses, which Googlebot reacts to automatically. Do not file an abuse report.

SEO tool from your own company → Talk to your team. Screaming Frog, Sitebulb, or a Botify crawl triggered by your marketing or SEO team is internal traffic. Coordinate crawl schedules to avoid impacting production. Definitely do not file an abuse report.

Third-party SEO crawler (AhrefsBot, SemrushBot) → Rate-limit or block via robots.txt. These respect robots.txt. Set a Crawl-delay or block specific paths. An abuse report is not appropriate here.

AI training crawler (GPTBot, ClaudeBot, CCBot) → Block via robots.txt if you do not want your content used for AI training. Most AI crawlers respect robots.txt. If they do not, escalate to server-level blocks.

Crawler ignoring robots.txt → Server-level block. Use Nginx/Apache rules or firewall rules. If the behavior is persistent and the operator is unresponsive to contact, this may warrant an abuse report.

Unknown bot with malicious behavior patterns → Investigate and potentially report. If the traffic does not have a legitimate crawler User-Agent, the IP does not belong to a known crawling service, and the behavior looks like scraping, DDoS, or reconnaissance, that is when you file an abuse report using the complete guide to reporting IP abuse.

When an Abuse Report Is Warranted

To be clear, there are situations where crawling behavior does cross the line into abuse:

• The crawler is ignoring robots.txt and you have contacted the operator without response
• The traffic is causing service degradation and the operator will not implement rate limiting
• The crawler is spoofing User-Agent strings to impersonate legitimate bots
• The IP does not belong to any known crawling service and the behavior is clearly automated scraping
• The "crawler" is actually performing port scanning or brute-force attacks alongside its requests

Should You Block AI Crawlers? Pros and Cons

This is the question every sysadmin is wrestling with in 2025. There is no universally right answer. It depends on your organization's priorities.

Reasons to Block

• Bandwidth and server load: AI crawlers can consume significant resources. If you are paying for bandwidth, this has a direct cost.
• Content used without consent: your content may be used to train models that compete with you or generate derivative content without attribution.
• No direct benefit: unlike search engine crawlers, AI training crawlers do not drive traffic to your site.
• Legal uncertainty: the legal status of AI training on web content is still being litigated in multiple jurisdictions.

Reasons to Allow

• AI search visibility: Perplexity, ChatGPT with browsing, and Google AI Overviews are becoming significant traffic sources. Blocking their crawlers may reduce your visibility in AI-powered search.
• Future discoverability: as AI becomes a primary way people find information, content that has been crawled and indexed by AI systems may have better long-term reach.
• Selective blocking is possible: you can block training crawlers (GPTBot, Google-Extended) while allowing search-adjacent ones (ChatGPT-User, OAI-SearchBot, PerplexityBot).

My Recommendation

Block the pure training crawlers (GPTBot, Google-Extended, ClaudeBot, Bytespider, CCBot, Meta-ExternalAgent) via robots.txt. Consider allowing the search-adjacent crawlers (ChatGPT-User, PerplexityBot, OAI-SearchBot) if AI search traffic matters to your business. Monitor your access logs monthly to stay on top of new crawlers as they appear. This landscape changes fast.