DNS Checker Bot & Scanner Documentation
DNS Checker operates two automated systems: a web crawler for on-page SEO audits and a DNS intelligence scanner that resolves domain records across 1,100+ TLD zone files. Both systems are designed with responsible scanning practices, honor opt-out requests, and follow industry best practices established by organizations like CAIDA and Censys.
DNSChkr-WebCheck (Web Crawler)
DNSChkr-WebCheck powers on-page SEO audits, link validation, and structured data checks across the free tool suite.
User-Agent
What It Does
On-Page SEO Audits
Fetches a single page to analyze meta tags, headings, structured data, images, links, and content quality.
Link Validation
Sends lightweight HEAD requests to verify that URLs referenced on a page return valid HTTP responses.
Security Header Checks
Reads HTTP response headers to evaluate security configurations like HSTS, CSP, and X-Frame-Options.
Redirect Tracing
Follows HTTP redirect chains to detect redirect loops, excessive hops, and protocol downgrades.
Crawl Behavior
Controlling Access
To block DNSChkr-WebCheck from accessing your site, add the following to your robots.txt file:
- Not a search engine crawler. DNS Checker does not index pages or build a search index.
- Not an AI training crawler. Page content is never stored or used for machine learning.
- User-initiated only. Every request originates from a real user running an audit through dnschkr.com.
DNSChkr Echo (DNS Intelligence Scanner)
DNSChkr Echo is a DNS research scanner that resolves domain name records across 1,100+ TLD zone files. The resulting data powers DNS security research, infrastructure analysis, and the publicly available insights on dnschkr.com — including the TLD Directory, rankings, provider analytics, and security research.
Overview
This is not port scanning or network probing. DNS Checker Echo sends standard DNS queries that are indistinguishable from normal recursive resolver traffic:
- Queries use DNS protocol only (UDP/TCP port 53) — no HTTP requests, no connection attempts to web servers, no fingerprinting
- Only domain names from published zone files are queried — no brute-force subdomain enumeration or dictionary attacks
- Resolvers cache responses per TTL, so the same nameserver is not queried repeatedly for the same domain
DNS Record Types & Research Use
The scanner queries 8 standard DNS record types for each domain. Each record type contributes to a different area of DNS research and security analysis:
| Record | RFC | Research Application |
|---|---|---|
| NS | RFC 1035 | Nameserver delegation analysis — provider market share, lame delegation detection, concentration risk |
| A | RFC 1035 | IPv4 infrastructure mapping — hosting provider identification, IP geolocation, shared hosting analysis |
| AAAA | RFC 3596 | IPv6 adoption research — tracking deployment rates across TLDs, providers, and regions |
| MX | RFC 5321 | Email infrastructure analysis — mail provider market share, domains actively receiving email |
| TXT | RFC 1035 | Security policy research — SPF adoption, DKIM deployment, domain verification patterns |
| CNAME | RFC 1035 | Infrastructure pattern detection — CDN adoption, domain parking, forwarding services |
| CAA | RFC 8659 | Certificate authority research — CA authorization patterns, HTTPS adoption trends |
| SOA | RFC 1035 | Zone administration data — serial numbers, refresh intervals, responsible party identification |
Research findings are published through the security dashboard, ranking pages, and blog. Individual per-domain DNS records are never exposed publicly.
Scanner IPs & Verification
All DNS Checker scanning infrastructure uses dedicated IP addresses with matching forward and reverse DNS records, allowing any nameserver operator to verify the source of queries.
Echo Scanner Fleet
DNS research queries originate from the following six dedicated servers. Each IP has a matching PTR (reverse DNS) record that resolves back to the forward A record for verification.
37.27.58.50echo-1.scanner.dnschkr.com46.4.73.107echo-2.scanner.dnschkr.com46.4.68.157echo-3.scanner.dnschkr.com65.109.111.103echo-4.scanner.dnschkr.com65.109.62.140echo-5.scanner.dnschkr.com135.181.61.236echo-6.scanner.dnschkr.comWebCheck Crawler IPs
See the full list at /bot/ips (also available as JSON and plain text).
How to Verify a DNS Checker Request
- Reverse DNS: Perform a PTR lookup on the source IP. It should resolve to a
*.scanner.dnschkr.comor*.dnschkr.comhostname. - Forward DNS: Resolve the hostname from step 1. It should point back to the original source IP.
- For web requests: Confirm the User-Agent contains
DNSChkr-WebCheck. - For DNS queries: The Unbound resolver identity is set to
dnschkr-echo-scanner.
All scanner IPs are static and dedicated to DNS Checker research infrastructure. Last updated April 27, 2026.
Permanent Exclusion List
DNS Checker maintains a permanent exclusion list of IP ranges, TLDs, and domains that are never scanned, following guidance from IANA Special-Purpose Address Registries (RFC 6890, RFC 8190) and responsible scanning practices established by the ZMap project.
Excluded IP Ranges (IANA Special-Purpose)
| CIDR Range | Purpose | Reference |
|---|---|---|
| 0.0.0.0/8 | This host on this network | RFC 1122 |
| 10.0.0.0/8 | Private network | RFC 1918 |
| 100.64.0.0/10 | Shared Address Space (carrier-grade NAT) | RFC 6598 |
| 127.0.0.0/8 | Loopback | RFC 1122 |
| 169.254.0.0/16 | Link-local | RFC 3927 |
| 172.16.0.0/12 | Private network | RFC 1918 |
| 192.0.0.0/24 | IETF Protocol Assignments | RFC 6890 |
| 192.0.2.0/24 | Documentation (TEST-NET-1) | RFC 5737 |
| 192.31.196.0/24 | AS112-v4 | RFC 7535 |
| 192.52.193.0/24 | Automatic Multicast Tunneling | RFC 7534 |
| 192.88.99.0/24 | 6to4 Relay Anycast (deprecated) | RFC 7526 |
| 192.168.0.0/16 | Private network | RFC 1918 |
| 192.175.48.0/24 | Direct Delegation AS112 Service | RFC 7534 |
| 198.18.0.0/15 | Benchmarking | RFC 2544 |
| 198.51.100.0/24 | Documentation (TEST-NET-2) | RFC 5737 |
| 203.0.113.0/24 | Documentation (TEST-NET-3) | RFC 5737 |
| 233.252.0.0/24 | MCAST-TEST-NET | RFC 5771 |
| 224.0.0.0/4 | Multicast | RFC 5771 |
| 240.0.0.0/4 | Reserved for future use | RFC 1112 |
| 255.255.255.255/32 | Limited broadcast | RFC 919 |
Excluded TLDs
| TLD | Reason |
|---|---|
| .mil | US Military (DISA) |
| .gov | US Government (CISA) |
| .arpa | Infrastructure TLD |
| .example | IANA reserved (RFC 2606) |
| .test | IANA reserved (RFC 2606) |
| .invalid | IANA reserved (RFC 2606) |
| .localhost | IANA reserved (RFC 2606) |
Per RFC 2606, all subdomains of example.com, example.net, and example.org are also excluded.
How to Opt Out
DNS Checker honors all opt-out requests. Requests are typically processed within 24 hours and take effect before the next weekly scan cycle.
For WebCheck (Web Crawler)
Add the robots.txt directive shown above. DNSChkr-WebCheck respects robots.txt before making any request.
For Echo (DNS Scanner)
Since DNS resolution queries authoritative nameservers through recursive resolvers (not your web server directly), robots.txt does not apply. To opt out, contact DNS Checker with one of:
- Domain names — specific domains or wildcard patterns (e.g., *.example.com)
- Nameserver hostnames — nameserver patterns to exclude (e.g., ns*.your-dns.com)
- IP ranges — CIDR blocks to avoid querying (e.g., 203.0.113.0/24)
- ASN — entire Autonomous System Numbers (e.g., AS12345)
What makes a valid opt-out request
To keep the research dataset complete and protect against unauthorized exclusions on behalf of resources the requester doesn't operate, opt-out requests are reviewed against the criteria below. Genuine requests from authoritative operators are accommodated quickly — usually within 24 hours, before the next scan cycle. Requests that don't meet these criteria are declined politely.
- Operational authority. You must operate the affected resource — DNS provider for the nameserver, IP block holder for an IP range, ASN holder for an autonomous system, or registrant for a specific domain. Third-party requests on behalf of resources you don't control are not honored.
- Specific scope. Provide an exact CIDR, domain pattern, nameserver hostname, or ASN — not vague language like "stop scanning us" or "exclude my company."
- Verifiable identity. The request must come from one of:
- The RIR / abuse contact registered for the IP range or ASN
- An email address at the affected domain (e.g.
[email protected]for a request aboutyourdomain.com) - A DNS TXT record at
_dnschkr-optout.yourdomain.comwith the valueopt-out— a quick way to prove DNS-level control without exchanging emails
- Operational reason. A short note on the operational, security, or compliance need — anything from "our abuse-detection system flags this volume" to "compliance policy requires it." A rate-limit (e.g. "please cap queries to AS12345 at 50 qps") is preferred over a full opt-out where it would meet the same need, since opt-outs reduce public DNS research coverage.
Requests targeting shared public infrastructure (TLD authoritative servers, root nameservers, large public DNS providers without operational standing) or that would otherwise compromise the integrity of the public research dataset are declined.
Contact for opt-out requests: [email protected] — include the resource type (domain, nameserver, IP range, ASN), the specific pattern, your verification method (RIR contact / domain email / DNS TXT record), and a short operational reason. Response within 24 hours either way.