See whether Googlebot and AI crawlers are allowed on a path, catch site-wide Disallow mistakes, and list Sitemap advertisements.
- Free
- No signup
- Fast
- Privacy-friendly
What is Robots.txt Checker?
A single bad robots.txt line hides an entire site from crawlers. Disallow: / under User-agent: * blocks matching bots site-wide. Missing Sitemap: lines slow discovery of new URLs. Publishers also require a clear answer on whether GPTBot, Google-Extended, or ClaudeBot are allowed to fetch a given path for training or grounding.
This Robots.txt Checker fetches /robots.txt for the origin of your URL, parses User-agent groups, resolves Allow/Disallow for the audited path (including Googlebot), builds an AI-bot access matrix, lists Sitemap advertisements, and scores common policy mistakes with evidence and fixes.
Use it after a CMS or CDN robots change, before a launch behind staging rules, when Search Console shows crawl anomalies, or when you set AI-crawler preferences.
What are common questions about Robots.txt Checker?
What problem does the Robots.txt Checker solve?
It exposes accidental crawl blocks, missing Sitemap lines, and AI-bot allow/deny status for a path so crawl-policy mistakes are fixed before index or agent access drifts.
Who needs a robots.txt audit?
Technical SEO, site ops, and platform teams use it after deploys, CDN rule changes, staging cutovers, and AI-bot policy decisions.
What does Disallow: / mean for User-agent *?
Disallow: / for * is a site-wide crawl block for bots that honor the star group. Already discovered URLs often stay indexed until noindex or removals apply.
Which AI bots appear in the access matrix?
Matrices include GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, PerplexityBot, Bytespider, CCBot, and Applebot-Extended resolved against the audited path.
Do Sitemap lines replace XML sitemaps?
No. Sitemap lines only advertise XML locations. Validate inventory structure with the Sitemap XML Validator.
Is robots.txt a security boundary?
No. Robots.txt is crawl guidance, not authentication. Sensitive paths require real access control on the server.
How do Allow and Disallow conflicts resolve?
Conflicts resolve by longest matching path prefix for the matching User-agent group. Specific bot groups override * when the product token matches.
What HTTP status is correct for robots.txt?
A healthy file returns HTTP 200 with a text/plain body. 404 means missing policy; 5xx blocks reliable discovery.
What are the key features of Robots.txt Checker?
User-agent group parsing
Splits robots.txt into User-agent groups with Allow and Disallow rule lists for review.
AI-bot access matrix
Maps GPTBot, Google-Extended, ClaudeBot, and related agents against the audited pathname.
Googlebot path verdict
States whether Googlebot Allow/Disallow rules permit the current path.
Sitemap advertisement list
Extracts Sitemap: lines so discovery URLs match live XML indexes.
Fetch status evidence
Records the robots.txt URL, HTTP status, and unreachable-file errors.
Policy scoring
Penalizes missing files, site-wide Disallow:/, and missing Sitemap lines with documented weights.
How do you use Robots.txt Checker?
Enter any URL on the target host
Paste a page URL on the origin. The checker always fetches /robots.txt for that host.
Run the checker
Parse groups, build the AI-bot matrix, evaluate the path, and score crawl-policy health.
Review matrix and Sitemap lines
Confirm intended Allow/Disallow rules and that Sitemap lines match live XML indexes.
Why robots.txt mistakes are high-impact
Robots.txt sits at the origin root and applies before most HTML audits. A staging rule left on production (Disallow: /) stops Googlebot and many other crawlers. A forgotten Sitemap line delays discovery after a large content push. Conflicting Allow/Disallow lines on nested paths confuse which URLs are crawlable.
IETF RFC 9309 (June 2022) standardizes User-agent records, rule lines, and Sitemap lines. Google Search Central documentation treats robots.txt as crawl guidance — not a firewall and not a substitute for noindex when the goal is removal from Search.
The checker reports the concrete evidence: file URL, status, matched group, path verdict, and AI-bot allow/deny cells — so fixes target the exact line, not a vague “check robots.”
Use cases: launches, AI bots, and sitemap discovery
Launch / un-block — After promoting staging, confirm * and Googlebot are not Disallow:/ and that money paths (/blog/, /products/) remain allowed.
AI-crawler policy — When legal or product asks whether training crawlers fetch the site, read the matrix for GPTBot, Google-Extended, ClaudeBot, and related agents for a representative path.
Discovery wiring — When new sections never appear in Search Console coverage, verify Sitemap: lines point at reachable sitemap.xml or index files, then validate XML on the Sitemap XML Validator.
Incident response — Sudden crawl drops after a CDN or WAF change: confirm robots.txt still returns 200 and that rules did not widen Disallow prefixes.
What the checker covers and its limits
Coverage includes prefix matching for Allow/Disallow, Googlebot and star-group path verdicts, AI-bot matrices, Sitemap line extraction, and documented policy scoring. Empty Disallow: allows all paths for that group; comments starting with # are ignored as policy.
Limits: exotic unsupported wildcard extensions and nonstandard Crawl-delay variants are not executed. Fetch timeouts bound remote reads. Robots.txt never authenticates browsers — protect private resources with auth, not Disallow lines alone.
Keep macro contexts separate: path policy stays here; URL inventory quality stays on Sitemap XML validation.
Free · API · MCPToolYour platform
Use Robots.txt Checker three ways
The free Robots.txt Checker on this page works in your browser. The same workflow is available via REST API for apps and via the remote MCP server for Cursor, Claude, and other agents — one API key, one plan, one quota. This tool is part of the seo-apis module under Digital Tools.
- Free in browserUse Robots.txt Checker here — no account required.
- REST APICall from your server with
X-Api-Key. - Remote MCP serverSame job from Cursor or Claude via the ToolYour remote MCP server — one API key with REST.
- 1. Get an API keySign in and open Dashboard → API Keys (works for REST and MCP).
- 2. Call or connectUse the REST API from your server, or paste the MCP URL into your agent config.
- 3. Same key & creditsREST and MCP share one API key and monthly credit quota with this browser tool when the tool is API-backed.
Which technical SEO tools sit in this cluster?
- Page speed analyzer
Load and performance signals for a URL.
- Sitemap XML validator
Sitemap coverage and crawl-load indicators.
- Canonical URL checker
Self-canonical, chains, and conflicts.
- HTTP status checker
Final status codes for publish and crawl hygiene.
- Redirect chain analyzer
Hop graph and crawl latency on redirect paths.
Where does this job fit on ToolYour?
Which security checks pair with this SEO workflow?
Where do you browse more digital tools tools?
- Digital Tools tools
Browse the full Digital Tools category on ToolYour.
- Developer Tools tools
Format JSON, encode Base64/URL, mint UUIDs, convert timestamps, and test regex in the browser. REST and MCP sh
- Marketing Tools tools
Build UTM links, check ads copy limits, QA email subjects, and calculate ROAS/CPC/CTR — free in the browser; R