Robots.txt Checker: Test Whether Googlebot and AI Crawlers Can Reach Your Pages
A robots.txt checker tells you, for any URL on your site, whether a particular crawler is allowed to fetch it and which rule decides the answer. Our free robots.txt tester downloads your live robots.txt, parses it the way Google does, and tests your URL against Googlebot, Bingbot and the AI crawlers that power ChatGPT, Claude and Perplexity. You can also paste or edit rules and test them before you publish, so a single typo never blocks your whole site from search.
This guide explains how robots.txt works, how Google matches rules, how to read the tester’s results and how to fix the mistakes that quietly cost sites their rankings.
Key takeaways
- robots.txt controls crawling, not indexing: a blocked URL can still appear in search results without a description.
- Google uses the most specific user-agent group, then the longest matching rule; Allow wins a tie.
- Never block CSS, JavaScript or images that pages need to render.
- Block AI training bots if you want, but keep AI search bots allowed if you want to be cited in AI answers.
What is robots.txt?
robots.txt is a plain-text file at the root of a domain, for example https://example.com/robots.txt, that tells crawlers which parts of the site they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309, and Google documents its own interpretation in the robots.txt specification.
What robots.txt can and can’t do
- It can stop well-behaved crawlers from fetching URLs, which saves server resources and keeps crawlers away from low-value areas such as internal search results.
- It can’t reliably keep a page out of search results. If other pages link to a blocked URL, Google can still index the URL without seeing its content.
- It can’t protect private content. The file is public, and malicious bots ignore it. Use passwords or authentication for anything sensitive.
Google’s introduction to robots.txt explains when to use it and when to use noindex instead.
Why you need a robots.txt checker
Rules interact in ways that aren’t obvious. A broad Disallow can override what you meant to allow, a wildcard can match more than you intended, and a bot with its own group ignores the rules in the * group entirely. A checker removes the guesswork by showing you the exact rule and line that applies.
How to use our robots.txt tester
- Enter your website or a page URL and click Fetch robots.txt. The tool downloads the file from that domain and shows its status, size and response time.
- Check the URL or path to test. If you entered a page URL, its path is filled in automatically.
- Choose a crawler from the list or enter a custom user-agent such as AhrefsBot.
- Read the verdict: Allowed or Blocked, with the matching line and the user-agent group that decided it.
- Check the table of every crawler to see the result for Googlebot, Bingbot and AI bots at once.
- Edit the rules in the text box to try fixes. Your live file isn’t changed.
How Google reads robots.txt rules
User-agent groups
A file is made of groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules. A crawler obeys only one group: the one with the most specific user-agent that matches its name. If none matches, it follows the User-agent: * group.
Why this catches people out
If you add a group for Googlebot-Image, Googlebot-Image ignores everything in the * group. Repeat any rules you want it to follow in its own group.
Real-world case
A site blocked /private/ for all bots, then added a separate group to slow down one SEO crawler. That crawler started fetching /private/, because its own group didn’t repeat the rule. The checker shows this instantly: test the path with that crawler selected.
Longest match wins
Within a group, Google applies the rule with the longest matching path. If an Allow and a Disallow match with the same length, Allow wins.
Example
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpFor /wp-admin/admin-ajax.php, both rules match, but the Allow path is longer, so the file is allowed. Every other URL in /wp-admin/ is blocked.
Wildcards
*matches any sequence of characters:Disallow: /*?sort=blocks URLs with a sort parameter.$marks the end of the URL:Disallow: /*.pdf$blocks URLs ending in .pdf.
Paths are case-sensitive
Disallow: /Private/ doesn’t block /private/. Directive names such as Disallow aren’t case-sensitive, but paths are.
What happens when robots.txt is missing or broken
| Response | How Google treats it |
|---|---|
| 200 with rules | Follows the rules |
| 404 or other 4xx | Assumes there are no restrictions |
| 5xx server error | Treats the site as temporarily disallowed and pauses crawling |
| Over 500 KiB | Ignores rules after the size limit |
Our checker reports each of these situations, including when a server returns an HTML page instead of a text file.
We built the matching engine to follow Google’s published rules and tested it against the worked examples in Google’s specification, so the verdict you see matches how Googlebot behaves.
Understanding the crawler list
Search engine crawlers
Googlebot (and its variants for images, news and ads) and Bingbot index pages for search. Blocking them on pages you want to rank is almost always a mistake.
AI search crawlers
OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot fetch pages to answer questions and cite sources in AI search. If you block them, your pages can’t be cited in those answers.
AI training crawlers
GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini and Vertex AI training) and CCBot (Common Crawl) collect content used to train AI models. Blocking these is a business decision and doesn’t affect search rankings. Note that Google-Extended is a control token: it doesn’t crawl separately, it tells Google how content fetched by Googlebot may be used.
A balanced setup for most businesses
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlThis keeps search and AI search crawlers allowed while opting out of model training. Build your own version with our robots.txt generator for WordPress.
The most common robots.txt mistakes
1. Blocking the whole site after launch
Development sites often use Disallow: /, and it gets copied to the live site. One line can remove a site from search over a few weeks. Test your home page in the checker after every launch.
2. Blocking CSS, JavaScript or images
Rules like Disallow: /wp-content/ or Disallow: /*.js$ stop Google from rendering pages properly, which can hurt how it evaluates them. Remove these rules.
3. Using robots.txt to remove pages from Google
Blocking a page doesn’t remove it from the index. To remove it, allow crawling and add a noindex tag, as explained in Google’s guide to blocking indexing. Then check the page with our Google index checker.
4. Noindex in robots.txt
Google stopped supporting Noindex: lines in robots.txt in 2019. The checker flags them so you can replace them with meta robots tags.
5. Missing Sitemap line
Adding Sitemap: with the full URL helps every search engine find your sitemap. Validate it with our sitemap validator.
6. Typos and invalid lines
Misspelled directives such as Dissallow, rules before any User-agent line and paths without a leading slash are ignored or misread. The File check section of the report lists each problem with its line number.
7. Relying on Crawl-delay for Google
Googlebot ignores Crawl-delay. Bing respects it, so a high value can slow how quickly Bing and Copilot see your updates.
How to edit robots.txt
WordPress
WordPress serves a virtual robots.txt by default. Yoast SEO (Tools → File editor) and Rank Math (General Settings → Edit robots.txt) let you edit it from the dashboard. If a physical robots.txt file exists in your site root, it overrides the virtual one.
Shopify
Shopify generates robots.txt automatically. You can customise it by adding a robots.txt.liquid template to your theme.
Other platforms and servers
Upload a plain UTF-8 text file named robots.txt to the root of each host. Subdomains need their own file: blog.example.com/robots.txt is separate from example.com/robots.txt.
After you publish changes
- Fetch the live file in our robots.txt checker and test key URLs.
- In Google Search Console, open Settings → robots.txt to see when Google last fetched it and request a recrawl if needed.
- Watch the Page indexing report for new “Blocked by robots.txt” entries.
robots.txt and crawl budget
For most small and medium sites, crawl budget isn’t a concern; Google crawls them comfortably. On large sites with millions of URLs, faceted navigation or endless calendar pages, robots.txt helps steer crawlers away from near-duplicate URLs so they spend more time on important pages.
What to block on large sites
- Internal site search results, for example
/?s=or/search/. - Endless filter and sort combinations that create duplicate listings.
- Cart, checkout and account areas.
What not to block
- Pages you want to rank, including category and product pages.
- Resources needed for rendering: CSS, JavaScript, fonts and images.
- URLs with canonical tags pointing elsewhere: Google needs to crawl them to see the canonical.
When to use a robots.txt checker
Robots.txt mistakes are silent: nothing on your site looks broken, but crawlers quietly stop visiting pages. Make a robots.txt checker part of these routines:
Before and after a website launch or redesign
Staging rules are the single most common cause of a new site failing to appear in Google. Test the home page and a few key pages on launch day and again a week later.
After installing or changing an SEO or security plugin
Plugins can rewrite the virtual robots.txt in WordPress. A quick check confirms nothing important has been blocked.
When pages drop out of search
If Search Console shows “Blocked by robots.txt” or traffic to a section suddenly falls, test those URLs first. The checker points to the exact line responsible.
When deciding on AI crawler access
Before blocking or allowing AI bots, see how your current file treats each one. Many sites discover they’re blocking AI search crawlers they actually want, through an old catch-all rule.
When auditing competitors
Because robots.txt is public, you can check any site. It’s a quick way to see which sections competitors keep out of crawlers’ reach and how they handle AI bots.
robots.txt checklist
- The file is at the domain root, returns 200 and is plain text under 500 KiB.
- The home page and key pages are allowed for Googlebot and Bingbot.
- CSS, JavaScript and image folders are not blocked.
- AI search crawlers are allowed if you want to appear in AI answers.
- Training crawlers are allowed or blocked according to your policy.
- No
Noindex:lines and no typos. - A
Sitemap:line with a full URL. - Tested in a robots.txt checker after every change.
Related free SEO tools
- Robots.txt generator for WordPress: build a correct file with AI crawler rules.
- Google index checker: check every reason a page might not be indexed.
- Sitemap validator: make sure your sitemap is valid and reachable.
- Free SEO checker: includes an AI search readiness check.
- 301 redirect checker: trace where your URLs end up.
Ready to test your rules? Run the robots.txt checker above and see exactly which crawlers can reach your pages.