Robots.txt Generator for WordPress: Build a Safe File in Seconds
Our robots.txt generator for WordPress builds a correct, ready-to-upload robots.txt file without you having to memorise the syntax. Choose whether crawlers are allowed by default, add the folders you want to keep private, pick which AI crawlers to block, add your sitemap and copy the result. It works for any website, but it includes one-click presets for WordPress and for filter and search URLs, plus live checks that stop you from accidentally blocking your whole site or the files Google needs to render your pages.
This guide explains what to put in robots.txt (and what to leave out), the ideal setup for a WordPress site, how to handle AI crawlers in 2026, and how to install and test your file.
Key takeaways
- A good WordPress robots.txt is short: block admin areas, allow
admin-ajax.php, list your sitemap. - Don’t block
/wp-content/,/wp-includes/, CSS or JavaScript: Google needs them to render pages. - Block AI training crawlers if you want, but keep AI search crawlers allowed if you want to be cited in AI answers.
- Always test the file before and after uploading it.
What does a robots.txt generator do?
A robots.txt generator writes the rules that tell search engine and AI crawlers which parts of your site they may fetch. Instead of typing directives by hand, you choose options and the generator outputs a valid file that follows the Robots Exclusion Protocol (RFC 9309) and Google’s guidance on creating robots.txt.
Why use a generator instead of writing it yourself
- Correct syntax: user-agent groups, Allow and Disallow lines and Sitemap lines in the right order.
- Safe defaults: warnings when a rule would block the whole site or rendering resources.
- Up-to-date AI crawler names: the exact user-agent tokens for GPTBot, ClaudeBot, Google-Extended and others.
- Speed: a complete file in under a minute, with a download button.
What robots.txt is not
It isn’t a security tool and it doesn’t remove pages from Google. The file is public, bad bots ignore it, and blocked URLs can still be indexed without their content. To keep a page out of search results, allow crawling and use a noindex tag.
How to use our robots.txt generator
- Choose the default for all crawlers. “Allow everything” is right for almost every live website. “Block everything” is only for staging or development sites.
- Add paths to disallow, one per line, such as
/wp-admin/or/cart/. Use paths, not full URLs. - Add paths to allow as exceptions inside blocked folders, such as
/wp-admin/admin-ajax.php. - Use a preset to add the recommended WordPress rules or rules for search and filter URLs in one click.
- Tick the AI crawlers to block. Each one gets its own group with
Disallow: /. - Add your sitemap URL in full, including
https://. - Review the checks and suggestions, then copy or download the file.
The best robots.txt for a WordPress site
WordPress doesn’t need a complicated robots.txt. Most of the rules you see in old tutorials do more harm than good. Here’s a safe, modern starting point:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlWhy block /wp-admin/
The admin area has no value in search results and requires a login anyway. Blocking it keeps crawlers focused on public pages.
Why allow admin-ajax.php
Many themes and plugins load content, forms and filters through admin-ajax.php. If Google can’t fetch it, parts of your pages may not render for Googlebot. The Allow rule is longer than the Disallow rule, so it wins for that file.
Why not block wp-includes or wp-content
These folders contain the CSS, JavaScript, fonts and images your pages need. Google renders pages like a browser and needs those files to see your layout and content. Old advice to block them is outdated.
Optional rules for WordPress
Internal search results
Search result pages such as /?s=keyword can create endless low-value URLs. Blocking them is reasonable on larger sites:
Disallow: /?s=
Disallow: /search/WooCommerce cart, checkout and account
These pages are personal to each visitor and shouldn’t rank. Many stores block them; others prefer a noindex tag so crawlers can still follow links. Either approach is acceptable.
Example
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/Filter and sort parameters
Faceted navigation (colour, size, price, sort order) can generate thousands of near-duplicate URLs. Blocking parameter patterns such as /*?orderby= or /*?filter_ helps large stores, but don’t block filter pages you deliberately optimise for search.
How to handle AI crawlers in robots.txt
AI companies use two kinds of crawlers, and treating them the same is a common mistake.
AI search crawlers: usually keep them allowed
OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot fetch pages to answer questions and cite sources in AI search tools. If you block them, your content can’t be quoted or linked in those answers, which is an increasingly important source of visibility. Learn more in our guide to generative engine optimization.
AI training crawlers: your choice
GPTBot, ClaudeBot, Google-Extended, CCBot (Common Crawl), Applebot-Extended, Bytespider and meta-externalagent collect content to train or improve AI models. Blocking them doesn’t affect your Google or Bing rankings. It’s a business decision about whether you want your content used for training.
A note on Google-Extended
Google-Extended isn’t a separate crawler. It’s a token that tells Google whether content Googlebot has already fetched may be used to improve Gemini and Vertex AI. Blocking it doesn’t affect Google Search, including whether your pages are crawled or ranked.
| Crawler | Company | Purpose | Typical choice |
|---|---|---|---|
| OAI-SearchBot, ChatGPT-User | OpenAI | ChatGPT search and browsing | Allow |
| Claude-SearchBot | Anthropic | Claude search | Allow |
| PerplexityBot | Perplexity | AI search answers | Allow |
| GPTBot | OpenAI | Model training | Your choice |
| ClaudeBot | Anthropic | Model training | Your choice |
| Google-Extended | Gemini training control | Your choice | |
| CCBot | Common Crawl | Open web dataset | Your choice |
Common robots.txt generator mistakes to avoid
Leaving “Block everything” on a live site
Disallow: / under User-agent: * blocks every compliant crawler. Use it only on staging sites, and better still, protect staging sites with a password.
Entering full URLs instead of paths
Rules match the part of the URL after the domain. Write /private/, not https://example.com/private/. Our generator offers to convert full URLs to paths automatically.
Blocking pages you want removed from Google
If a page is already indexed, blocking it stops Google from seeing a noindex tag, so it may stay in results. Use noindex first, wait until it drops out, then block it if you still want to. Check the status with our Google index checker.
Using Crawl-delay for Google
Googlebot ignores Crawl-delay. Bing respects it, so only set it if crawlers are genuinely overloading your server.
Relative sitemap URLs
The Sitemap line must contain a full URL. Our generator flags anything that isn’t.
* group. Repeat any shared rules in its own group.How to add robots.txt to your website
WordPress with Yoast SEO
Go to Yoast SEO → Tools → File editor, click Create robots.txt file if prompted, paste the generated rules and save.
WordPress with Rank Math
Go to Rank Math → General Settings → Edit robots.txt, paste the rules and save. If a physical robots.txt file exists in your site root, delete it first so the editor’s version is used.
Uploading the file manually
Download the file from the generator, then upload it to your site’s root folder (usually public_html) with your hosting file manager or SFTP. It must be reachable at https://yourdomain.com/robots.txt.
Shopify, Wix and Squarespace
These platforms generate robots.txt automatically. Shopify allows customisation through a robots.txt.liquid theme template; Wix offers a robots.txt editor in its SEO settings; Squarespace doesn’t allow full edits but provides an option to block AI crawlers.
Test after uploading
- Open
/robots.txtin a browser to confirm it’s live. - Fetch it in our robots.txt checker and test key URLs.
- In Google Search Console, check Settings → robots.txt for fetch status and errors.
Why use a robots.txt generator for WordPress specifically?
WordPress has quirks that generic generators don’t account for. A robots.txt generator for WordPress knows them, so you don’t have to.
The virtual robots.txt
WordPress creates a virtual robots.txt on the fly when no physical file exists. SEO plugins edit that virtual file. If you upload a physical file as well, it silently takes over and your plugin settings stop applying. Decide on one method and stick with it.
The “Discourage search engines” setting
Under Settings → Reading, WordPress has a checkbox that asks search engines not to index the site. On current versions it adds a noindex robots meta tag rather than changing robots.txt, but it’s still the most common reason a WordPress site doesn’t appear in Google. Untick it on live sites, even if your robots.txt is perfect.
Plugin and theme assets
WordPress loads CSS and JavaScript from /wp-content/plugins/, /wp-content/themes/ and /wp-includes/. A generic generator might suggest blocking these “system” folders; a WordPress-aware one keeps them open so pages render correctly for Googlebot.
Multisite and subdomains
Each host needs its own robots.txt. On a WordPress multisite network that uses subdomains, every subsite is served its own virtual file. If you use subdirectories, one file covers the whole network, so write rules carefully to avoid blocking a subsite by accident.
Our recommendation
Generate your rules here, paste them into your SEO plugin’s robots.txt editor, and re-test with the robots.txt checker whenever you change plugins or themes. For background on what robots.txt is for, Google’s robots.txt introduction is the best primer.
Robots.txt, sitemaps and indexing work together
robots.txt decides what crawlers may fetch, your XML sitemap lists what you want them to find, and meta robots tags decide what may be indexed. Keep them consistent: don’t list blocked URLs in your sitemap, and don’t block pages that carry important canonical or noindex tags. Validate your sitemap with our sitemap validator after changing robots.txt.
Example: a complete file for a WordPress business site
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Sitemap: https://example.com/sitemap_index.xmlThis file lets search engines and AI search tools crawl everything public, opts out of AI training, keeps the admin area and search results out of crawls, and points every crawler to the sitemap.
After any change, test the new file with our robots.txt tester against your most important URLs, then watch the Page indexing report in Search Console for a few weeks. A sudden rise in “Blocked by robots.txt” is the earliest warning that a rule is broader than you intended.
Robots.txt checklist
- Default set to allow on live sites.
/wp-admin/blocked withadmin-ajax.phpallowed (WordPress).- No rules blocking CSS, JavaScript, images or
/wp-content/. - AI search crawlers allowed; training crawlers set according to your policy.
- Paths start with
/; no full URLs in rules. - A full-URL Sitemap line.
- File live at the domain root and tested in a robots.txt checker.
Related free SEO tools
- Robots.txt checker: test any URL against your live or draft rules.
- Sitemap validator: check the sitemap you list in robots.txt.
- Google index checker: find out why a page isn’t indexed.
- Meta tag generator online: create noindex and other robots meta tags.
- Free SEO checker: check AI crawler access for any page.
Ready to build yours? Open the robots.txt generator above and download a safe file in under a minute.