Robots.txt Generator

1. Settings

Rank Math uses /sitemap_index.xml, Yoast /sitemap_index.xml, core WordPress /wp-sitemap.xml.
Start each path with /. Use * as a wildcard and $ for “ends with”.
Blocking these does not affect your Google or Bing search rankings.

2. Your robots.txt

Upload the file to the root of your domain so it opens at https://yourdomain.com/robots.txt. On WordPress you can also paste it into Rank Math → General Settings → Edit robots.txt.

Complete Guide to Robots.txt: Control Search Crawlers & Protect Server Bandwidth

A robots.txt file is a plain text file placed in the root directory of your web server that instructs web search crawlers (such as Googlebot, Bingbot, and modern AI scrapers) which pages and directories they are permitted or forbidden to crawl.

Using the Toolfix Robots.txt Generator, you can create clean, syntax-compliant rules in seconds without risking accidental de-indexing of your key pages.

Core Directives Explained

  • User-agent: * — Applies instructions to all automated search engine crawlers and bots.
  • Disallow: /wp-admin/ — Tells crawlers not to request administrative or private backend directories.
  • Allow: /wp-admin/admin-ajax.php — Specifically permits crawlers to access dynamic scripts necessary to render public pages properly.
  • Sitemap: https://yourdomain.com/sitemap.xml — Directly provides crawlers with the location of your XML sitemap for efficient indexing.

Blocking Aggressive AI Web Scrapers in 2026

AI scrapers can consume excessive server bandwidth. You can specifically block AI training bots while maintaining search rankings for Google and Bing:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Frequently Asked Questions

Where should I upload my robots.txt file?
The robots.txt file must always be uploaded to the root directory of your website (e.g. https://yourdomain.com/robots.txt). Search engine bots will not recognize robots.txt files placed inside subfolders.
Does robots.txt prevent pages from appearing in Google search?
No. Disallowing a page in robots.txt stops Google from crawling its content, but if other sites link to it, the URL can still be indexed without a snippet. To completely remove a page from search results, use a noindex meta tag instead.
How do I test if my robots.txt syntax is valid?
Upload the file and open https://yourdomain.com/robots.txt in a browser to check that it loads. Then use the robots.txt report in Google Search Console (Settings → robots.txt), which shows the version Google fetched and any parsing errors. To test whether a specific URL is blocked, use the URL Inspection tool.