Robots.txt Generator

Assemble a robots.txt from a preset and your own crawl rules, then copy it straight into your site root.

Runs on your deviceWeb & SEOWorkingNo account needed

What this tool does

Robots.txt Generator builds the file line by line from two things: a preset you choose, and a list of paths you type. The presets carry rules that are tedious to remember — the WordPress one disallows /wp-admin/, /wp-login.php, /xmlrpc.php and the internal search and trackback paths while explicitly allowing /wp-admin/admin-ajax.php and /wp-content/uploads/, and the e-commerce one covers /cart, /checkout, /account and the ?sort= and ?filter= query patterns that turn a faceted catalogue into millions of crawlable permutations. Whatever you add is merged with the preset, duplicates are dropped, and the result is one User-agent group with the Allow lines above the Disallow lines. When nothing is disallowed it writes a bare "Disallow:", which is the correct way to say that everything is permitted.

Four settings sit around that group. The user-agent field takes a single token — * by default, or a name such as Googlebot — and is rejected if it contains a space. Sitemap URLs have to be absolute, and several can be given at once separated by spaces or commas. The crawl-delay field runs from 0 to 3600 seconds and the line is dropped entirely at 0; it is worth knowing that Google has never obeyed that directive and that Bing is now the main crawler that still reads it. The two checkboxes append fully blocked groups: one covers twenty named AI training and answer crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and Bytespider, and the other covers ten backlink crawlers such as AhrefsBot, SemrushBot and MJ12bot. Those blocks only work on crawlers that send the name they are matched against and choose to honour the file.

Everything happens in this browser tab, so nothing is uploaded and the tool works with the network disconnected. The limits are worth stating flatly. robots.txt is a crawling instruction, not access control: a disallowed URL can still be listed in search results as a bare link, and anyone who knows the address can still open it — which is exactly why the file is a poor way to hide a staging site. The generator writes one set of rules for the user-agent you name, plus the all-or-nothing blocked groups, so genuinely different path rules for two different crawlers mean running it twice and joining the groups by hand. And it never fetches your site, so it cannot tell you whether a path you typed exists, or warn you that you have just blocked the CSS and JavaScript Google needs to render your pages. It also does not spot a collision between its own two halves: name GPTBot as the main user-agent while the AI checkbox is ticked and the file ends up with two GPTBot groups, one carrying your paths and one saying Disallow: /, with nothing on screen pointing that out.

How to use it

  1. Choose a starting point. Custom, Allow everything, Block everything (staging), WordPress, E-commerce, or Docs / marketing site. The preset's paths are loaded first and anything you type is added to them.
  2. List your own paths. One per line. Each must begin with / or *, and a leading ! turns that line into an Allow instead of a Disallow. Lines starting with # are ignored, and a pasted "Disallow: /x" or "Allow: /x" prefix is stripped for you.
  3. Set the user-agent and sitemap. Leave the user-agent as * to address every crawler, or name one. Sitemap URLs must be absolute, such as https://example.com/sitemap.xml; list several separated by spaces or commas.
  4. Decide on crawl-delay and the block lists. Leave crawl-delay at 0 to omit the line. Tick either checkbox to append a Disallow: / group for every AI crawler or every backlink crawler on the list.
  5. Read the warnings, then copy. A Before you publish list flags the risky combinations — the staging block still in place, a missing Sitemap line, or a named user-agent that leaves every other crawler ungoverned. Copy output puts the plain text on the clipboard.

Processing & privacy

Runs on your device. Your file is processed directly in this browser.

Common use cases

Stopping a shop from burning crawl budget on filters

The e-commerce preset disallows the cart, checkout and account paths together with the sort, filter and add-to-cart query strings, so crawlers spend their time on product pages instead of colour-and-size combinations.

Tidying a WordPress install

The preset blocks the admin and login paths without breaking anything, because it keeps admin-ajax.php and the uploads directory allowed — the two exceptions people forget and then wonder why images stopped appearing in image search.

Opting out of AI training crawlers

One checkbox writes the twenty groups you would otherwise look up individually, covering the training and answer bots that publish a user-agent name and read the file.

Holding a staging site back from search

The Block everything preset produces the Disallow: / group, and the tool reminds you to remove it before launch. Pair it with HTTP authentication, since the file asks crawlers to stay away rather than keeping anyone out.

Pointing crawlers at your sitemaps

Add one or several absolute sitemap URLs and they are appended after the rules, which is how a crawler finds new pages without waiting to stumble across a link.

Questions

Does any of this leave the tab?
No. The file is assembled in this browser tab from the preset and the text you type, so there is nothing to transmit and the tool still works offline.
Will a Disallow line remove a page from Google?
No, and this is the most common misunderstanding about the format. Disallow stops the page being crawled, not indexed — the URL can still appear as a bare result with no description, gathered from links pointing at it. Removal needs a noindex tag or an X-Robots-Tag header, and Google can only see either one if the URL is crawlable, so blocking it here actively prevents the removal you wanted.
Does blocking AI crawlers actually stop them?
It stops the ones that send the user-agent name on the list and choose to obey the file, which covers the operators who publish a bot name and documentation. It has no effect on a scraper that lies about its identity or ignores robots.txt, and nothing in the file reaches a model that was trained before you published it.
Why is crawl-delay described as ignored?
Google's crawler has never supported the directive and simply skips it, and Bing is the one major crawler that still reads it — Yandex retired its support in 2018 in favour of a crawl-rate control in its own webmaster console. If Googlebot is requesting pages faster than your server likes, the crawl rate setting in Search Console is the lever, not this line.
Can I give Googlebot different rules from everyone else?
Not in a single pass. The tool writes one rule group for the user-agent you name, plus the fully blocked groups from the checkboxes. For two groups with different paths, generate each one and paste them into the same file — and note the warning it shows when you name a specific agent, because a file with only a Googlebot group leaves every other crawler unrestricted.
Where does the file go, and does it cover subdomains?
It has to sit at the root of the host and answer at https://your-domain/robots.txt with that exact filename. Rules apply only to the host and protocol serving them, so a subdomain such as shop.example.com needs its own file.
Does the generator check my site?
No. It never makes a request, so it cannot confirm that a path exists, that your sitemap URL resolves, or that a rule blocks something important. Run the result through Robots.txt Validator to see how a specific URL is judged against the rules.

Related tools