Skip to content

Robots.txt Generator

Builds a robots.txt for your CMS and checks it with the same parser, so you see not only the text but which URLs it blocks. Every line has an explanation.

Website engine

We will fill in the service paths of this engine: the admin panel, cache, and system folders. The sets do not block the site's content — the checkboxes below handle that.

Sitemap address

Type the address in full — the one that actually opens. It cannot be built from the domain: one site keeps its sitemap at /sitemap.xml, another at /sitemap_index.xml. If you have no sitemap, leave the field empty — a made-up address is worse than a blank.

What to block

Block site search Search result pages are endless near-identical URLs. Every site has its own path: we fill it in from the engine preset, and with no engine you enter it yourself. You can list several paths, comma-separated.

List several paths comma-separated or one per line.

Block URLs with ad tags The same page with a tag and without it are copies. The tags are the same on every site, so there is nothing to guess.

Block the cart and checkout Stores need this: these pages are different for every visitor. There are usually two URLs here — the cart and checkout — and you can list both in the field next to it, comma-separated.

List several paths comma-separated or one per line.

Block the account area, login and registration Personal pages are of no use in search. There are usually several URLs here: the account itself, login, registration, password recovery. List them in the field next to it, comma-separated.

List several paths comma-separated or one per line.

Block print versions A print version is a copy of the regular page. We block it by a URL parameter, which is nearly always the same.

Block paginated navigation A debatable setting. Blocking navigation, you cut the robot's path to products and articles on far pages. Check it only if you know exactly why.

Block SEO tool bots We add a separate block for each: AhrefsBot, MJ12bot, ia_archiver, SemrushBot and DotBot. Sites usually do this so competitors cannot see their links and the bots do not load the server. Remember: robots.txt does not forbid, it asks.

Block AI crawlers We add a separate block for each: GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, Google-Extended, Applebot-Extended and others — 17 names in total. Your site will not drop out of regular search: Googlebot and Applebot crawl on their own and never touch these lines.

Allow images, styles and scripts Google's help pages ask you not to block styles and scripts: without them it sees the page differently from a visitor. These lines override any stray blocks above.

Add your own rules Collapse your own rules

Sections and pages to block from indexing

One rule per line. Write it in full (Disallow: /folder/) or just the path: we will add what's missing — the leading slash and the Disallow word.

Sections and pages to keep open for indexing

Needed when a blocked folder contains something that must reach the search results. Here a bare path becomes Allow. When rules overlap, the longer one wins, so the exception will work.

Fine-tuning Collapse fine-tuning

Clean-param

Only Yandex understands it: it will glue together addresses that differ only by these parameters. List them separated by & without spaces. The line will go into a separate User-agent: Yandex block.

Crawl-delay, seconds

Google does not support this directive, as its own help pages say, and Yandex asks you to set the crawl speed in Yandex Webmaster. Set it only if your server truly can't keep up.

Block the entire site from crawling For a staging environment. On a live site this will drop pages from search within days.

Check the result on your own URLs Collapse the URL check

URLs or paths — one per line

We will run the generated file through our own parser and show which URLs got blocked and which stayed open.

A generator that checks itself

The file you build is parsed right away by the same engine that runs our robots.txt validator. You see not just the text but what it does: how many rules it contains, whether there are syntax errors, and which addresses end up blocked. Enter your own URLs and confirm that the pages you need stay open.

Every line explained

Next to the file sits a breakdown: which folder is closed and why. A file whose origin you cannot explain is a risky thing to put on a live site. One extra line can drop an entire section from search results.

Presets for CMS engines

For WordPress, 1C-Bitrix, Joomla, OpenCart, DataLife Engine, MODX, Drupal, PrestaShop, Magento, Webasyst, uCoz and Tilda the generator adds the standard service sections: the admin panel, cache, system folders, and service URLs with parameters. The presets leave your content alone. Whether to close search results, the cart, or tag pages is your call.

What we do not add

The Host directive is deprecated by Yandex, and the main mirror has long been set with redirects and the canonical tag, so the generator has no field for it. Crawl-delay is not supported by Google, and Yandex asks for crawl speed to be set in its webmaster tools. The field is still there, but it stays empty, with a warning next to it.

What to do with the finished file

Download it and place it in the root of your site, where it must open at an address like example.com/robots.txt. After uploading, check the file again by its URL. That way you confirm the server returns the file itself and not an error page.

Rate this tool

Help us get better

Other tools

Link Analyzer

Internal link analysis and a check of outgoing links on your site's pages.

Anchor Text Generator

A bracketed template expands into a set of phrases without repeats.

Sitemap Generator

A site map by crawling pages or from a ready list of addresses.

HTML Link Generator

Ready-made link code: a regular link, an email, phone or anchor.

Open Graph Checker

How your link looks when shared to social networks and messengers

Meta Tag Checker

Title, description and robots: what's in the code and what survives in results

Robots.txt Validator

Robots.txt errors and a check of whether a URL is blocked

UTM Link Builder

Tracking labels with platform macros, the erid token and link breakdown