Generator robots.txt

A free robots.txt generator that runs in your browser

This robots.txt generator turns a simple form into a valid robots.txt file — no syntax to memorize and nothing to install. Choose a preset or build your own crawl rules, watch the file assemble live, then copy or download it and drop it at the root of your site. Everything happens locally in your browser, so the rules you write are never uploaded to a server.

Whether you need a quick robots.txt example to start from, a custom robots.txt for a specific crawler, or a file tuned for WordPress, Blogger, or blocking AI training bots, you can have it ready in seconds. The built-in path tester validates your rules before you publish — something most competing generators lack.

Build crawl rules without memorizing the syntax

Each rule group pairs a User-agent with the paths you want to Allow or Disallow, plus an optional Crawl-delay. Use * to target every crawler, or name a specific bot like Googlebot or Bingbot — start typing and the field suggests common crawlers for you. Add as many groups as you need to give different bots different rules.

List one path per line; the tool formats each as a proper Disallow: or Allow: directive and stitches your groups together in the correct order. Add your Sitemap URL and it's appended as an absolute Sitemap: line that points crawlers at every page you want found. Google's crawler picks this up and uses it alongside any sitemap submitted directly in Search Console.

Presets for WordPress, Blogger, and blocking AI bots

One click loads a ready-made starting point. Allow all and Block all cover the two extremes. WordPress loads the recommended default: block /wp-admin/ but keep /wp-admin/admin-ajax.php open for the AJAX calls plugins need. Blogger loads Google's suggested Blogger setup: allow everything except /search to avoid duplicate-content issues.

The Block AI bots preset is useful if you want your content to stay on Google and Bing without being used for AI model training. It disallows GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended (Gemini), ClaudeBot (Anthropic), PerplexityBot, and others — while leaving standard search crawlers untouched. The New York Times, The Guardian, and many major publishers added similar blocks in 2023.

Test a URL before you publish

A generator that can't test itself only does half the job. The built-in path tester lets you enter any URL or path and a crawler name, then tells you whether your current rules allow or block it — and exactly which Allow or Disallow rule decides, using the same longest-match logic Google applies (the most specific rule wins, and Allow breaks a tie of equal length).

As you type, the tool also surfaces live warnings for mistakes that quietly break real robots.txt files: a path missing its leading slash, a relative sitemap URL, or a Crawl-delay aimed at Googlebot (which ignores it and requires a separate setting in Search Console). Fixing these before you ship saves a debugging round-trip through Google Search Console.

Crawling vs indexing — and other common mistakes

The biggest robots.txt misconception: Disallow removes a page from Google. It doesn't. It stops crawling, but a blocked URL can still appear in search results if other pages link to it. Google may even index it with no snippet — just a URL and no description. To keep a page out of search, leave it crawlable and add noindex on the page itself. If you block it in robots.txt, Google can never crawl the page to see that noindex, and the URL can stay in the index indefinitely.

Remember too that robots.txt is public and advisory. It lives at a fixed, readable URL, it isn't a security barrier, and only well-behaved bots obey it. Use it to guide crawl budget and tidy up duplicate or thin pages — not to hide anything that truly needs protecting. For that, use authentication. This generator runs entirely in your browser, so your site's internal paths never leave your device.

The robots.txt protocol: from a 1994 mailing list to RFC 9309

The Robots Exclusion Standard was not born in a standards body — it came from a single email. In June 1994, Martijn Koster posted a proposal to the www-talk mailing list describing a convention for web robots to check a site's wishes before crawling. The idea spread quickly because it solved a real problem: early web crawlers were hammering servers without any mechanism for site owners to opt out. Within months, the major search engines of the era had adopted the convention, and the simple robots.txt file has sat at the root of the web ever since.

Despite its ubiquity, the Robots Exclusion Protocol remained a de facto convention for nearly three decades — it was never ratified as a formal W3C or IETF standard. That changed in 2019 when Google, with support from other search engines, submitted a proposal to the IETF to codify the protocol. The result, RFC 9309, was published in September 2022. The RFC formalizes the core directives (User-agent, Allow, Disallow), defines the longest-match rule for conflicting directives, and specifies how crawlers must handle HTTP status codes when fetching the file. It deliberately does not standardize Crawl-delay or Sitemap — those remain widely supported extensions outside the formal spec.

Understanding this history matters in practice. Because the standard evolved from informal consensus, different crawlers have historically interpreted edge cases differently — trailing wildcards, case sensitivity of paths, or the behaviour when a server returns a 5xx error. RFC 9309 resolves most of those ambiguities, but sites targeting older or less common crawlers may still encounter divergent behaviour. The safe approach: stick to the directives documented in the RFC and test your rules against the specific bots you care about.

Directive syntax in depth: User-agent, Allow, Disallow, Crawl-delay, Sitemap

Every robots.txt file is built from a small set of directives. User-agent: opens a rule group and names the crawler it applies to. The value * matches every bot; a specific name like Googlebot, Bingbot, GPTBot, anthropic-ai, or CCBot targets only that crawler. A single crawler obeys the most specific group that matches its name — a group for Googlebot takes priority over the * group for Google's web crawler. Disallow: tells the crawler not to fetch matching paths. An empty value (Disallow:) means allow everything; a single slash (Disallow: /) blocks the entire site. Allow: is not part of the original 1994 spec but is supported by Google, Bing, and most major crawlers; it carves out an exception inside a broader disallow, and when a path matches both, the longest (most specific) rule wins — with Allow breaking ties of equal length.

Crawl-delay: asks a crawler to pause a given number of seconds between requests — useful when aggressive crawling strains your server. Bing and Yandex honour it; Googlebot ignores it entirely. To throttle Google's crawl rate, use the Crawl Stats report in Google Search Console instead. The Sitemap: directive accepts a full absolute URL and tells crawlers where to find your XML sitemap. It can appear anywhere in the file, it is not tied to any User-agent group, and you can include multiple Sitemap: lines for separate sitemap index files. Google, Bing, and Yandex all process it; it is a practical complement to direct sitemap submission in Search Console because it makes the location discoverable by any crawler that reads your robots.txt.

Path matching follows prefix logic by default: Disallow: /private/ blocks /private/, /private/page, and /private/docs/file.pdf. The * wildcard inside a path matches any sequence of characters, and the $ anchor at the end pins the match to the exact end of the URL — for instance, Disallow: /*.pdf$ blocks URLs ending in .pdf. Google supports both wildcards and the dollar anchor; not every crawler does, so rely on the anchor only when targeting Google or bots with documented support for it.

What robots.txt does not do — the limits you must know

robots.txt does not prevent indexing. This is the single most misunderstood aspect of the protocol. If you add Disallow: /secret-page/, Google will not crawl that URL — but if another site links to it, Google can still discover the URL and include it in search results, typically with a truncated or absent snippet. The only reliable way to exclude a page from the index is a noindex directive on the page itself (via a <meta name="robots" content="noindex"> tag or an X-Robots-Tag HTTP header) and critically, the page must remain crawlable so Google can read that directive. Blocking a URL in robots.txt and expecting it to stay out of the index is a well-documented SEO trap.

robots.txt does not provide security. The file is publicly readable at yoursite.com/robots.txt, so listing paths you want to hide under Disallow effectively advertises those paths to anyone who opens the file — including attackers looking for admin panels, staging environments, or internal APIs. Listing Disallow: /admin-login-backup/ is an invitation, not a lock. Sensitive resources must be protected with real access controls: server-side authentication, IP allowlisting, or firewall rules. Robots.txt is an instruction to cooperative crawlers, not a mechanism that enforces anything.

robots.txt is voluntary. Ethical, well-behaved crawlers — Googlebot, Bingbot, Apple's Applebot, and legitimate research bots — obey the protocol because they have reputations and business relationships to protect. Malicious scrapers, content thieves, and vulnerability scanners routinely ignore it. Even among compliant crawlers, the protocol has edge-case gaps: Google has documented that it can index a URL that is disallowed in robots.txt if that same URL appears in a sitemap you submitted — the sitemap acts as a competing signal. The practical takeaway: use robots.txt to manage crawl budget and guide well-behaved bots, not to enforce any real restriction.

AI crawlers and the evolving robots.txt landscape

The past two years have introduced a new class of robots.txt consideration: AI training crawlers. Generative AI companies scrape the open web to build training datasets for large language models. The main crawlers in this space are GPTBot (OpenAI, used for GPT-4 and later models), Google-Extended (Google, used for Bard and Gemini training — separate from Googlebot), anthropic-ai (Anthropic, used for Claude), CCBot (Common Crawl, a non-profit whose dataset is widely used by academic and commercial AI research alike), PerplexityBot, Applebot-Extended, and meta-externalagent (Meta, used for Llama). Each exposes its own User-agent string specifically so site owners can opt out via robots.txt.

Adoption of AI crawler blocks has grown rapidly. Studies tracking a sample of top-ranked domains found that within a year of OpenAI publishing GPTBot's user-agent string (August 2023), a substantial share of major publishers had blocked it — including the New York Times, Reuters, and a large portion of the news industry. One analysis put the figure at over 25% of the top 1,000 sites by late 2024. Publishers cite copyright concerns, the absence of compensation, and the potential to train competitors' products as their reasons. Whether blocking AI crawlers is the right call depends on your priorities: it removes your content from future training runs but has no direct effect on your search rankings.

A question that often arises: does blocking Google-Extended hurt your standing with Google Search? Google has been explicit that Google-Extended and Googlebot are distinct crawlers with separate controls. Disallowing Google-Extended in robots.txt tells Google not to use your content to train Gemini, but it has no effect on Googlebot's crawling or your organic search indexation. You can block all the AI training crawlers listed above while leaving Googlebot and Bingbot fully enabled, and your search visibility is unaffected. The Block AI bots preset in this generator does exactly that.