Robots.txt Generator

robots.txt is a small text file at the root of your site that tells crawlers which paths they may fetch. A single wrong line can hide your whole site from Google; a missing file means crawlers waste time on search pages, carts and admin areas.

This generator builds a clean robots.txt from simple choices: allow everything or block everything (for staging sites), the paths to keep crawlers out of, your sitemap URL, an optional crawl-delay for Bing and Yandex, and an optional block list for well-known AI training crawlers such as GPTBot, CCBot, Google-Extended and ClaudeBot.

It validates paths as you go — every rule has to start with “/” — and explains the one misunderstanding that causes the most damage: robots.txt controls crawling, not indexing.

The file it produces is intentionally short and readable. Long robots.txt files copied from forums often contain contradictory rules, outdated directives and blocks on CSS or JavaScript that stop Google from rendering pages properly. Starting from a minimal file and adding only what you need avoids those problems.

How It Works

1. Choose the default — allow all crawlers (normal public site) or disallow all (staging, development or private sites).
2. Add rules — paths to block, your sitemap URL, optional crawl-delay and optional AI-crawler blocking. Invalid paths are flagged before anything is generated.
3. Download — save the file as robots.txt and upload it to the root of your domain so it is reachable at https://yoursite.com/robots.txt.

Compatibility & Support

Supported Input → Output Formats

• Input: default rule, disallowed paths, sitemap URL, crawl-delay, AI crawler preference
• Output: robots.txt following the Robots Exclusion Protocol (RFC 9309)
• Crawlers covered: all (*), plus GPTBot, ChatGPT-User, CCBot, Google-Extended, ClaudeBot, anthropic-ai, PerplexityBot, Bytespider, Applebot-Extended when blocking AI

Unsupported Formats

robots.txt is a request, not security: well-behaved crawlers follow it, but it doesn't hide or protect pages, and disallowed URLs can still appear in Google if other sites link to them. To keep a page out of search results use a noindex meta tag (see the Meta Tag Generator) and don't block that page in robots.txt, or Google will never see the tag. Google ignores Crawl-delay.

File Size Limits

Google reads the first 500 KB of a robots.txt file, far more than any normal site needs. There is no limit on the number of paths you can list here.

Changes take effect when crawlers next fetch the file — Google usually re-reads robots.txt within a day. If you have just unblocked a site after launch, it can take a few days for pages to be crawled and indexed again.

Who Should Use This Tool?

• Students: A student launching a first website adds a correct robots.txt with the sitemap so Google discovers every page.
• Freelancers & Designers: A web developer blocks the whole staging site with “Disallow: /” and remembers to switch it to allow-all at launch.
• Office Professionals: A marketing manager blocks internal search-result pages and cart URLs that waste crawl budget on an e-commerce site.
• Developers: A developer adds an AI-crawler block for a documentation site whose owners don't want their content used for model training.

Why This Tool Beats the Alternatives

• Explains crawling versus indexing so you don't accidentally de-index pages.
• Current AI crawler user-agents included.
• Validates paths before you publish.
• No account or email needed.
• Free and instant.

Pro Tips

• Never block CSS or JavaScript folders — Google needs them to render and understand your pages.
• After uploading, test the file in Google Search Console's robots.txt report, which shows the version Google last fetched and flags any lines it could not parse.
• When a staging site goes live, remove “Disallow: /” immediately; forgetting it is one of the most common causes of a site vanishing from Google.

Choose your rules above and download a ready-to-upload robots.txt file.