Your Robots.txt Generator
Generate a robots.txt file to control how search engines crawl your site. Allow or disallow paths, add your sitemap, set crawl-delay and block AI crawlers.
- Free to use
- No signup
- Nothing to install
- Runs in your browser
User-agent: * Disallow: /admin/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xmlOpen the Robots.txt Generator
Illustration. Open the tool for the real thing.
See it in action
Sample input: Crawl-delay (seconds, optional) = 1000; Disallow these paths (one per line) = /admin/ /cart/ /checkout/ /search; Allow these paths (one per line, optional) = The bakery on Main Street opens at ...; Sitemap URL(s) (one per line) = https://example.com/sitemap.xml
What is the Robots.txt Generator?
A robots.txt file sits at the root of your website and tells crawlers which pages or folders they may or may not fetch. A well-made file keeps crawlers away from admin pages, search results and duplicate content, saves crawl budget for the pages that matter and points bots to your XML sitemap.
Pick which crawlers the rules apply to, list the paths to allow or disallow, add your sitemap URL and an optional crawl delay, and choose whether to block popular AI training crawlers. The generator writes a valid robots.txt you can copy or download and upload to your site’s root. Remember that robots.txt is a request to well-behaved bots, not a security measure.
3 steps to your result
Open the tool and follow along. Nothing to install.
What you get
Allow and disallow paths for all or specific crawlers.
Adds your sitemap URL correctly.
Optionally disallow GPTBot, CCBot and other AI crawlers.
Ready to upload as robots.txt.
AZRS tools are free to open and use.
It opens in your browser, so there is nothing to download.
Used for
Pick one to see how it fits.
How it compares
A plain look at this tool next to the usual alternative.
| This seo tool | Desktop software | |
|---|---|---|
| Install | Nothing to install | Usually an app or program to install |
| Account | No signup | Depends on the product |
| Cost | Free to use | Free or paid, depends on the product |
| Where it runs | In your browser tab | On your computer |
Questions people ask
What is robots.txt?
A text file that tells search engine crawlers which parts of a site they may crawl.
Where do I put it?
In the root directory of your domain.
Does robots.txt stop pages appearing in Google?
Not always. Blocked pages can still be indexed if linked elsewhere; use noindex to remove them.
Should I block AI crawlers?
It is your choice; many sites now disallow AI training bots.
What does Disallow: / do?
It blocks the entire site for the chosen crawler.
More free tools
Related to the Robots.txt Generator.