Robots.txt Generator

Create crawler rulesRuns locally · nothing uploaded

Default rule (all crawlers)Disallowed paths (one per line)Allowed exceptions (one per line)Crawl-delay (seconds, optional)Sitemap URLs (one per line)

Rules for specific crawlers

CrawlerRule

Block common AI training crawlers (GPTBot, CCBot…)

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /*?sessionid=

Sitemap: https://www.example.com/sitemap.xml

Also works as:robots txt generatorblock crawlersblock gptbot

Where does robots.txt go?

In the site root, e.g. https://www.example.com/robots.txt.

Does Disallow stop indexing?

It only stops crawling; linked pages can still appear in results. Use noindex to keep pages out.