robots.txt / sitemap.xml Generator

About this tool

Generate a robots.txt file that tells search engine crawlers which parts of your site to skip, with an optional link to your sitemap — or generate a sitemap.xml from a list of page URLs to help search engines discover and index your pages.

Everything is built locally in your browser — nothing is uploaded to a server.

Frequently asked questions

Where do robots.txt and sitemap.xml need to go?

Both must be placed at the root of your domain — e.g. https://example.com/robots.txt and https://example.com/sitemap.xml — for search engines to find them automatically.

Does robots.txt actually block anyone from visiting a page?

No — it's a voluntary convention that well-behaved crawlers respect. It doesn't restrict access for people or for crawlers that choose to ignore it, so it isn't a substitute for real access control on sensitive content.

What does "Block common AI crawlers" do?

It adds disallow-all rules for a handful of widely known AI training/data-collection bots (like GPTBot and CCBot), in addition to whatever custom paths you've listed.