Robots.txt Generator
Generate crawler directives from disallowed paths and an optional sitemap URL.
Ready. Processing happens on your device.
Maximum 2 MB per text file. CSV limits: 20,000 rows and 200 columns. Drop one supported text file onto the input.
How to use Robots.txt Generator
- Type or paste your input, open a supported local file, or try the example.
- Choose the available options and select Run tool.
- Review the output, then copy it or download it.
What to expect
Robots.txt controls cooperative crawlers. It does not protect private content or reliably remove indexed pages.
Common questions
Does Robots.txt Generator send my input to a server?
No. Processing runs in a browser worker. Files are read locally and downloads are created on your device.
What if the result is empty or an error appears?
Check the input format and options. A filter or extraction may legitimately return no matches. Invalid formats show an explanation, and expensive operations stop after five seconds so you can revise the input.