robots.txt Generator — Build a Crawler Rules File Online
Generate a valid robots.txt file with User-agent, Allow, Disallow, Crawl-delay and Sitemap directives — runs 100% in your browser, no upload, no sign-up.
User-agent: * Disallow: /admin Disallow: /private
🔒 Generated in your browser — nothing is uploaded.
What this robots.txt generator does
A robots.txt file lives at the root of your domain and tells search engine crawlers which parts of your site they may or may not fetch. This generator builds that file for you from plain inputs: set the User-agent the rules apply to (use * for all bots or a name like Googlebot), list the paths to Allow and Disallow one per line, optionally add a Crawl-delay in seconds, and point crawlers to your Sitemap URL. The output updates live as you type and one click copies the whole file, ready to save as robots.txt in your site root.
How the file is assembled
The generator emits a User-agent: line first, then every Allow: path, then every Disallow: path, keeping a predictable order so your file stays readable and diff-friendly. Blank and whitespace-only lines are ignored, so a stray empty row never becomes an invalid directive. A Crawl-delay: line is added only when you enter a positive number, and a trailing Sitemap: line is appended after a blank separator when you supply a sitemap URL. The result follows the widely supported robots exclusion format that Google, Bing and other crawlers understand.
Frequently asked questions
Where do I put the generated robots.txt file?
It must sit at the root of your domain, reachable at https://yourdomain.com/robots.txt. Crawlers only look there — a robots.txt inside a subfolder is ignored. Copy the output, save it as a plain text file named robots.txt, and upload it to your site root.
Does Disallow actually hide a page from Google?
No. Disallow asks well-behaved crawlers not to fetch a path, but it does not guarantee secrecy and a blocked URL can still appear in results if other pages link to it. To keep a page out of search results use a noindex meta tag or password protection instead.
Is anything I type uploaded to a server?
No. The entire robots.txt file is assembled locally in your browser with JavaScript. Your user-agent, paths and sitemap URL are never sent, stored or uploaded anywhere.