SEO Tools
Robots.txt Generator
Build a robots.txt file by selecting which crawlers to block, specifying allow and disallow paths, and adding your sitemap URL. Download the file or copy the content.
User-agent: *
How it works
Select bots to block individually or check "Block All Crawlers" to deny all access. Add allowed and disallowed paths to fine-tune which sections of your site crawlers can visit. Include your sitemap URL so search engines can find it automatically. Copy the result or download it as robots.txt.
Frequently asked questions
What is a robots.txt file?
A robots.txt file tells web crawlers which pages or sections of your site they can and cannot access. It is placed in the root directory of your website.
Does robots.txt block all crawlers?
No. robots.txt is a convention that well-behaved crawlers follow. Malicious bots often ignore it. It cannot replace server-side access controls for truly private content.
What is the difference between Allow and Disallow?
Disallow prevents crawlers from accessing a path. Allow explicitly permits access to a path, useful to override a broader disallow rule.
Where should robots.txt be placed?
Place the file at the root of your domain: https://example.com/robots.txt. It must be accessible at exactly that URL.