Robots.txt Generator

Generate robots.txt file

Website URL

User Agents

Disallow (paths to block)

Allow (paths to allow)

Sitemaps

Crawl Delay (Optional)

seconds

Time between requests. Note: Not supported by Google.

Quick Presets

robots.txt

# robots.txt
# Generated by WebTools

User-agent: *
Disallow:

About robots.txt

Location: Place at your website root (e.g., example.com/robots.txt)
Disallow: Paths that crawlers should not access
Allow: Explicitly allow paths within disallowed directories
Note: robots.txt is a suggestion, not a security measure