Field explanations
- Top-level settings define the tool behavior
- Environment-specific blocks override defaults
- Paths are usually relative to the config file or project root
- Secrets should be referenced, not committed
Minimal template
User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml
Common usage patterns
- Keep robots.txt small and reviewable
- Use comments only when the format supports them
- Commit example files with placeholder values
- Validate locally before deploying
Common errors
- Blocking important pages
- Using wildcards incorrectly
- Forgetting sitemap URL
- Disallowing CSS or JS
Online validation
Validate syntax first, then compare behavior against your deploy target. For JSON-based configs, use the JSON formatter before debugging tool-specific behavior.