AI crawler allow/block matrix
Separate search crawlers from training crawlers without blocking useful discovery.
llms.txt generation
Create concise AI-readable site maps for docs, tools, APIs, and canonical pages.
Cloudflare bot rules
Avoid accidentally blocking ClaudeBot, OAI-SearchBot, PerplexityBot, or Google-Extended.
GEO visibility
Make pages easy for AI answers to cite without exposing private or paid content.