Error text / 报错原文
Cloudflare blocking ClaudeBotClaudeBot blockedAI crawler blocked by WAF
What it means
Bot protection or WAF rules block desired AI crawlers even when robots.txt allows them.
Most common causes
- Managed bot rules too strict
- User-agent not allowlisted
- Rate limiting catches crawlers
- Origin blocks known crawler IPs
Fastest fix
- Reproduce the smallest failing case.
- Check environment, platform, and production settings.
- Use the related local tool to classify the issue.
- Fix the highest-risk security or data issue first.
Safe fix
- Keep secrets out of client code and logs.
- Prefer least privilege and explicit allowlists.
- Add a regression test or checklist before retrying.
- Document the working production configuration.
What not to do
- Do not disable security controls as a permanent fix.
- Do not paste secrets into public issue trackers or AI chats.
- Do not trust preview success as production readiness.
Diagnostic commands
curl https://example.com/robots.txt curl https://example.com/llms.txt grep -i "GPTBot\|ClaudeBot\|OAI-SearchBot" access.log