Home/Wiki Errors/AI Crawlers
AI Crawlers / Robots

allow OAI-SearchBot but block GPTBot

Site owners often want AI search visibility while blocking model-training crawlers.

Error text / 报错原文

  • allow OAI-SearchBot block GPTBot
  • OAI-SearchBot robots.txt
  • GPTBot disallow

What it means

Site owners often want AI search visibility while blocking model-training crawlers.

Most common causes

  • Confusing search bots with training bots
  • One wildcard rule blocks all AI bots
  • Cloudflare bot rule overrides robots.txt
  • Missing sitemap

Fastest fix

  • Reproduce the smallest failing case.
  • Check environment, platform, and production settings.
  • Use the related local tool to classify the issue.
  • Fix the highest-risk security or data issue first.

Safe fix

  • Keep secrets out of client code and logs.
  • Prefer least privilege and explicit allowlists.
  • Add a regression test or checklist before retrying.
  • Document the working production configuration.

What not to do

  • Do not disable security controls as a permanent fix.
  • Do not paste secrets into public issue trackers or AI chats.
  • Do not trust preview success as production readiness.

Diagnostic commands

curl https://example.com/robots.txt
curl https://example.com/llms.txt
grep -i "GPTBot\|ClaudeBot\|OAI-SearchBot" access.log

Related tools

Related errors

Sources