Home/Wiki Errors/AI Crawlers
AI Crawlers / Robots

PerplexityBot ignored robots.txt

A crawler appears to fetch pages despite rules, or logs are mixing crawler user agents with referrers and proxy behavior.

Error text / 报错原文

  • PerplexityBot ignored robots.txt
  • AI crawler ignored robots.txt

What it means

A crawler appears to fetch pages despite rules, or logs are mixing crawler user agents with referrers and proxy behavior.

Most common causes

  • Cached content
  • Different user-agent than expected
  • Robots path mismatch
  • Bot impersonation

Fastest fix

  • Reproduce the smallest failing case.
  • Check environment, platform, and production settings.
  • Use the related local tool to classify the issue.
  • Fix the highest-risk security or data issue first.

Safe fix

  • Keep secrets out of client code and logs.
  • Prefer least privilege and explicit allowlists.
  • Add a regression test or checklist before retrying.
  • Document the working production configuration.

What not to do

  • Do not disable security controls as a permanent fix.
  • Do not paste secrets into public issue trackers or AI chats.
  • Do not trust preview success as production readiness.

Diagnostic commands

curl https://example.com/robots.txt
curl https://example.com/llms.txt
grep -i "GPTBot\|ClaudeBot\|OAI-SearchBot" access.log

Related tools

Related errors

Sources