Home/Wiki Errors/AI Crawlers
AI Crawlers / llms.txt

llms.txt missing canonical pages

The llms.txt file exists but does not point models toward the best canonical pages.

Error text / 报错原文

  • llms.txt missing canonical pages
  • llms.txt not working
  • AI search not citing site

What it means

The llms.txt file exists but does not point models toward the best canonical pages.

Most common causes

  • Only homepage listed
  • No summaries
  • Stale URLs
  • Blocked by robots.txt

Fastest fix

  • Reproduce the smallest failing case.
  • Check environment, platform, and production settings.
  • Use the related local tool to classify the issue.
  • Fix the highest-risk security or data issue first.

Safe fix

  • Keep secrets out of client code and logs.
  • Prefer least privilege and explicit allowlists.
  • Add a regression test or checklist before retrying.
  • Document the working production configuration.

What not to do

  • Do not disable security controls as a permanent fix.
  • Do not paste secrets into public issue trackers or AI chats.
  • Do not trust preview success as production readiness.

Diagnostic commands

curl https://example.com/robots.txt
curl https://example.com/llms.txt
grep -i "GPTBot\|ClaudeBot\|OAI-SearchBot" access.log

Related tools

Related errors

Sources