Home/Tools/AI crawler robots.txt

LOCAL POLICY PLANNER · CURRENT VENDOR TOKENS · RFC-AWARE

AI Crawler robots.txt Policy Planner

Decide separately whether AI search, model-training crawlers and user-triggered fetchers should access your public pages. Generate a reviewable robots.txt draft or audit an existing file against a specific path.

Important boundary: robots.txt is a voluntary crawl control, not authentication, a privacy barrier, a deindex command or a ranking boost. Some user-triggered fetchers may not follow it. Keep private content behind access control.

Plan and verify crawler access

Bot directory reviewed 2026-08-02.1 · no network requests from the tool

100% BROWSER-LOCAL

Specific bot groups do not inherit the wildcard group. The generator repeats these paths inside every specifically allowed bot group so an explicit Allow: / does not expose a path that User-agent: * blocks.

Choose a starting policy

Search discovery allowed; training blocked.

Generate or audit a file to calculate effective access.

robots.txt draft

Generate a policy to see the output.

No output action yet.

Findings

  • Generate or audit a file to see findings.
Effective access for the tested path
Bot tokenPurposeDecisionMatched rule

Search visibility and model training are different decisions

“Block AI” is too vague to produce a reliable policy. OpenAI documents OAI-SearchBot for surfacing sites in ChatGPT search and GPTBot for content that may be used to improve foundation models. Anthropic similarly distinguishes Claude-SearchBot from the training-oriented ClaudeBot. Perplexity describes PerplexityBot as a search crawler rather than a foundation-model training crawler. A publisher can therefore allow documented search bots while declining documented training use.

This separation does not promise citations, traffic or rankings. An allowed crawler may never visit a page, and a crawled page may not be selected in an answer. The page still needs accessible HTML, useful original content, stable canonical URLs and trustworthy internal links. Robots rules only describe permitted crawling.

The wildcard inheritance trap

Suppose the wildcard group blocks /admin/, then a later group says User-agent: OAI-SearchBot and Allow: /. Under the Robots Exclusion Protocol, a crawler with a matching specific product token uses the matching group instead of the wildcard fallback. The private-path rule is not inherited. The specific group can therefore allow /admin/ unless that path is repeated there.

This planner prevents that mistake by copying every private path into each specifically allowed group. The audit also detects a specific known-bot group that omits wildcard disallows. It is still your responsibility to list the correct paths and to protect sensitive resources with authentication. A URL blocked from crawling can remain discoverable through links, and robots.txt itself is public.

How rule matching works

RFC 9309 says matching starts at the beginning of the URI path, is normally case-sensitive for paths and uses the most specific matching rule—the match with the most octets. If equally specific Allow and Disallow rules conflict, Allow should win. Multiple groups matching the same product token are combined. If no product-token group matches, a User-agent: * group is the fallback; if neither exists, no robots rule applies.

The local auditor implements those core choices for an exact bot token and supports the widely used * wildcard and terminal $ extension when testing a path. Production crawlers can have vendor-specific extensions, caching and interpretation. Always test the published file with the relevant vendor tooling and server logs.

Current bot categories in this planner

AI search

OAI-SearchBot, Claude-SearchBot and PerplexityBot are documented for search discovery or result quality. Blocking them can reduce eligibility or visibility in the corresponding search experiences.

Training or AI-use controls

GPTBot and ClaudeBot are documented in relation to model improvement or training. Google-Extended is a robots control token for Gemini training and grounding; Google says it is not a separate HTTP user agent and does not affect Google Search inclusion or ranking.

User-triggered retrieval

ChatGPT-User, Claude-User and Perplexity-User support user-requested access. OpenAI says robots rules may not apply to ChatGPT-User; Perplexity says its user fetcher generally ignores robots.txt. Treat a Block line as a preference, not an access-control guarantee.

robots.txt cannot remove a page from search

Blocking crawling and preventing indexing are not the same operation. A search engine can sometimes know a blocked URL from links without fetching its contents. For a public page that should disappear from an index, use the engine-supported noindex mechanism while allowing the crawler to read that directive, then remove internal and Sitemap references as appropriate. For non-public content, require authentication or authorization; do not rely on a robots rule, an obscure URL or a noindex tag.

Publishing and verification workflow

  1. Inventory real surfaces. List public content, account areas, internal APIs, search results, staging paths and parameter patterns. Avoid blocking CSS or JavaScript required to understand public pages.
  2. Choose by purpose. Decide separately about search discovery, training/AI use and user-triggered retrieval. Record the business owner and review date.
  3. Generate and inspect. Read every group. Confirm private path prefixes are repeated in specifically allowed groups and the Sitemap URL is correct.
  4. Publish correctly. Serve UTF-8 plain text at the lowercase top-level /robots.txt URL for each hostname or subdomain. Avoid redirect chains and server errors.
  5. Verify outside this tool. Check the live response, content type, caching, CDN/WAF policy, vendor IP verification where published and actual access logs. Bot user-agent strings can be spoofed.
  6. Review changes. Vendor identities and behavior evolve. Subscribe to vendor updates and revalidate when the site, CDN or crawler documentation changes.

Sitemap and robots.txt relationship

A Sitemap: record is a commonly supported extension, not an access rule. The Sitemap should list canonical, indexable URLs you want search systems to discover; it should not list account, redirected, duplicate, blocked or noindex URLs. Adding a Sitemap record does not override Disallow and does not guarantee crawling or indexing. Conversely, leaving a URL out of the Sitemap does not hide it if links expose it.

Primary sources

Editorial and bot-directory review: 2026-08-02. Verify the linked current documentation before publishing because product tokens, IP ranges and crawler behavior can change.