AI Crawler Access Policy: Separating Search Display, Model Use, and Controlled Crawling

"Allow AI" is not a sufficiently precise policy. A site may be willing to let public documents enter search results while rejecting certain automated uses that cover all content; it may also need to allow user-initiated tools to read a public URL, but reject continuous full-site crawling. Merging all bots into User-agent: * lets a single commercial decision unexpectedly change search visibility. Programmatic SEO Template Release Gates determine which pages are worth indexing; this article determines which automated agents may crawl which public paths.

Maintain an Agent and Purpose Registry First

Build crawlerRegistry.v1, first distinguishing ROBOTS_PRODUCT_TOKEN from NETWORK_OBSERVABLE_CRAWLER. Each entry records the official token, declared purpose, official documentation, owner, last review time, and default policy; optional transportIdentity, dynamic IP profile, and verification method are attached only when the vendor actually publishes request identity. Google's documentation defines Google-Extended as an independent robots product token, but it has no independent HTTP User-Agent or dedicated IP, so it is marked POLICY_ONLY and its traffic cannot be attributed from request logs. OpenAI's published searchbot.json and gptbot.json can serve as dynamic range evidence for the corresponding network profiles, but IP matching still cannot substitute for robots rules and content licensing.

The policy layer uses only registry IDs; vendor strings are not scattered across templates:

policyVersion: crawler-access-2026-07
classes:
  public-search:
    allow: ["/articles/", "/docs/"]
    deny: ["/search", "/preview/"]
  model-use:
    allow: ["/docs/public/"]
    deny: ["/"]

The compiler then maps each class to the current product token and generates a deterministic robots.txt. This way, when vendor naming or policy changes, you only update the registry and version, without manually editing paths in multiple places. POLICY_ONLY entries only produce configuration decisions and test evidence; they do not generate "verified crawl volume"; only observable crawlers can enter request metrics via transport profiles. Policy reviews must record content owners, legal basis, and expected growth impact; the technical team must not unilaterally interpret "crawlable" as "usable for any purpose."

robots Is a Crawl Hint, Not an Authorization System

RFC 9309 defines the grouping, matching, and crawl rules of the robots exclusion protocol; it is not access authorization. Any client can ignore the file; systems that already know a URL may also retain link information from other sources. Drafts, tenant content, signed resources, and admin pages must rely on authentication, authorization, and network boundary protection—Disallow must not be treated as a confidentiality measure. The "authorize first, then expose evidence" principle from RAG Retrieval Access Control applies equally here.

robots.txt must return a predictable status on a fixed host/protocol; CDN error pages must not masquerade as a successful policy. The tester computes allow/deny for each token and representative URL per RFC matching semantics, covering case sensitivity, percent-encoding, query parameters, wildcards, and longest-match behavior. It must also check whether sitemap declarations and page-level noindex conflict with the policy: a crawler that needs to read noindex cannot see that directive if it is blocked by robots.

Validate with Real Requests, Not Just Text Diffs

Before release, compile a decision matrix of old and new policies in staging; any path that changes from deny to allow requires explicit approval. Production shadow splits into two evidence tracks: for observable crawlers, the edge computes wouldAllow per the new policy while still executing the old rule and recording transport verification; for POLICY_ONLY tokens, only the matching matrix is replayed against the real robots artifact and representative URLs—fabricating a class of nonexistent requests is forbidden. Search Crawler Observability covers that first category of sources and responses, preventing spoofed User-Agents from being mistaken for vendor behavior.

Negative fixtures must include at least: unknown tokens, spoofed IPs, registry expiry, robots cached old versions, cross-use of different subdomain policies, /docs/publicity mistakenly matching /docs/public/, policy service timeouts, and CDN origin-fetch failures. Unknown or unverifiable requests must be treated as ordinary public visitors and cannot obtain hidden content; but do not return site-wide 403 just because the verification service is down, or you will cut off ordinary users together with legitimate crawls.

Effect observation is reported separately: verified crawl volume, denied paths, and origin bandwidth for observable crawlers; compiled/cached version coverage and decision-fixture pass rates for policy-only tokens; plus a separate column for search exposure and indexing changes. More crawling does not equal more citations, and a robots token policy change must not be reported as a search ranking improvement or decline. AI Answer Citation Visibility Experiment measures citations with a fixed question set, which is a separate evidence chain.

Rollback points the policy alias back to the previous signed artifact, flushes the CDN robots cache, and continues recording old vs. new decision diffs. If the erroneous version exposed sensitive paths, you cannot just roll back the text: you must also inspect access logs, revoke leaked URLs or tokens, and fix the actual authorization boundary. Agent Egress SSRF and DNS Rebinding Defense explains that automated clients themselves are subject to network constraints—robots never takes on that responsibility.

A qualified policy ultimately answers three distinct questions: which verifiable agent this is, what purpose it declares and has been approved for, and whether the target path is public to begin with. Keeping these three separate means you will neither sacrifice access control for AI visibility nor damage search entry points with ambiguous blocks.

Further Reading

• Programmatic SEO Template Release Gates
• Search Crawler Observability
• AI Answer Citation Visibility Experiment
• RAG Retrieval Access Control