Crawler Policy on the Modern Web: robots.txt and Beyond

Key concepts

Robots Exclusion Protocol

A standard, developed in 1994, by which a robots.txt file at the site root tells crawlers which portions of a website they may visit. It relies on voluntary compliance. S5 · en.wikipedia.org

Training versus search crawlers

Major vendors split crawlers by purpose: GPTBot is not OAI-SearchBot, and ClaudeBot is not Claude-SearchBot. Each has its own user-agent and can be controlled independently in robots.txt. S8 · digitalapplied.com

llms.txt

A proposal from Jeremy Howard of Answer.AI (September 2024) framed as helping language models use a site at inference time. It does not control training or crawl permissions. S2 · flatlineagency.comS8 · digitalapplied.com

Reverse-proxy crawler blocking

Network-level blocking provided by reverse proxies. It is lightly deployed today but offers stronger protection against AI crawlers than robots.txt, with its own limitations. S1 · cseweb.ucsd.edu

Web Bot Auth

A signed-request mechanism that determines which automated visitors are treated as legitimate. It is one of three parts of the access layer, alongside llms.txt and infrastructure rules. S2 · flatlineagency.com

Building a layered crawler policy

1. Publish robots.txt at the site rootPlace the file at the root of the site hierarchy, such as https://www.example.com/robots.txt. Compliant robots fetch it before other files; if it is absent, they assume no limits on crawling.
2. Write rules per user-agentAddress each crawler by its own user-agent string. Because training and search bots are separate, one rule no longer blocks both.
3. Disallow training, allow search botsA defensible default disallows GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended, while allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot that send referral traffic.
4. Enforce with WAF or proxy rulesA Cloudflare or firewall rule is enforced before robots.txt is read, so it beats any robots.txt Allow. For non-compliant crawlers such as Bytespider, IP or WAF blocking is the reliable defense.
5. Verify legitimate bots with signed requestsWeb Bot Auth determines which automated visitors get treated as legitimate, so infrastructure decisions are not left to whoever last touched the firewall.

What robots.txt does and where it stops

The Robots Exclusion Protocol was proposed by Martijn Koster in February 1994 and had become a de facto standard by June 1994. It was published as RFC 9309 in September 2022. Compliant robots read the file before fetching anything else, and a missing file is treated as no restriction. The weakness is that compliance is voluntary. Malicious bots can use the file as a directory of pages to visit, and some archival sites ignore it. A study of 203 professional artists found strong demand for tools like robots.txt, but constrained by gaps in technical awareness, agency in deploying them, and limited efficacy against unresponsive crawlers. One practitioner source notes that a robots.txt block can signal where valuable content lives, and recommends pairing it with authentication walls, canonical tagging and noindex headers. Treat it as a request to polite crawlers, not as access control.

Separating training, search and real-time bots

OpenAI runs GPTBot for training and OAI-SearchBot for ChatGPT search. Anthropic runs three separate bots, and Amazon, Google and Apple each separate training access from search and assistant access. The consequence is that blocking all AI bots as one bucket can remove a site from AI search citations. One source frames the defensible default as disallowing training crawlers while allowing search and retrieval crawlers. Cloudflare data shows why the distinction matters operationally: GPTBot rose from 5% to 30% share of AI crawling between May 2024 and May 2025, and Meta-ExternalAgent entered at 19%. Cloudflare also measured Anthropic crawling roughly 70,900 pages per referred visitor at its June 2025 peak, against about 5:1 for traditional Googlebot. Weigh these crawl-to-referral ratios against your own traffic goals before choosing which bots to allow.

Enforcement and verification beyond robots.txt

Existing tools include robots.txt, NoAI meta tags and active blocking by reverse proxies. Few are designed to resist a resource-rich adversary determined to scrape a site. Researchers who tested network-level blockers from reverse proxies found stronger protection against AI crawlers than robots.txt gives, despite limited deployment, but with their own limitations. Firewall rules sit earlier in the request path: a WAF block is enforced before robots.txt is read. For crawlers that ignore the file, such as Bytespider, IP or WAF blocking is described as the only reliable defense. The access layer also includes Web Bot Auth, a signed-request mechanism that decides which automated visitors are treated as legitimate. One agency source argues that infrastructure rules, not llms.txt, decide whether AI crawlers reach a commerce catalog at all. Review CDN and firewall settings as part of crawler policy.

robots.txt versus llms.txt

Timeline

1. 1994-02
   Martijn Koster proposes robots.txt on the www-talk mailing list.en.wikipedia.org
2. 2019-07
   Google announces proposal of the Robots Exclusion Protocol as an IETF standard.en.wikipedia.org
3. 2022-09
   Robots Exclusion Protocol published as RFC 9309.en.wikipedia.org
4. 2024-09
   Jeremy Howard of Answer.AI proposes llms.txt.flatlineagency.com
5. 2026-05
   Chrome Lighthouse adds an agentic browsing audit that checks for llms.txt.flatlineagency.com
6. 2026-06
   Google updates its generative AI search guide, repeating that special files are not needed.flatlineagency.com

Common pitfalls

• Blocking all AI bots as one bucket can remove a site from AI search citations, because training and search crawlers now use separate user-agents.
• robots.txt is not privacy or security: it relies on voluntary compliance, and malicious bots can use it as a directory of pages to visit.
• Do not treat llms.txt as a training opt-out. It guides navigation at inference time and does not control crawl permissions.
• Do not expect robots.txt to stop non-compliant crawlers such as Bytespider; WAF or IP blocking is the reliable defense for those.

Key takeaways

• Use robots.txt as a voluntary signal with separate rules for each training, search and real-time user-agent.
• Back policy with WAF or reverse-proxy rules, since firewall blocks are enforced before robots.txt is read.
• Treat llms.txt as a navigation aid for developer-facing tools, not as an access-control or training opt-out mechanism.

FAQ

How this page is made. Built from the full text of 8 sources found with the AISA Tavily API (advanced search), refreshed weekly. A language model organizes what those sources state into the blocks above; each block links the sources it draws on, and figures that do not appear in the sources are removed.