AgentFlare

Research · Discovery & crawling · updated 2026-11-19

Pay-Per-Crawl and the New Bot Web

Pay-per-crawl lets publishers charge AI bots for access, with Cloudflare answering unpaid requests with HTTP 402. The hard part is pricing and policy: training, answer and user-triggered crawlers need different treatment, and blocking the wrong one removes a site from AI answers.

Sources88 domains
Concepts6
Reading time6 min
RefreshedWeekly
402 · Pay-per-Crawl$0.02

You're reading the full report — free for humans and search engines. The same content ships as a structured JSON dataset (normalized facts, sources and citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open HTTP 402 payment protocol.

⚡ Agents: fetch .data.json · $0.02 → agents: crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify
65%LM Tree revenue gain over single static priceS1 · arxiv.org
47%LM Tree gain over two-category pricingS1 · arxiv.org
1 billionHTTP 402 responses to AI crawlers sent dailyS3 · seresa.io
187%AI bot traffic growth through 2025S3 · seresa.io

Key concepts

Pay-per-crawl
A monetization model in which AI companies or other automated crawlers must pay websites to access and scrape content, positioned as a replacement for traffic-based revenue. S1 · arxiv.orgS8 · learnworkecosystemlibrary.com
HTTP 402 Payment Required
A status code present in the HTTP specification since the early 1990s and long unused. Cloudflare returns it to AI crawlers on domains with a pay-per-crawl price, instead of 200. S3 · seresa.io
Training crawlers
Bots such as GPTBot, ClaudeBot and CCBot that collect content which may shape future models. Blocking them is a licensing decision and does not affect live answers. S2 · neurise.ioS4 · squirrelscan.com
Answer crawlers
Bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot that feed the indexes behind live AI answers. Blocking them removes a site from citations. S2 · neurise.ioS4 · squirrelscan.com
User-triggered fetchers
Agents like ChatGPT-User that fetch a single URL because a person just asked about it. They act as a proxy for a human reader rather than a bulk crawler. S2 · neurise.ioS4 · squirrelscan.com
LM Tree
An adaptive pricing agent that grows a segmentation tree over a content library, using LLMs to find what separates high-value from low-value items, learning from binary purchase feedback alone. S1 · arxiv.org

How a Cloudflare pay-per-crawl request flows

  1. Publisher sets a domain priceThe site owner configures a pay-per-crawl price on the domain, using AI Crawl Control, which is free on every Cloudflare plan.
  2. Crawler requests a pageA known AI crawler from Cloudflare's AI Crawl Control list asks for a URL on the priced domain.
  3. Server answers 402, not 200Instead of returning the page, the server responds with HTTP 402 Payment Required to the bot.
  4. Crawler pays or backs offThe crawler can pay through Cloudflare's settlement layer or stop requesting the content.
  5. Human visitors are unaffectedPeople still receive 200 as always; only bots in the AI Crawl Control list see the 402.

Why pricing crawls is a mechanism problem

Search-era publishers earned revenue from direct traffic, through advertising, subscriptions or commerce. AI systems now crawl content for training and retrieval-augmented generation without directing users to the source, so the traffic model erodes. Services like Cloudflare and Tollbit provide infrastructure to charge bots, but the open question is what to charge. The LM Tree paper argues that pricing-relevant features are unstructured, lying in topic specificity, data richness and timeliness, and that the space of content types is large and hierarchical, so manual rule design is infeasible. Its proposed LM Tree grows a segmentation tree over the content library, using LLMs to find what separates high-value from low-value items from binary purchase feedback alone. On 8,939 articles and 80,451 buyer queries from a German technology publisher, it reports a 65% revenue gain over a single static price, a 47% gain over two-category pricing, and a 40% gain over the publisher's own 8-segment editorial taxonomy.

Cloudflare's 402 mechanism and its defaults

On a domain with a pay-per-crawl price, the server answers an AI crawler with HTTP 402 instead of 200; the crawler can pay through Cloudflare's settlement layer or back off. Human visitors still receive 200, and only bots in Cloudflare's AI Crawl Control list see the 402. Cloudflare customers are now sending over 1 billion HTTP 402 responses to AI crawlers every day, and AI Crawl Control is free on every Cloudflare plan. Since July 2025 Cloudflare has blocked AI crawlers by default on new domains, so some sites are invisible to assistants without ever having decided to be. Early adopters named in one guide include Condé Nast, Time, AP, BuzzFeed, Reddit and Pinterest. The same guide cites Cloudflare Radar figures showing AI bot traffic grew 187% through 2025 while human traffic grew 3.1%.

Per-crawler policy: charge, allow or block

Treating AI crawlers as one category is the common mistake. Training crawlers collect content for future models, answer crawlers feed live citations, and user-triggered fetchers grab one URL when a person asks. Blocking a training crawler is a licensing decision; blocking an answer crawler is a visibility decision. A WooCommerce guide recommends allowing OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User, and charging or blocking GPTBot, ClaudeBot, Google-Extended and CCBot. It marks Meta-ExternalAgent as charge, with quarterly review. It cites roughly 15.9% conversion for OAI-SearchBot and roughly 16.8% for Claude visitors. Vendors reuse similar names across all three families, such as GPTBot, OAI-SearchBot and ChatGPT-User, and each OpenAI token is its own switch: allowing one does not allow the others.

Three AI crawler families compared

AspectTraining crawlersAnswer crawlersUser-triggered fetchers
ExamplesGPTBot, ClaudeBot, CCBotOAI-SearchBot, Claude-SearchBot, PerplexityBotChatGPT-User, Claude-User, Perplexity-User
PurposeCollect content that may shape future modelsFeed indexes behind live answers and citationsFetch one URL when a person asks
Effect of blockingOpts out of future training; live answers unaffectedRemoved from citationsAssistant cannot open a link a user pastes in
robots.txt behaviorGPTBot and ClaudeBot respect itOAI-SearchBot, Claude-SearchBot and PerplexityBot respect itChatGPT-User does not; Claude-User does
Policy in the Cloudflare-based guideCharge or blockAllowAllow

Timeline

  1. 2023
    New York Times sues OpenAI and Microsoft over use of its content.learnworkecosystemlibrary.com
  2. 2024
    Reddit signs a $60 million/year deal with Google for AI training access.learnworkecosystemlibrary.com
  3. 2025-07
    Cloudflare begins blocking AI crawlers by default on new domains.neurise.io
  4. 2026-03
    ChatGPT-User traffic jumps 201% versus February, per Cloudflare Radar.seresa.io

Common pitfalls

  • Blocking all AI bots with one wildcard rule can remove a site from ChatGPT search results, not just from training.
  • Google-Extended is a robots.txt token, not a crawler: Googlebot fetches the page, and the token only governs Gemini training and grounding.
  • Redirect chains can drop pages from AI answers; the final URL should return 200 OK in 1 to 3 hops maximum.
  • AI crawler visits often leave no analytics trace, so server logs are needed to see which bots are actually fetching pages.

Key takeaways

  • Decide per crawler family: training bots are a licensing choice, answer bots a visibility choice, and user-triggered fetchers act for a human reader.
  • Cloudflare pay-per-crawl returns HTTP 402 to listed AI bots on priced domains while human visitors keep receiving 200.
  • Uniform pricing leaves revenue unclaimed; the LM Tree study reports a 65% gain over a single static price by segmenting content.

FAQ

What is pay-per-crawl?

A monetization model in which AI companies or other automated crawlers must pay websites to access and scrape content. It is meant to replace traffic-based revenue as AI systems consume content without sending users back to the source.

What does HTTP 402 do in Cloudflare pay-per-crawl?

When an AI crawler requests a page on a domain with a pay-per-crawl price, the server returns 402 instead of 200. The crawler can pay through Cloudflare's settlement layer or back off. Human visitors still receive 200.

Does blocking GPTBot remove my site from ChatGPT answers?

No. GPTBot is a training crawler, and disallowing it keeps future content out of the training set and nothing more. OAI-SearchBot builds the search index behind ChatGPT, and blocking it removes your pages as citation candidates.

How should content be priced for AI crawlers?

The LM Tree paper argues one fixed price does not fit heterogeneous content. Its segmentation-tree agent learns from binary purchase feedback and reports a 65% revenue gain over a single static price on a German technology publisher's content.

How this page is made. Built from the full text of 8 sources found with the AISA Tavily API (advanced search), refreshed weekly. A language model organizes what those sources state into the blocks above; each block links the sources it draws on, and figures that do not appear in the sources are removed.

Sources & citations

  1. Pay-Per-Crawl Pricing for AI: The LM-Tree Agent arxiv.org
  2. AI crawlers: GPTBot, ClaudeBot and PerplexityBot explained · Neurise neurise.io
  3. Cloudflare Pay-Per-Crawl for WooCommerce: 2026 Guide seresa.io
  4. AI crawlers: GPTBot, ClaudeBot, PerplexityBot explained squirrelscan.com
  5. AI Crawlers 101: GPTBot, ClaudeBot & PerplexityBot Explained hginsights.com
  6. AI Crawlers: How GPTBot, ClaudeBot, and PerplexityBot Read Your Site in 2026 sorank.com
  7. AI crawlers & redirects: GPTBot, ClaudeBot, Perplexity 2026 captaindns.com
  8. Pay-Per-Crawl | Learn & Work Ecosystem Library learnworkecosystemlibrary.com