# Pay-Per-Crawl and the New Bot Web

> AI web crawlers now fall into three operational buckets: training crawlers that feed model development, search/indexing crawlers that power AI answers, and user-triggered…

- **Canonical:** https://agentflare.org/research/pay-per-crawl-and-the-new-bot-web.html
- **Updated:** 2026-06-15
- **Category:** research
- **Full structured data:** `https://agentflare.org/research/pay-per-crawl-and-the-new-bot-web.data.json` — $0.02 via AISA HTTP 402 (https://cdn.aisa.one/api/v1/access/verify; agents set crawlerAutoPrice=true)

## Key data

- **Topic:** discovery
- **Sources:** 10
- **Updated:** 2026-06-15

AI web crawlers now fall into three operational buckets: **training crawlers** that feed model development, **search/indexing crawlers** that power AI answers, and **user-triggered fetchers** that retrieve pages on demand. For developers, the practical shift in 2026 is that the major vendors split these roles into separate user agents, so you can no longer treat “AI bots” as one blockable class.[2][4][7]

_…full analysis and the complete dataset are available to agents for $0.02 — fetch `/research/pay-per-crawl-and-the-new-bot-web.data.json` (HTTP 402)._

## Sources

1. [https://www.tencentcloud.com/techpedia/143900](https://www.tencentcloud.com/techpedia/143900)
2. [https://www.anagram.ai/blog/ai-crawlers-explained-gptbot-claudebot-perplexitybot-and-how-to-let-them-in-2026](https://www.anagram.ai/blog/ai-crawlers-explained-gptbot-claudebot-perplexitybot-and-how-to-let-them-in-2026)
3. [https://evolveamz.com/ai-crawler-list-2026-ecommerce/](https://evolveamz.com/ai-crawler-list-2026-ecommerce/)
4. [https://nohacks.co/blog/ai-user-agents-landscape-2026](https://nohacks.co/blog/ai-user-agents-landscape-2026)
5. [https://www.tryaivo.com/blog/ai-crawler-cheat-sheet-2025-which-bots-should-you-allow](https://www.tryaivo.com/blog/ai-crawler-cheat-sheet-2025-which-bots-should-you-allow)
6. [https://www.oncrawl.com/ai/what-ai-bots-really-doing-your-site/](https://www.oncrawl.com/ai/what-ai-bots-really-doing-your-site/)
7. [https://www.digitalapplied.com/blog/ai-crawler-access-control-2026-robots-llms-txt-decision-matrix](https://www.digitalapplied.com/blog/ai-crawler-access-control-2026-robots-llms-txt-decision-matrix)
8. [https://www.humansecurity.com/learn/blog/crawlers-list-known-bots-guide/](https://www.humansecurity.com/learn/blog/crawlers-list-known-bots-guide/)
9. [AI Crawlers Explained: GPTBot, ClaudeBot, and PerplexityBot - Contently](https://contently.com/2026/05/06/ai-crawlers-explained-gptbot-claudebot-perplexitybot)
10. [Understanding AI Crawlers: Complete Guide 2025 | Qwairy](https://www.qwairy.co/blog/understanding-ai-crawlers-complete-guide)

## Related

- [Generative Engine Optimization (GEO): A Primer](https://agentflare.org/research/generative-engine-optimization-geo-a-primer.html)
- [llms.txt: The Standard for AI-Readable Sites](https://agentflare.org/research/llmstxt-the-standard-for-ai-readable-sites.html)
- [HTTP 402 & x402: How AI Agents Pay for Content](https://agentflare.org/research/http-402-x402-how-ai-agents-pay-for-content.html)
- [The AI Agent Economy in 2026](https://agentflare.org/research/the-ai-agent-economy-in-2026.html)
- [Model Context Protocol (MCP) Explained](https://agentflare.org/research/model-context-protocol-mcp-explained.html)
- [Stablecoins as Rails for Autonomous Agents](https://agentflare.org/research/stablecoins-as-rails-for-autonomous-agents.html)

---
_Part of AgentFlare, an agent-native data network powered by AISA. https://aisa.one/docs_