# robots.txt, llms.txt & Sitemaps for AI Crawlers

> robots.txt, llms.txt, and sitemap.xml best practices for AI crawlers and pay‑per‑crawl Use robots.txt to control access, sitemap.xml to advertise indexable URLs, and…

- **Canonical:** https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html
- **Updated:** 2026-06-23
- **Category:** guides

## Key data

- **Access:** Free / open
- **Updated:** 2026-06-23

# robots.txt, llms.txt, and sitemap.xml best practices for AI crawlers and pay‑per‑crawl

## Sources

1. [https://www.qwairy.co/guides/complete-guide-to-robots-txt-and-llms-txt-for-ai-crawlers](https://www.qwairy.co/guides/complete-guide-to-robots-txt-and-llms-txt-for-ai-crawlers)
2. [https://lseo.com/generative-engine-optimization/llms-txt-vs-robots-txt-vs-sitemap-xml-what-each-file-does-for-ai-discovery/](https://lseo.com/generative-engine-optimization/llms-txt-vs-robots-txt-vs-sitemap-xml-what-each-file-does-for-ai-discovery/)
3. [https://omnirank.net/blog/robots-txt-sitemap-llms-txt-complete-guide](https://omnirank.net/blog/robots-txt-sitemap-llms-txt-complete-guide)
4. [https://www.vanitech.com.au/blog/llms-txt-vs-robots-txt-vs-sitemap-what-businesses-should-allow/](https://www.vanitech.com.au/blog/llms-txt-vs-robots-txt-vs-sitemap-what-businesses-should-allow/)
5. [https://www.webless.ai/blog/from-search-engines-to-ai-bots-the-role-of-robots-txt-sitemap-xml-and-the-rise-of-llm-txt](https://www.webless.ai/blog/from-search-engines-to-ai-bots-the-role-of-robots-txt-sitemap-xml-and-the-rise-of-llm-txt)
6. [https://aicrawlercheck.com/blog/robots-txt-best-practices-ai-seo](https://aicrawlercheck.com/blog/robots-txt-best-practices-ai-seo)
7. [https://dageno.ai/academy/llms-txt-vs-robots-txt](https://dageno.ai/academy/llms-txt-vs-robots-txt)
8. [https://www.sitemap.ai/blog/best-practices-ai-crawlers-website](https://www.sitemap.ai/blog/best-practices-ai-crawlers-website)

## Related

- [Getting Started with HTTP 402 Pay-Per-Crawl](https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html)
- [Crawler Tokens & Auto-Pay: A How-To](https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html)
- [Reading .data.json: A Field Guide for Agents](https://agentflare.org/guides/reading-datajson-a-field-guide-for-agents.html)
- [Pricing Content by Directory: Best Practices](https://agentflare.org/guides/pricing-content-by-directory-best-practices.html)

---
_Part of AgentFlare, an agent-native data network powered by AISA. https://aisa.one/docs_