robots.txt, llms.txt & Sitemaps for AI Crawlers

This guide is for publishers who want AI crawlers to find their content, and for agent developers who need to read a publisher's rules before crawling. You will see how AgentFlare publishes its free discovery files, how llms.txt points to pricing.json, and how pricing.json turns directories into free, paid or blocked zones. You will also learn the order an agent should follow: free files first, free teasers next, paid deep files last, with the crawler token sent in the right header and nowhere else.

How the flow works

Before you start

• curl installed on your machine
• Basic familiarity with HTTP status codes
• A registered AISA crawler account (only for paid fetches)

Step by step

1. Publish an open robots.txt

   AgentFlare's robots.txt opens with a wildcard Allow, then names each AI and search crawler with its own Allow: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Googlebot and Bingbot. It ends with a Sitemap line. Listing bots by name makes your intent explicit for each one. Keep the Sitemap line so agents can discover every free page from one file.

   # AgentFlare - AI crawlers welcome
   User-agent: *
   Allow: /
   
   User-agent: GPTBot
   Allow: /
   
   User-agent: ClaudeBot
   Allow: /
   
   Sitemap: https://agentflare.org/sitemap.xml

   S2 · agentflare.org

2. Publish llms.txt as a site map

   llms.txt gives an agent a plain-text overview of the site. AgentFlare's file explains that every article exists at one path in three forms: HTML, a free Markdown teaser (.md), and a metered .data.json or .full.md. It links the verify endpoint, the pricing map, registration and onboarding. Each section also has its own llms.txt listing its pages. Fetch it to see the layout.

   curl -s https://agentflare.org/llms.txt

   S3 · agentflare.org

3. Read pricing.json before paying

   llms.txt links to pricing.json, the directory price map. Its tiers field marks L1_free as index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt, and L2_paid as *.data.json and *.full.md. The default_mode is allow, so anything unmatched stays free. Each rule has a urlPattern, a mode of pay, allow or block, and a price. Guides are allow at 0.

   curl -s https://agentflare.org/pricing.json

   S3 · agentflare.orgS4 · agentflare.org

4. Fetch the free teaser, then the paid file

   Read the free .md teaser first to decide whether the deep file is worth buying. For the paid .data.json, send your crawler token in the X-Crawler-Token header (it replaces the earlier name X-AISA-Crawler-Token) plus your genuine User-Agent. Never put the token in Authorization: Bearer, which is reserved for the gateway layer. A granted fetch of a priced file deducts its price once.

   curl 'https://PUBLISHER-SITE/research/report.md'
   
   curl 'https://PUBLISHER-SITE/research/report.data.json' \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     -A 'MyBot/1.0 (+https://example.com/bot)'

   S1 · esa.aisa.oneS5 · esa.aisa.one

5. Handle 402 and 403 responses

   A 402 body is self-describing JSON. Read the error field: invalid_token means the token is missing or unknown, so fix the header and retry the same URL. insufficient_balance means top up and retry. A 403 access_blocked means the publisher withholds that resource from crawlers, and payment will not unlock it. Do not retry. Return to pricing.json and pick a pay-mode directory instead. Denials never charge.

   S5 · esa.aisa.one

6. Wire the rules into your own edge

   If you publish, copy the structure of pricing.json. The notes field says to import the rules array into cdn-agent Content Controls (ai_publisher_rules). The fields patternType, mode, price, billingType and priority map 1:1 to the publisher rule schema. Higher priority numbers are listed first, and unmatched paths stay free. That keeps teasers and discovery files open for search while deep data is monetized by directory.

   S4 · agentflare.org

AgentFlare directory rules for deep files

Common pitfalls

• An Allow: / in robots.txt only opens crawling. Files ending in .data.json or .full.md still follow the pricing.json rules and may cost money.
• Older AISA documentation pages still show X-AISA-Crawler-Token. Send X-Crawler-Token, and never send the token in Authorization: Bearer.
• A 403 access_blocked, such as on members-only paths, is a policy choice. Topping up does not help, so pivot to a pay-mode directory instead.
• Claiming another bot's User-Agent gets you marked unverified. Identify honestly so any free passes for verified bots can apply to you.

Key takeaways

• Free discovery files (robots.txt, llms.txt, sitemap.xml) come first; pricing.json then states exactly what deep data costs per directory.
• Teasers (.md) stay open for search; only .data.json and .full.md are metered, and unmatched paths stay free by default.
• Send the crawler token only in X-Crawler-Token, and treat a 403 access_blocked as final for that URL.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.