{
  "@context": "https://agentflare.org/schema",
  "type": "Article",
  "tier": "L2-full",
  "title": "Crawler Tokens & Auto-Pay: A How-To",
  "description": "How an AI crawler sets X-AISA-Crawler-Token and uses crawlerAutoPrice to auto-pay If you’re building an AI agent or onboarding your site to pay-per-crawl, you need to…",
  "canonical": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html",
  "category": "guides",
  "updated": "2026-06-23",
  "generated_at": "2026-06-23T15:42:04.744Z",
  "facts": [
    {
      "label": "Access",
      "value": "Free / open"
    },
    {
      "label": "Updated",
      "value": "2026-06-23"
    }
  ],
  "data": {
    "topic": "how an AI crawler sets an X-AISA-Crawler-Token and uses crawlerAutoPrice to auto-pay",
    "access": "free",
    "summary": "How an AI crawler sets X-AISA-Crawler-Token and uses crawlerAutoPrice to auto-pay If you’re building an AI agent or onboarding your site to pay-per-crawl, you need to…"
  },
  "analysis_md": "# How an AI crawler sets `X-AISA-Crawler-Token` and uses `crawlerAutoPrice` to auto-pay\n\nIf you’re building an AI agent or onboarding your site to pay-per-crawl, you need to understand how your crawler signals identity via `X-AISA-Crawler-Token` and how `crawlerAutoPrice` enables automatic payment negotiation using HTTP 402. The flow is simple: the crawler declares its identity, then either accepts the price sent by the publisher or offers a maximum price it’s willing to pay.\n\n## 1. Set `X-AISA-Crawler-Token` on every request\n\nPublishers and CDNs use the `X-AISA-Crawler-Token` header to identify and authenticate your AI crawler. You must include it on every crawl request:\n\n```http\nGET /article.html HTTP/1.1\nHost: example.com\nUser-Agent: MyAICrawler/1.0\nX-AISA-Crawler-Token: aisa-1234567890abcdef\n```\n\nThis token is typically issued by your platform or crawler operator and is used to tie requests to your billing account. Without a valid token, the edge may return an HTTP 402 Payment Required or deny access, depending on the publisher’s policy.\n\n## 2. Handle HTTP 402 and `crawler-price` for reactive auto‑pay\n\nWhen a publisher has pay-per-crawl enabled, your crawler may receive:\n\n```http\nHTTP/1.1 402 Payment Required\ncrawler-price: USD 0.001\n```\n\nTo auto-pay, your crawler must:\n\n1. Parse the `crawler-price` value.\n2. Decide whether the price is acceptable.\n3. Retry the request with `crawler-exact-price` set to that value:\n\n```http\nGET /article.html HTTP/1.1\nHost: example.com\nX-AISA-Crawler-Token: aisa-1234567890abcdef\ncrawler-exact-price: USD 0.001\n```\n\nIf the price is valid and within policy, the publisher returns HTTP 200 and the content.\n\n## 3. Use `crawlerAutoPrice` (or `crawler-max-price`) for proactive intent\n\nTo avoid repeated 402s, your crawler can proactively signal its willingness to pay:\n\n```http\nGET /article.html HTTP/1.1\nHost: example.com\nX-AISA-Crawler-Token: aisa-1234567890abcdef\ncrawler-max-price: USD 0.002\n```\n\nIf the publisher’s configured price is ≤ `crawler-max-price`, they return HTTP 200 and the content. If it’s higher, they return HTTP 402 with `crawler-price` set to the actual cost, and your crawler can choose to retry with `crawler-exact-price` or skip the page.\n\n## Key takeaways\n\n- Always send a valid `X-AISA-Crawler-Token` so publishers can identify and bill your crawler.\n- When you receive HTTP 402 with `crawler-price`, retry with `crawler-exact-price` to auto‑pay the quoted amount.\n- Use `crawler-max-price` (or equivalent `crawlerAutoPrice`) to express your maximum acceptable price and reduce 402 round‑trips.",
  "sources": [
    {
      "url": "https://blog.cloudflare.com/introducing-pay-per-crawl/"
    },
    {
      "url": "https://stytch.com/blog/how-to-block-ai-web-crawlers/"
    },
    {
      "url": "https://www.fastly.com/learning/what-are-ai-crawlers"
    },
    {
      "url": "https://www.facebook.com/groups/1404116417142065/posts/1790313531855683/"
    },
    {
      "url": "https://vercel.com/blog/the-rise-of-the-ai-crawler"
    },
    {
      "url": "https://www.botify.com/insight/ai-crawler-bots"
    },
    {
      "url": "https://www.radware.com/blog/ai-and-user-experience/understanding-ai-crawlers/"
    },
    {
      "url": "https://news.ycombinator.com/item?id=43476337"
    }
  ],
  "related": [
    {
      "name": "Getting Started with HTTP 402 Pay-Per-Crawl",
      "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html"
    },
    {
      "name": "Reading .data.json: A Field Guide for Agents",
      "url": "https://agentflare.org/guides/reading-datajson-a-field-guide-for-agents.html"
    },
    {
      "name": "robots.txt, llms.txt & Sitemaps for AI Crawlers",
      "url": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html"
    },
    {
      "name": "Pricing Content by Directory: Best Practices",
      "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html"
    }
  ],
  "pricing": {
    "price_usd": 0,
    "method": "402",
    "endpoint": "https://esa.aisa.one/api/v1/access/verify",
    "autopay_hint": "set crawlerAutoPrice=true with X-AISA-Crawler-Token",
    "onboarding": "https://esa.aisa.one/cdn/guide.html"
  },
  "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)"
}