Crawler Tokens & Auto-Pay: A How-To

How an AI crawler sets X-AISA-Crawler-Token and uses crawlerAutoPrice to auto-pay

If you’re building an AI agent or onboarding your site to pay-per-crawl, you need to understand how your crawler signals identity via X-AISA-Crawler-Token and how crawlerAutoPrice enables automatic payment negotiation using HTTP 402. The flow is simple: the crawler declares its identity, then either accepts the price sent by the publisher or offers a maximum price it’s willing to pay.

1. Set X-AISA-Crawler-Token on every request

Publishers and CDNs use the X-AISA-Crawler-Token header to identify and authenticate your AI crawler. You must include it on every crawl request:

``http GET /article.html HTTP/1.1 Host: example.com User-Agent: MyAICrawler/1.0 X-AISA-Crawler-Token: aisa-1234567890abcdef ``

This token is typically issued by your platform or crawler operator and is used to tie requests to your billing account. Without a valid token, the edge may return an HTTP 402 Payment Required or deny access, depending on the publisher’s policy.

2. Handle HTTP 402 and crawler-price for reactive auto‑pay

When a publisher has pay-per-crawl enabled, your crawler may receive:

``http HTTP/1.1 402 Payment Required crawler-price: USD 0.001 ``

To auto-pay, your crawler must:

1. Parse the crawler-price value.
2. Decide whether the price is acceptable.
3. Retry the request with crawler-exact-price set to that value:

``http GET /article.html HTTP/1.1 Host: example.com X-AISA-Crawler-Token: aisa-1234567890abcdef crawler-exact-price: USD 0.001 ``

If the price is valid and within policy, the publisher returns HTTP 200 and the content.

3. Use crawlerAutoPrice (or crawler-max-price) for proactive intent

To avoid repeated 402s, your crawler can proactively signal its willingness to pay:

``http GET /article.html HTTP/1.1 Host: example.com X-AISA-Crawler-Token: aisa-1234567890abcdef crawler-max-price: USD 0.002 ``

If the publisher’s configured price is ≤ crawler-max-price, they return HTTP 200 and the content. If it’s higher, they return HTTP 402 with crawler-price set to the actual cost, and your crawler can choose to retry with crawler-exact-price or skip the page.

Key takeaways

• Always send a valid X-AISA-Crawler-Token so publishers can identify and bill your crawler.
• When you receive HTTP 402 with crawler-price, retry with crawler-exact-price to auto‑pay the quoted amount.
• Use crawler-max-price (or equivalent crawlerAutoPrice) to express your maximum acceptable price and reduce 402 round‑trips.

Free guide synthesized by the AISA LLM layer (AISA Perplexity API). 2026-06-23.