Getting Started with HTTP 402 Pay-Per-Crawl

This guide is for developers building agents or crawlers that run into HTTP 402 Payment Required on a publisher site. You will walk through the full first paid fetch: recognising what a 402 means, registering an account, saving the crawler token, checking and funding your balance, retrying the original URL with the right header, and interpreting the 200, 402 or 403 that comes back. Every command uses the AISA account endpoints and the single crawl header, so you can paste the snippets into a script and complete one paid request end to end.

How the flow works

Before you start

• A paid publisher URL that returned HTTP 402 to your crawler
• curl, or Python with the requests library
• An email address and a password of at least 8 characters

Step by step

1. Recognise what the 402 means

   A 402 from a publisher edge is a standard payment signal, not a block. The edge checked the URL's access rules and found no valid crawler token, or a token with too little balance, so it answered 402 instead of the content. The body is JSON: read error first, then follow the guidance block and the onboarding link. Denials never charge you.

   HTTP/1.1 402 Payment Required
   Content-Type: application/json
   X-Tengine-Error: payment required by http_crawler
   
   error: invalid_token
   onboarding: https://esa.aisa.one/agent/guide.html

   S2 · esa.aisa.oneS4 · agentflare.org

2. Register and save the crawler token

   Register once with POST /api/agent/auth/register. The response holds crawler.crawlerToken (format aisa_crawler_ followed by 64 hex characters) and a session token in token. Keep both, and never print the full crawler token in logs. The header that carries it is X-Crawler-Token, which replaces the earlier name X-AISA-Crawler-Token. If signupBonus is present, you already have trial credit.

   import requests
   BASE = 'https://cdn.aisa.one'
   r = requests.post(BASE + '/api/agent/auth/register', json={
       'email': 'mybot@example.com',
       'password': 'AtLeast8Chars',
       'name': 'My Crawler',
       'crawlerType': 'custom'})
   data = r.json()
   crawlerToken = data['crawler']['crawlerToken']
   sessionToken = data['token']
   print(data['crawler']['balance'], data.get('signupBonus'))

   S1 · esa.aisa.oneS3 · esa.aisa.one

3. Check the balance and top up

   Fetch the real balance with GET /api/agent/balance, using the same crawler token you crawl with. If you need funds, POST /api/agent/topup/create-session returns a checkout_url for Stripe, with a minimum of 0.50 USD. The top-up call takes the session token in Authorization: Bearer; a human must open the checkout_url and pay. Never send the crawler token in Authorization: Bearer.

   import requests
   BASE = 'https://cdn.aisa.one'
   b = requests.get(BASE + '/api/agent/balance',
       headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})
   print(b.json()['crawler']['balance'])
   
   t = requests.post(BASE + '/api/agent/topup/create-session',
       headers={'Authorization': 'Bearer SESSION_TOKEN'},
       json={'amount': 5.00})
   print(t.json()['checkout_url'])

   S1 · esa.aisa.oneS3 · esa.aisa.one

4. Retry the original URL with your token

   Send the same publisher URL that returned 402, this time with X-Crawler-Token and an honest User-Agent. Do not change the path or call a payment endpoint: the publisher edge verifies the token, deducts the price from your balance and serves the content in the same request. One request from you results in at most one charge.

   curl -i 'https://PUBLISHER-SITE/research/report.data.json' \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     -A 'MyBot/1.0 (+https://example.com/bot)'

   S1 · esa.aisa.oneS3 · esa.aisa.one

5. Read the outcome: 200, 402 or 403

   On 200, the content is yours; a paid fetch may carry the receipt headers X-AISA-Charged and X-AISA-Charge-Amount. On 402, read error: invalid_token means fix or register the token, insufficient_balance means top up using crawler.shortfall, then retry. On 403 access_blocked, the publisher withholds the resource from crawlers, so paying will not unlock it; look at the site's /pricing.json instead.

   import requests
   url = 'https://PUBLISHER-SITE/research/report.data.json'
   r = requests.get(url, headers={
       'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN',
       'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'})
   if r.status_code == 200:
       print('charged:', r.headers.get('X-AISA-Charged'),
             r.headers.get('X-AISA-Charge-Amount'))
   else:
       body = r.json()
       print(r.status_code, body.get('error'))
       print(body.get('guidance', {}).get('steps'))

   S2 · esa.aisa.oneS3 · esa.aisa.one

Crawl outcomes and what to do next

Common pitfalls

• Sending the crawler token in Authorization: Bearer makes the request count as unauthenticated and returns 402 invalid_token. Use X-Crawler-Token only.
• The session token from register or login is for account APIs such as top-up only. The crawler token is what you send to publisher sites.
• Do not quote a remembered balance. Call GET /api/agent/balance before and after a top-up, since every paid fetch changes it.
• Fetching a priced resource twice charges twice. A 403 access_blocked is a publisher policy, so retrying or topping up will not change it.

Key takeaways

• A 402 is an invitation to pay: register, hold a balance, and retry the same URL with X-Crawler-Token.
• Denials never charge; only a granted fetch of a priced resource deducts its per-request price.
• Treat 403 access_blocked differently from 402: payment cannot unlock it, so pivot to the pay-mode directories in /pricing.json.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.