Crawler Tokens & Auto-Pay: A How-To

This guide is for developers building agents or crawlers that fetch paid publisher content on the AISA pay-per-crawl network. You will register an account, tell the two tokens apart, send the right one in the right header, and fetch your real balance before and after spending. You will also create a Stripe top-up link for your user, read a 402 body to see a price, and tell a priced directory from a blocked one. Billing happens at the publisher's edge, so your side stays small: one header, an honest User-Agent, and a positive balance.

How the flow works

Before you start

• curl and Python with the requests library installed
• An email and password (8+ characters) for the crawler account
• A human who can complete a Stripe payment later

Step by step

1. Register and separate your two tokens

   Call the register endpoint once. The response gives you two different credentials. The crawlerToken inside the crawler object (format aisa_crawler_ plus 64 hex characters) is your crawl credential. The top-level token is a short-lived session token for account APIs only. Save both, but never print the full crawler token; log a prefix instead. If the email already exists you get a 409, so use login instead.

   import requests
   
   BASE = 'https://cdn.aisa.one'
   r = requests.post(BASE + '/api/agent/auth/register', json={
       'email': 'mybot@example.com',
       'password': 'YOUR_PASSWORD',
       'name': 'MyBot',
       'crawlerType': 'custom',
   })
   data = r.json()
   crawlerToken = data['crawler']['crawlerToken']
   sessionToken = data['token']
   print(crawlerToken[:17] + '...')
   print(data['crawler']['balance'])

   S2 · esa.aisa.oneS3 · esa.aisa.one

2. Send the crawler token on publisher requests

   Request publisher URLs with the crawler token in the X-Crawler-Token header. This name replaces the earlier X-AISA-Crawler-Token, which older AISA pages still show. Add your genuine User-Agent too, because the gateway passes it through for identification. Never put the crawler token in Authorization: Bearer; that header is reserved for the gateway layer, and a request that does is treated as unauthenticated (402 invalid_token). A paid 200 may carry receipt headers such as X-AISA-Charged.

   curl -s -i 'https://PUBLISHER-SITE/research/report.data.json' \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     -A 'MyBot/1.0 (+https://example.com/bot)'

   S1 · esa.aisa.oneS2 · esa.aisa.one

3. Fetch your real balance after every spend

   Read the balance with the same crawler token you crawl with: GET /api/agent/balance needs no email, password or session. Always fetch it before reporting a figure, because a remembered value goes stale as soon as you spend. The docs give a worked case: a real balance of 0.06 plus a 100 top-up is 100.06, not 100.10. Stay on one account throughout and never re-register mid-task.

   curl -s 'https://cdn.aisa.one/api/agent/balance' \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'
   # response: crawler.balance and crawler.totalSpent

   S1 · esa.aisa.oneS2 · esa.aisa.oneS4 · esa.aisa.one

4. Create a top-up link and hand it to your user

   Top-ups use the session token, not the crawler token: send it as Authorization: Bearer on POST /api/agent/topup/create-session with an amount (minimum 0.50 USD). The response contains checkout_url, a Stripe page. A human must complete the payment, so show the link to your user and wait; the balance credits automatically afterwards. Never call POST /api/agent/balance/add to fund yourself, since it exists only for operators pre-seeding test accounts.

   r = requests.post(
       BASE + '/api/agent/topup/create-session',
       headers={'Authorization': 'Bearer ' + sessionToken},
       json={'amount': 5.00},
   )
   print('Ask the user to pay here:', r.json()['checkout_url'])

   S2 · esa.aisa.oneS4 · esa.aisa.one

5. Read pricing modes before you spend

   Publishers price per URL directory, and each rule has an access mode: pay, allow or block. Pay paths charge their per-request price on each granted fetch, so fetching twice means paying twice. Allow and block paths cost nothing. Many publishers expose /pricing.json, a directory-to-price-and-mode map, and describe tiers in /llms.txt; both are free to fetch. Free HTML pages and .md summaries help you choose what is worth buying before you pay for .data.json or .full.md files.

   curl -s 'https://PUBLISHER-SITE/pricing.json'
   curl -s 'https://PUBLISHER-SITE/llms.txt'

   S2 · esa.aisa.oneS3 · esa.aisa.oneS4 · esa.aisa.one

6. Handle 402 quotes and price negotiation states

   Denials never charge, so a 402 is a free way to discover a price: the rule object in the body shows price, currency and billingType. The negotiation errors price_required, price_mismatch and price_too_high are normally handled by the gateway and rarely seen; when one appears, the quote is in price and currency. For insufficient_balance, crawler.shortfall says how much more you need. A 403 access_blocked is a publisher policy, so pivot to pay-mode directories.

   resp = requests.get('https://PUBLISHER-SITE/research/report.data.json',
       headers={'X-Crawler-Token': crawlerToken,
                'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'})
   if resp.status_code == 200:
       print('Charged:', resp.headers.get('X-AISA-Charge-Amount'))
   else:
       body = resp.json()
       err = body.get('error')
       if err == 'insufficient_balance':
           print('Shortfall:', body['crawler']['shortfall'])
       elif err in ('price_required', 'price_mismatch', 'price_too_high'):
           print('Quote:', body.get('price'), body.get('currency'))
       elif err == 'access_blocked':
           print('Blocked by publisher policy; try pay-mode directories')
       else:
           print(err)

   S2 · esa.aisa.oneS3 · esa.aisa.oneS4 · esa.aisa.one

Crawler token vs session token

Common pitfalls

• Sending the crawler token in Authorization: Bearer makes the request unauthenticated and returns 402 invalid_token. Use the X-Crawler-Token header only.
• Copying X-AISA-Crawler-Token from older documentation pages. The current header name is X-Crawler-Token.
• Quoting a balance from memory, such as the signup bonus. Fetch GET /api/agent/balance first, then add any top-up amount to that real figure.
• Retrying or topping up after a 403 access_blocked. Payment does not unlock it, so move on to pay-mode directories listed in /pricing.json.

Key takeaways

• Crawl with the crawler token in X-Crawler-Token; use the session token only as a Bearer token on account APIs.
• Fetch the real balance from GET /api/agent/balance before reporting it, and give the user the checkout_url to top up.
• Read pricing.json and 402 bodies to learn prices for free; denials never charge, and 403 means choose another directory.

FAQ

How this page is made. Written from 4 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.