Building a Pay-Per-Crawl Client for Your Agent

This guide is for developers who want their agent to fetch paid publisher content on the AISA pay-per-crawl network without hand-written special cases. You will build a small client with four parts: get a crawler token by registering or logging in, request URLs with the token header and your real User-Agent, read the charge receipt headers on success, and turn each denial into a clear next action. The pattern is shown in Python and in Node.js, and it ports to any HTTP stack.

How the flow works

Before you start

• An email and password for your crawler account
• An HTTP client library in your agent's runtime
• A real User-Agent string with a contact URL

Step by step

1. Register or log in for a token

   Call POST /api/agent/auth/register first. A 409 means the email already exists, so fall back to POST /api/agent/auth/login. Both responses carry crawler.crawlerToken, which is your crawl credential, and a separate session token for account APIs only. If signupBonus is present, a trial credit was granted. Prefer the host in guidance.registerApi when a denial names one, and keep one account throughout.

   import os, requests
   
   BASE = os.environ.get('AISA_BASE_URL', 'https://cdn.aisa.one')
   
   def get_account(email, password, name):
       body = {'email': email, 'password': password,
               'name': name, 'crawlerType': 'custom'}
       r = requests.post(BASE + '/api/agent/auth/register', json=body)
       if r.status_code == 409:
           r = requests.post(BASE + '/api/agent/auth/login',
                             json={'email': email, 'password': password})
       r.raise_for_status()
       data = r.json()
       crawler = data['crawler']
       # crawlerToken is for crawling; session token is for account APIs only
       return crawler['crawlerToken'], crawler['balance'], data['token']

   S2 · esa.aisa.oneS3 · esa.aisa.one

2. Crawl with the token header and honest User-Agent

   Send the token in the X-Crawler-Token header, which replaces the earlier X-AISA-Crawler-Token name that older pages still show. Never put the crawler token in Authorization: Bearer, because that header is reserved for the gateway layer. Set a User-Agent that names your real bot. Claiming another bot's identity marks you unverified. Billing happens at the publisher's edge, so the request itself is ordinary.

   UA = 'MyBot/1.0 (+https://example.com/bot)'
   
   def fetch(url, token):
       headers = {
           'X-Crawler-Token': token,
           'User-Agent': UA,
       }
       return requests.get(url, headers=headers)
   
   resp = fetch('https://PUBLISHER-SITE/research/report.data.json',
                'YOUR_CRAWLER_TOKEN')
   print(resp.status_code)

   S1 · esa.aisa.oneS2 · esa.aisa.one

3. Read the charge receipt headers

   On a 200 for a priced resource, gateways may attach X-AISA-Charged: true and X-AISA-Charge-Amount, for example 0.02. Treat them as optional: a free page simply lacks them. Log the amount next to the URL so your agent can report what each fetch cost. Fetching the same priced URL twice charges twice, so cache what you buy.

   def read_receipt(resp):
       charged = resp.headers.get('X-AISA-Charged') == 'true'
       amount = resp.headers.get('X-AISA-Charge-Amount')
       return {'url': resp.url, 'charged': charged, 'amount': amount}
   
   if resp.status_code == 200:
       print(read_receipt(resp))

   S2 · esa.aisa.oneS3 · esa.aisa.one

4. Branch on denial errors

   Denials are self-describing JSON, so branch on the error field. invalid_token means register or fix the header, then retry the same URL. insufficient_balance carries crawler.shortfall, so stop and tell your user. access_blocked is the publisher's policy: do not retry or top up for that URL, and move on to other content. Denials never charge. An anonymous call to a paid file returns 402 invalid_token.

   def handle(resp):
       if resp.status_code == 200:
           return {'next': 'use content'}
       denial = resp.json()
       err = denial.get('error')
       if err == 'invalid_token':
           return {'next': 'register or login, then retry same URL'}
       if err == 'insufficient_balance':
           return {'next': 'ask user to top up',
                   'shortfall': denial['crawler']['shortfall']}
       if err == 'access_blocked':
           return {'next': 'skip this URL and try other pay-mode content'}
       # price_required, price_mismatch, price_too_high
       return {'next': 'read the quote',
               'price': denial.get('price'),
               'currency': denial.get('currency')}

   S2 · esa.aisa.oneS4 · agentflare.org

5. Port the same client to Node.js

   The contract is identical in any runtime: one header, one honest User-Agent, then branch on status and error. This Node.js version folds the receipt read and the denial branches into a single function that returns a plain result object. Your agent loop can then decide whether to continue, ask the user to pay, or skip the URL, without knowing anything about the billing layer.

   const UA = 'MyBot/1.0 (+https://example.com/bot)';
   
   async function crawl(url, token) {
     const res = await fetch(url, {
       headers: { 'X-Crawler-Token': token, 'User-Agent': UA },
     });
     if (res.status === 200) {
       return { ok: true, body: await res.text(),
         charged: res.headers.get('X-AISA-Charged') === 'true',
         amount: res.headers.get('X-AISA-Charge-Amount') };
     }
     const denial = await res.json();
     switch (denial.error) {
       case 'invalid_token':
         return { ok: false, next: 'register or login, then retry' };
       case 'insufficient_balance':
         return { ok: false, next: 'ask user to top up',
           shortfall: denial.crawler.shortfall };
       case 'access_blocked':
         return { ok: false, next: 'skip this URL' };
       default:
         return { ok: false, next: 'read the quote',
           price: denial.price, currency: denial.currency };
     }
   }

   S1 · esa.aisa.oneS2 · esa.aisa.one

6. Fetch balance and hand top-ups to the user

   Always fetch the real balance with GET /api/agent/balance, using the same crawler token, before reporting it. Never quote it from memory. When funds run out, create a Stripe checkout session with the session token and show the checkout_url to your user. The minimum amount is 0.50. A human must pay, so do not call POST /api/agent/balance/add.

   def get_balance(token):
       r = requests.get(BASE + '/api/agent/balance',
                        headers={'X-Crawler-Token': token})
       r.raise_for_status()
       return r.json()['crawler']['balance']
   
   def topup_link(session_token, amount):
       r = requests.post(BASE + '/api/agent/topup/create-session',
                         json={'amount': amount},
                         headers={'Authorization': 'Bearer ' + session_token})
       r.raise_for_status()
       return r.json()['checkout_url']  # show this to your user

   S2 · esa.aisa.oneS3 · esa.aisa.one

Response outcomes and client actions

Common pitfalls

• Sending the crawler token in Authorization: Bearer makes the request count as unauthenticated and returns 402 invalid_token. Use X-Crawler-Token only.
• Copying X-AISA-Crawler-Token from older documentation pages. The current name is X-Crawler-Token.
• Printing the full crawler token in logs or reports. Show only a prefix such as aisa_crawler_9a4d...
• Calling POST /api/agent/balance/add to refill yourself. It exists only for a human operator pre-seeding a test account; real top-ups go through Stripe checkout.

Key takeaways

• One header, X-Crawler-Token, plus your honest User-Agent is the whole crawl contract; billing happens at the publisher's edge.
• Branch on the error field: fix the token on invalid_token, ask the user to pay on insufficient_balance, skip on access_blocked.
• Fetch the real balance from GET /api/agent/balance each time you report it, and let a human complete payment.

FAQ

How this page is made. Written from 4 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.