{ "@context": "https://agentflare.org/schema", "type": "Article", "tier": "L2-full", "title": "Building a Pay-Per-Crawl Client for Your Agent", "description": "Build one small client that registers or logs in, crawls with the X-Crawler-Token header and an honest User-Agent, reads charge receipt headers, and…", "canonical": "https://agentflare.org/guides/building-a-pay-per-crawl-client-for-your-agent.html", "category": "guides", "updated": "2026-10-16", "generated_at": "2026-10-16T00:37:01.652Z", "facts": [ { "label": "Track", "value": "For agent developers" }, { "label": "Level", "value": "Intermediate" }, { "label": "Access", "value": "Free / open" }, { "label": "Updated", "value": "2026-10-16" } ], "data": { "topic": "Building a Pay-Per-Crawl Client for Your Agent", "track": "agent", "access": "free", "generated_at": "2026-10-16T00:37:01.652Z", "difficulty": "Intermediate", "tldr": "Build one small client that registers or logs in, crawls with the X-Crawler-Token header and an honest User-Agent, reads charge receipt headers, and branches on 402 and 403 denials. Any agent can embed it.", "intro": "This guide is for developers who want their agent to fetch paid publisher content on the AISA pay-per-crawl network without hand-written special cases. You will build a small client with four parts: get a crawler token by registering or logging in, request URLs with the token header and your real User-Agent, read the charge receipt headers on success, and turn each denial into a clear next action. The pattern is shown in Python and in Node.js, and it ports to any HTTP stack.", "prerequisites": [ "An email and password for your crawler account", "An HTTP client library in your agent's runtime", "A real User-Agent string with a contact URL" ], "steps": [ { "title": "Register or log in for a token", "body": "Call POST /api/agent/auth/register first. A 409 means the email already exists, so fall back to POST /api/agent/auth/login. Both responses carry crawler.crawlerToken, which is your crawl credential, and a separate session token for account APIs only. If signupBonus is present, a trial credit was granted. Prefer the host in guidance.registerApi when a denial names one, and keep one account throughout.", "code": { "lang": "python", "text": "import os, requests\n\nBASE = os.environ.get('AISA_BASE_URL', 'https://cdn.aisa.one')\n\ndef get_account(email, password, name):\n body = {'email': email, 'password': password,\n 'name': name, 'crawlerType': 'custom'}\n r = requests.post(BASE + '/api/agent/auth/register', json=body)\n if r.status_code == 409:\n r = requests.post(BASE + '/api/agent/auth/login',\n json={'email': email, 'password': password})\n r.raise_for_status()\n data = r.json()\n crawler = data['crawler']\n # crawlerToken is for crawling; session token is for account APIs only\n return crawler['crawlerToken'], crawler['balance'], data['token']" }, "sources": [ "S2", "S3" ] }, { "title": "Crawl with the token header and honest User-Agent", "body": "Send the token in the X-Crawler-Token header, which replaces the earlier X-AISA-Crawler-Token name that older pages still show. Never put the crawler token in Authorization: Bearer, because that header is reserved for the gateway layer. Set a User-Agent that names your real bot. Claiming another bot's identity marks you unverified. Billing happens at the publisher's edge, so the request itself is ordinary.", "code": { "lang": "python", "text": "UA = 'MyBot/1.0 (+https://example.com/bot)'\n\ndef fetch(url, token):\n headers = {\n 'X-Crawler-Token': token,\n 'User-Agent': UA,\n }\n return requests.get(url, headers=headers)\n\nresp = fetch('https://PUBLISHER-SITE/research/report.data.json',\n 'YOUR_CRAWLER_TOKEN')\nprint(resp.status_code)" }, "sources": [ "S1", "S2" ] }, { "title": "Read the charge receipt headers", "body": "On a 200 for a priced resource, gateways may attach X-AISA-Charged: true and X-AISA-Charge-Amount, for example 0.02. Treat them as optional: a free page simply lacks them. Log the amount next to the URL so your agent can report what each fetch cost. Fetching the same priced URL twice charges twice, so cache what you buy.", "code": { "lang": "python", "text": "def read_receipt(resp):\n charged = resp.headers.get('X-AISA-Charged') == 'true'\n amount = resp.headers.get('X-AISA-Charge-Amount')\n return {'url': resp.url, 'charged': charged, 'amount': amount}\n\nif resp.status_code == 200:\n print(read_receipt(resp))" }, "sources": [ "S2", "S3" ] }, { "title": "Branch on denial errors", "body": "Denials are self-describing JSON, so branch on the error field. invalid_token means register or fix the header, then retry the same URL. insufficient_balance carries crawler.shortfall, so stop and tell your user. access_blocked is the publisher's policy: do not retry or top up for that URL, and move on to other content. Denials never charge. An anonymous call to a paid file returns 402 invalid_token.", "code": { "lang": "python", "text": "def handle(resp):\n if resp.status_code == 200:\n return {'next': 'use content'}\n denial = resp.json()\n err = denial.get('error')\n if err == 'invalid_token':\n return {'next': 'register or login, then retry same URL'}\n if err == 'insufficient_balance':\n return {'next': 'ask user to top up',\n 'shortfall': denial['crawler']['shortfall']}\n if err == 'access_blocked':\n return {'next': 'skip this URL and try other pay-mode content'}\n # price_required, price_mismatch, price_too_high\n return {'next': 'read the quote',\n 'price': denial.get('price'),\n 'currency': denial.get('currency')}" }, "sources": [ "S2", "S4" ] }, { "title": "Port the same client to Node.js", "body": "The contract is identical in any runtime: one header, one honest User-Agent, then branch on status and error. This Node.js version folds the receipt read and the denial branches into a single function that returns a plain result object. Your agent loop can then decide whether to continue, ask the user to pay, or skip the URL, without knowing anything about the billing layer.", "code": { "lang": "javascript", "text": "const UA = 'MyBot/1.0 (+https://example.com/bot)';\n\nasync function crawl(url, token) {\n const res = await fetch(url, {\n headers: { 'X-Crawler-Token': token, 'User-Agent': UA },\n });\n if (res.status === 200) {\n return { ok: true, body: await res.text(),\n charged: res.headers.get('X-AISA-Charged') === 'true',\n amount: res.headers.get('X-AISA-Charge-Amount') };\n }\n const denial = await res.json();\n switch (denial.error) {\n case 'invalid_token':\n return { ok: false, next: 'register or login, then retry' };\n case 'insufficient_balance':\n return { ok: false, next: 'ask user to top up',\n shortfall: denial.crawler.shortfall };\n case 'access_blocked':\n return { ok: false, next: 'skip this URL' };\n default:\n return { ok: false, next: 'read the quote',\n price: denial.price, currency: denial.currency };\n }\n}" }, "sources": [ "S1", "S2" ] }, { "title": "Fetch balance and hand top-ups to the user", "body": "Always fetch the real balance with GET /api/agent/balance, using the same crawler token, before reporting it. Never quote it from memory. When funds run out, create a Stripe checkout session with the session token and show the checkout_url to your user. The minimum amount is 0.50. A human must pay, so do not call POST /api/agent/balance/add.", "code": { "lang": "python", "text": "def get_balance(token):\n r = requests.get(BASE + '/api/agent/balance',\n headers={'X-Crawler-Token': token})\n r.raise_for_status()\n return r.json()['crawler']['balance']\n\ndef topup_link(session_token, amount):\n r = requests.post(BASE + '/api/agent/topup/create-session',\n json={'amount': amount},\n headers={'Authorization': 'Bearer ' + session_token})\n r.raise_for_status()\n return r.json()['checkout_url'] # show this to your user" }, "sources": [ "S2", "S3" ] } ], "reference": { "title": "Response outcomes and client actions", "columns": [ "Status and error", "Meaning", "Client action" ], "rows": [ [ "200 (free)", "Free tier content, no charge", "Use the content" ], [ "200 (paid)", "Priced resource paid for; may carry X-AISA-Charged and X-AISA-Charge-Amount", "Record the receipt" ], [ "402 invalid_token", "No, unknown or inactive token", "Register or fix the header, retry same URL" ], [ "402 insufficient_balance", "Balance too low; crawler.shortfall gives the gap", "Surface checkout_url, wait for user, retry" ], [ "402 price_required / price_mismatch / price_too_high", "Price negotiation states, rarely seen", "Read price and currency from the body" ], [ "403 access_blocked", "Publisher forbids crawler access; payment does not unlock it", "Do not retry; pivot to pay-mode directories in /pricing.json" ] ], "sources": [ "S2" ] }, "pitfalls": [ { "text": "Sending the crawler token in Authorization: Bearer makes the request count as unauthenticated and returns 402 invalid_token. Use X-Crawler-Token only.", "source": "S1" }, { "text": "Copying X-AISA-Crawler-Token from older documentation pages. The current name is X-Crawler-Token.", "source": "S1" }, { "text": "Printing the full crawler token in logs or reports. Show only a prefix such as aisa_crawler_9a4d...", "source": "S2" }, { "text": "Calling POST /api/agent/balance/add to refill yourself. It exists only for a human operator pre-seeding a test account; real top-ups go through Stripe checkout.", "source": "S3" } ], "takeaways": [ "One header, X-Crawler-Token, plus your honest User-Agent is the whole crawl contract; billing happens at the publisher's edge.", "Branch on the error field: fix the token on invalid_token, ask the user to pay on insufficient_balance, skip on access_blocked.", "Fetch the real balance from GET /api/agent/balance each time you report it, and let a human complete payment." ], "faq": [ { "q": "Which header carries the crawler token?", "a": "X-Crawler-Token. It replaces the earlier X-AISA-Crawler-Token name that older AISA pages still show. Never use Authorization: Bearer for the crawler token; that header is reserved for the gateway layer." }, { "q": "How do I know what a request cost?", "a": "Check X-AISA-Charged and X-AISA-Charge-Amount on a 200 response; gateways may attach them. You can also fetch the balance with GET /api/agent/balance before and after the request." }, { "q": "Should my client retry a 403 access_blocked?", "a": "No. It is the publisher's policy choice, and payment does not unlock it. Skip that URL, check the site's /pricing.json for purchasable pay-mode directories, and continue the task." }, { "q": "Can my agent top up its own balance?", "a": "No. Create a Stripe checkout session with POST /api/agent/topup/create-session, using the session token, and give the checkout_url to your user. A human completes the payment, and the minimum amount is 0.50." } ], "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html", "domain": "esa.aisa.one" }, { "title": "AISA Agent Integration Guide — API reference", "url": "https://esa.aisa.one/agent/docs.html", "domain": "esa.aisa.one" }, { "title": "AISA agent skill (SKILL.md)", "url": "https://esa.aisa.one/agent/skill.md", "domain": "esa.aisa.one" }, { "title": "Live capture: anonymous request to a paid AgentFlare file", "url": "https://agentflare.org/markets/index.data.json", "domain": "agentflare.org" } ] }, "analysis_md": "Build one small client that registers or logs in, crawls with the X-Crawler-Token header and an honest User-Agent, reads charge receipt headers, and branches on 402 and 403 denials. Any agent can embed it.\n\nThis guide is for developers who want their agent to fetch paid publisher content on the AISA pay-per-crawl network without hand-written special cases. You will build a small client with four parts: get a crawler token by registering or logging in, request URLs with the token header and your real User-Agent, read the charge receipt headers on success, and turn each denial into a clear next action. The pattern is shown in Python and in Node.js, and it ports to any HTTP stack.\n\n## Before you start\n\n- An email and password for your crawler account\n- An HTTP client library in your agent's runtime\n- A real User-Agent string with a contact URL\n\n## Step by step\n\n### 1. Register or log in for a token\n\nCall POST /api/agent/auth/register first. A 409 means the email already exists, so fall back to POST /api/agent/auth/login. Both responses carry crawler.crawlerToken, which is your crawl credential, and a separate session token for account APIs only. If signupBonus is present, a trial credit was granted. Prefer the host in guidance.registerApi when a denial names one, and keep one account throughout.\n\n```python\nimport os, requests\n\nBASE = os.environ.get('AISA_BASE_URL', 'https://cdn.aisa.one')\n\ndef get_account(email, password, name):\n body = {'email': email, 'password': password,\n 'name': name, 'crawlerType': 'custom'}\n r = requests.post(BASE + '/api/agent/auth/register', json=body)\n if r.status_code == 409:\n r = requests.post(BASE + '/api/agent/auth/login',\n json={'email': email, 'password': password})\n r.raise_for_status()\n data = r.json()\n crawler = data['crawler']\n # crawlerToken is for crawling; session token is for account APIs only\n return crawler['crawlerToken'], crawler['balance'], data['token']\n```\n\n### 2. Crawl with the token header and honest User-Agent\n\nSend the token in the X-Crawler-Token header, which replaces the earlier X-AISA-Crawler-Token name that older pages still show. Never put the crawler token in Authorization: Bearer, because that header is reserved for the gateway layer. Set a User-Agent that names your real bot. Claiming another bot's identity marks you unverified. Billing happens at the publisher's edge, so the request itself is ordinary.\n\n```python\nUA = 'MyBot/1.0 (+https://example.com/bot)'\n\ndef fetch(url, token):\n headers = {\n 'X-Crawler-Token': token,\n 'User-Agent': UA,\n }\n return requests.get(url, headers=headers)\n\nresp = fetch('https://PUBLISHER-SITE/research/report.data.json',\n 'YOUR_CRAWLER_TOKEN')\nprint(resp.status_code)\n```\n\n### 3. Read the charge receipt headers\n\nOn a 200 for a priced resource, gateways may attach X-AISA-Charged: true and X-AISA-Charge-Amount, for example 0.02. Treat them as optional: a free page simply lacks them. Log the amount next to the URL so your agent can report what each fetch cost. Fetching the same priced URL twice charges twice, so cache what you buy.\n\n```python\ndef read_receipt(resp):\n charged = resp.headers.get('X-AISA-Charged') == 'true'\n amount = resp.headers.get('X-AISA-Charge-Amount')\n return {'url': resp.url, 'charged': charged, 'amount': amount}\n\nif resp.status_code == 200:\n print(read_receipt(resp))\n```\n\n### 4. Branch on denial errors\n\nDenials are self-describing JSON, so branch on the error field. invalid_token means register or fix the header, then retry the same URL. insufficient_balance carries crawler.shortfall, so stop and tell your user. access_blocked is the publisher's policy: do not retry or top up for that URL, and move on to other content. Denials never charge. An anonymous call to a paid file returns 402 invalid_token.\n\n```python\ndef handle(resp):\n if resp.status_code == 200:\n return {'next': 'use content'}\n denial = resp.json()\n err = denial.get('error')\n if err == 'invalid_token':\n return {'next': 'register or login, then retry same URL'}\n if err == 'insufficient_balance':\n return {'next': 'ask user to top up',\n 'shortfall': denial['crawler']['shortfall']}\n if err == 'access_blocked':\n return {'next': 'skip this URL and try other pay-mode content'}\n # price_required, price_mismatch, price_too_high\n return {'next': 'read the quote',\n 'price': denial.get('price'),\n 'currency': denial.get('currency')}\n```\n\n### 5. Port the same client to Node.js\n\nThe contract is identical in any runtime: one header, one honest User-Agent, then branch on status and error. This Node.js version folds the receipt read and the denial branches into a single function that returns a plain result object. Your agent loop can then decide whether to continue, ask the user to pay, or skip the URL, without knowing anything about the billing layer.\n\n```javascript\nconst UA = 'MyBot/1.0 (+https://example.com/bot)';\n\nasync function crawl(url, token) {\n const res = await fetch(url, {\n headers: { 'X-Crawler-Token': token, 'User-Agent': UA },\n });\n if (res.status === 200) {\n return { ok: true, body: await res.text(),\n charged: res.headers.get('X-AISA-Charged') === 'true',\n amount: res.headers.get('X-AISA-Charge-Amount') };\n }\n const denial = await res.json();\n switch (denial.error) {\n case 'invalid_token':\n return { ok: false, next: 'register or login, then retry' };\n case 'insufficient_balance':\n return { ok: false, next: 'ask user to top up',\n shortfall: denial.crawler.shortfall };\n case 'access_blocked':\n return { ok: false, next: 'skip this URL' };\n default:\n return { ok: false, next: 'read the quote',\n price: denial.price, currency: denial.currency };\n }\n}\n```\n\n### 6. Fetch balance and hand top-ups to the user\n\nAlways fetch the real balance with GET /api/agent/balance, using the same crawler token, before reporting it. Never quote it from memory. When funds run out, create a Stripe checkout session with the session token and show the checkout_url to your user. The minimum amount is 0.50. A human must pay, so do not call POST /api/agent/balance/add.\n\n```python\ndef get_balance(token):\n r = requests.get(BASE + '/api/agent/balance',\n headers={'X-Crawler-Token': token})\n r.raise_for_status()\n return r.json()['crawler']['balance']\n\ndef topup_link(session_token, amount):\n r = requests.post(BASE + '/api/agent/topup/create-session',\n json={'amount': amount},\n headers={'Authorization': 'Bearer ' + session_token})\n r.raise_for_status()\n return r.json()['checkout_url'] # show this to your user\n```\n\n## Response outcomes and client actions\n\n| Status and error | Meaning | Client action |\n| --- | --- | --- |\n| 200 (free) | Free tier content, no charge | Use the content |\n| 200 (paid) | Priced resource paid for; may carry X-AISA-Charged and X-AISA-Charge-Amount | Record the receipt |\n| 402 invalid_token | No, unknown or inactive token | Register or fix the header, retry same URL |\n| 402 insufficient_balance | Balance too low; crawler.shortfall gives the gap | Surface checkout_url, wait for user, retry |\n| 402 price_required / price_mismatch / price_too_high | Price negotiation states, rarely seen | Read price and currency from the body |\n| 403 access_blocked | Publisher forbids crawler access; payment does not unlock it | Do not retry; pivot to pay-mode directories in /pricing.json |\n\n## Common pitfalls\n\n- Sending the crawler token in Authorization: Bearer makes the request count as unauthenticated and returns 402 invalid_token. Use X-Crawler-Token only.\n- Copying X-AISA-Crawler-Token from older documentation pages. The current name is X-Crawler-Token.\n- Printing the full crawler token in logs or reports. Show only a prefix such as aisa_crawler_9a4d...\n- Calling POST /api/agent/balance/add to refill yourself. It exists only for a human operator pre-seeding a test account; real top-ups go through Stripe checkout.\n\n## Key takeaways\n\n- One header, X-Crawler-Token, plus your honest User-Agent is the whole crawl contract; billing happens at the publisher's edge.\n- Branch on the error field: fix the token on invalid_token, ask the user to pay on insufficient_balance, skip on access_blocked.\n- Fetch the real balance from GET /api/agent/balance each time you report it, and let a human complete payment.\n\n## FAQ\n\n**Which header carries the crawler token?**\nX-Crawler-Token. It replaces the earlier X-AISA-Crawler-Token name that older AISA pages still show. Never use Authorization: Bearer for the crawler token; that header is reserved for the gateway layer.\n\n**How do I know what a request cost?**\nCheck X-AISA-Charged and X-AISA-Charge-Amount on a 200 response; gateways may attach them. You can also fetch the balance with GET /api/agent/balance before and after the request.\n\n**Should my client retry a 403 access_blocked?**\nNo. It is the publisher's policy choice, and payment does not unlock it. Skip that URL, check the site's /pricing.json for purchasable pay-mode directories, and continue the task.\n\n**Can my agent top up its own balance?**\nNo. Create a Stripe checkout session with POST /api/agent/topup/create-session, using the session token, and give the checkout_url to your user. A human completes the payment, and the minimum amount is 0.50.\n", "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html" }, { "title": "AISA Agent Integration Guide — API reference", "url": "https://esa.aisa.one/agent/docs.html" }, { "title": "AISA agent skill (SKILL.md)", "url": "https://esa.aisa.one/agent/skill.md" }, { "title": "Live capture: anonymous request to a paid AgentFlare file", "url": "https://agentflare.org/markets/index.data.json" } ], "related": [ { "name": "Getting Started with HTTP 402 Pay-Per-Crawl", "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html" }, { "name": "Crawler Tokens & Auto-Pay: A How-To", "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html" }, { "name": "Handling 402 and 403 Responses: An Error Reference for Agents", "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html" }, { "name": "Budgeting a Paid Crawl: Discover, Survey, Then Buy", "url": "https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html" }, { "name": "Pricing Content by Directory: Best Practices", "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html" }, { "name": "robots.txt, llms.txt & Sitemaps for AI Crawlers", "url": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html" } ], "pricing": { "price_usd": 0, "method": "402", "endpoint": "https://esa.aisa.one/api/v1/access/verify", "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header", "onboarding": "https://esa.aisa.one/agent/guide.html" }, "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)" }