{ "@context": "https://agentflare.org/schema", "type": "Article", "tier": "L2-full", "title": "Handling 402 and 403 Responses: An Error Reference for Agents", "description": "Every AISA denial arrives as self-describing JSON: read the error field, act on the guidance block, and report plainly to the user. This guide covers each…", "canonical": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html", "category": "guides", "updated": "2026-10-16", "generated_at": "2026-10-16T00:37:01.652Z", "facts": [ { "label": "Track", "value": "For agent developers" }, { "label": "Level", "value": "Beginner" }, { "label": "Access", "value": "Free / open" }, { "label": "Updated", "value": "2026-10-16" } ], "data": { "topic": "Handling 402 and 403 Responses: An Error Reference for Agents", "track": "agent", "access": "free", "generated_at": "2026-10-16T00:37:01.652Z", "difficulty": "Beginner", "tldr": "Every AISA denial arrives as self-describing JSON: read the error field, act on the guidance block, and report plainly to the user. This guide covers each 402 and 403 code, the next action, and the wording.", "intro": "This guide is for developers building agents or crawlers that fetch paid publisher content over the AISA pay-per-crawl network. You will be able to branch on every denial code, follow the guidance block that comes with each denial, and know when to retry, when to hand the user a payment link, and when to skip a URL and move on. The guide ends with a reference table of error codes and a template for what the agent should tell its user, so the conversation stays accurate and calm.", "prerequisites": [ "A registered AISA crawler account with its crawler token", "A paid publisher URL behind an AISA-integrated gateway", "curl or Python with the requests library" ], "steps": [ { "title": "Send the token header and read the status", "body": "Request the URL with your crawler token and your genuine User-Agent. The header is X-Crawler-Token; it replaces the earlier name X-AISA-Crawler-Token that older AISA pages still show. Never put the token in Authorization: Bearer, which is reserved for the gateway layer. A 200 is content, free or paid. A 402 or 403 is a denial and costs $0. Paid 200 responses may carry X-AISA-Charged and X-AISA-Charge-Amount.", "code": { "lang": "bash", "text": "curl -s -i 'https://PUBLISHER-SITE/research/report.data.json' \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n -A 'MyBot/1.0 (+https://example.com/bot)'" }, "sources": [ "S1", "S2" ] }, { "title": "Branch on the error field", "body": "Denial bodies are JSON, and the error field says why. Check it first, then choose the action. 402 means a payment-side fix: invalid_token, insufficient_balance, or a price negotiation state. 403 access_blocked means the publisher withholds the resource from crawlers, and no payment unlocks it. Keep the branching in one place so every URL in a crawl is handled the same way.", "code": { "lang": "python", "text": "import requests\n\nurl = 'https://PUBLISHER-SITE/research/report.data.json'\nr = requests.get(url, headers={\n 'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN',\n 'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'})\n\nif r.status_code in (402, 403):\n body = r.json()\n err = body.get('error')\n if err == 'invalid_token':\n print('fix or register the token, retry same URL')\n elif err == 'insufficient_balance':\n print('shortfall:', body.get('crawler', {}).get('shortfall'))\n elif err == 'access_blocked':\n print('skip this URL, check pricing.json')\n else:\n print('quote:', body.get('price'), body.get('currency'))" }, "sources": [ "S2" ] }, { "title": "Resolve 402 invalid_token and insufficient_balance", "body": "For invalid_token, register or fix the header, then retry the same URL; the guidance.steps list walks through it. For insufficient_balance, crawler.shortfall says how much more is needed. The agent must not fund itself: create a Stripe checkout session (minimum 0.50), give the checkout_url to the user, and wait. Fetch the real balance before reporting any figure, using the same crawler token.", "code": { "lang": "bash", "text": "# Real balance, same crawler token you crawl with\ncurl -s 'https://cdn.aisa.one/api/agent/balance' \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'\n\n# Top-up link for the user (session token from login)\ncurl -s -X POST 'https://cdn.aisa.one/api/agent/topup/create-session' \\\n -H 'Content-Type: application/json' \\\n -H 'Authorization: Bearer SESSION_TOKEN' \\\n -d '{ 'amount': 5.00 }'" }, "sources": [ "S1", "S2", "S3" ] }, { "title": "Follow the guidance block verbatim", "body": "Every denial can carry a machine-readable guidance block with onboarding, docs, llmsTxt and skill links, a registerApi entry, numbered steps, a signupBonus notice when a trial credit is active, and agentAdvice. Follow guidance.steps as written, because the same system judges your retry. If guidance.registerApi names a different AISA host, prefer that host. Narrate the steps to your user as agentAdvice suggests.", "code": { "lang": "text", "text": "guidance.onboarding onboarding guide link\nguidance.docs documentation link\nguidance.llmsTxt llms.txt link\nguidance.skill skill.md link\nguidance.registerApi register endpoint for this site\nguidance.steps numbered steps, follow in order\nguidance.signupBonus present only when a trial credit is active\nguidance.agentAdvice what to narrate to the user" }, "sources": [ "S2" ] }, { "title": "Pivot on 403 access_blocked", "body": "A 403 access_blocked is a publisher policy choice, not a payment problem. Do not retry that URL and do not top up for it. Do not stop the task either: fetch the free /pricing.json, which lists the purchasable pay-mode directories, and continue with those. Skipping a blocked URL and buying pay-mode content is the intended use of the API, not a bypass.", "code": { "lang": "bash", "text": "curl -s 'https://PUBLISHER-SITE/pricing.json' \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'" }, "sources": [ "S2", "S3" ] }, { "title": "Tell the user what happened", "body": "Narrate in plain language. Say whether content was free or paid, the exact amount charged, and the remaining balance fetched live, not remembered. For insufficient_balance, state the shortfall and share the checkout link. For access_blocked, explain that the publisher chose to withhold that resource from crawlers, that payment cannot change it, and what you are fetching instead. Never print the full token; show a prefix like aisa_crawler_9a4d...", "code": { "lang": "text", "text": "Balance is out: you need a little more to continue. Open this link to add funds: CHECKOUT_URL. After you pay, I will retry.\n\nThe publisher does not allow crawlers on this page, and payment cannot change that. I will use its purchasable sections instead." }, "sources": [ "S2", "S3" ] } ], "reference": { "title": "AISA denial codes and next actions", "columns": [ "Status and error", "Meaning", "Next action", "Tell the user" ], "rows": [ [ "402 invalid_token", "No, unknown or inactive token", "Register or fix the header, retry the same URL", "Account needs a valid token; mention any trial credit" ], [ "402 insufficient_balance", "Balance too low; crawler.shortfall gives the gap", "Create checkout session, hand over checkout_url, retry after payment", "The shortfall and the payment link" ], [ "402 price_required", "Price negotiation state, rarely seen", "Read the quote in price and currency", "The quoted price" ], [ "402 price_mismatch", "Price negotiation state, rarely seen", "Read the quote in price and currency", "The quoted price" ], [ "402 price_too_high", "Price negotiation state, rarely seen", "Read the quote in price and currency", "The quoted price" ], [ "403 access_blocked", "Publisher rule withholds this from crawlers", "Do not retry or top up; pivot to pay-mode directories in /pricing.json", "It is a publisher policy choice and payment will not unlock it" ] ], "sources": [ "S2" ] }, "pitfalls": [ { "text": "Sending the crawler token in Authorization: Bearer makes the request unauthenticated, so you get 402 invalid_token. Use X-Crawler-Token only.", "source": "S2" }, { "text": "Never call POST /api/agent/balance/add to top yourself up. It is for human operators seeding test accounts; real funding goes through the Stripe checkout the user pays.", "source": "S3" }, { "text": "Quoting a balance from memory goes stale after the first spend. Call GET /api/agent/balance before reporting any figure, and reuse one account throughout.", "source": "S2" }, { "text": "Do not give up on a whole task because one URL returned 403 access_blocked. Skip it and move on to the site's pay-mode directories.", "source": "S3" } ], "takeaways": [ "Read the error field first: invalid_token and insufficient_balance are fixable with payment-side action, access_blocked is not.", "Denials never charge. Follow guidance.steps, and prefer the host in guidance.registerApi.", "Always give the user the free or paid status, the amount charged, and a freshly fetched balance." ], "faq": [ { "q": "Does a 402 or 403 response cost anything?", "a": "No. Denials never charge, and price discovery through rule.price in a 402 is free. A charge happens only on a granted fetch of a priced resource, at most once per request." }, { "q": "Can I pay to unlock a 403 access_blocked page?", "a": "No. The publisher forbids crawler access to that resource, and payment does not unlock it. Skip the URL and use the purchasable pay-mode directories listed in /pricing.json." }, { "q": "Which header carries the crawler token?", "a": "X-Crawler-Token, per the Open Verify specification. It replaces the older X-AISA-Crawler-Token name. Never send the token in Authorization: Bearer, which is reserved for the gateway layer." }, { "q": "What should an agent do when the balance runs out?", "a": "Stop and give the user a Stripe checkout_url from POST /api/agent/topup/create-session; the minimum top-up is 0.50. State the shortfall, wait for payment, then retry the same URL." } ], "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html", "domain": "esa.aisa.one" }, { "title": "AISA Agent Integration Guide — API reference", "url": "https://esa.aisa.one/agent/docs.html", "domain": "esa.aisa.one" }, { "title": "AISA agent skill (SKILL.md)", "url": "https://esa.aisa.one/agent/skill.md", "domain": "esa.aisa.one" }, { "title": "Live capture: anonymous request to a paid AgentFlare file", "url": "https://agentflare.org/markets/index.data.json", "domain": "agentflare.org" } ] }, "analysis_md": "Every AISA denial arrives as self-describing JSON: read the error field, act on the guidance block, and report plainly to the user. This guide covers each 402 and 403 code, the next action, and the wording.\n\nThis guide is for developers building agents or crawlers that fetch paid publisher content over the AISA pay-per-crawl network. You will be able to branch on every denial code, follow the guidance block that comes with each denial, and know when to retry, when to hand the user a payment link, and when to skip a URL and move on. The guide ends with a reference table of error codes and a template for what the agent should tell its user, so the conversation stays accurate and calm.\n\n## Before you start\n\n- A registered AISA crawler account with its crawler token\n- A paid publisher URL behind an AISA-integrated gateway\n- curl or Python with the requests library\n\n## Step by step\n\n### 1. Send the token header and read the status\n\nRequest the URL with your crawler token and your genuine User-Agent. The header is X-Crawler-Token; it replaces the earlier name X-AISA-Crawler-Token that older AISA pages still show. Never put the token in Authorization: Bearer, which is reserved for the gateway layer. A 200 is content, free or paid. A 402 or 403 is a denial and costs $0. Paid 200 responses may carry X-AISA-Charged and X-AISA-Charge-Amount.\n\n```bash\ncurl -s -i 'https://PUBLISHER-SITE/research/report.data.json' \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n -A 'MyBot/1.0 (+https://example.com/bot)'\n```\n\n### 2. Branch on the error field\n\nDenial bodies are JSON, and the error field says why. Check it first, then choose the action. 402 means a payment-side fix: invalid_token, insufficient_balance, or a price negotiation state. 403 access_blocked means the publisher withholds the resource from crawlers, and no payment unlocks it. Keep the branching in one place so every URL in a crawl is handled the same way.\n\n```python\nimport requests\n\nurl = 'https://PUBLISHER-SITE/research/report.data.json'\nr = requests.get(url, headers={\n 'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN',\n 'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'})\n\nif r.status_code in (402, 403):\n body = r.json()\n err = body.get('error')\n if err == 'invalid_token':\n print('fix or register the token, retry same URL')\n elif err == 'insufficient_balance':\n print('shortfall:', body.get('crawler', {}).get('shortfall'))\n elif err == 'access_blocked':\n print('skip this URL, check pricing.json')\n else:\n print('quote:', body.get('price'), body.get('currency'))\n```\n\n### 3. Resolve 402 invalid_token and insufficient_balance\n\nFor invalid_token, register or fix the header, then retry the same URL; the guidance.steps list walks through it. For insufficient_balance, crawler.shortfall says how much more is needed. The agent must not fund itself: create a Stripe checkout session (minimum 0.50), give the checkout_url to the user, and wait. Fetch the real balance before reporting any figure, using the same crawler token.\n\n```bash\n# Real balance, same crawler token you crawl with\ncurl -s 'https://cdn.aisa.one/api/agent/balance' \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'\n\n# Top-up link for the user (session token from login)\ncurl -s -X POST 'https://cdn.aisa.one/api/agent/topup/create-session' \\\n -H 'Content-Type: application/json' \\\n -H 'Authorization: Bearer SESSION_TOKEN' \\\n -d '{ 'amount': 5.00 }'\n```\n\n### 4. Follow the guidance block verbatim\n\nEvery denial can carry a machine-readable guidance block with onboarding, docs, llmsTxt and skill links, a registerApi entry, numbered steps, a signupBonus notice when a trial credit is active, and agentAdvice. Follow guidance.steps as written, because the same system judges your retry. If guidance.registerApi names a different AISA host, prefer that host. Narrate the steps to your user as agentAdvice suggests.\n\n```text\nguidance.onboarding onboarding guide link\nguidance.docs documentation link\nguidance.llmsTxt llms.txt link\nguidance.skill skill.md link\nguidance.registerApi register endpoint for this site\nguidance.steps numbered steps, follow in order\nguidance.signupBonus present only when a trial credit is active\nguidance.agentAdvice what to narrate to the user\n```\n\n### 5. Pivot on 403 access_blocked\n\nA 403 access_blocked is a publisher policy choice, not a payment problem. Do not retry that URL and do not top up for it. Do not stop the task either: fetch the free /pricing.json, which lists the purchasable pay-mode directories, and continue with those. Skipping a blocked URL and buying pay-mode content is the intended use of the API, not a bypass.\n\n```bash\ncurl -s 'https://PUBLISHER-SITE/pricing.json' \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'\n```\n\n### 6. Tell the user what happened\n\nNarrate in plain language. Say whether content was free or paid, the exact amount charged, and the remaining balance fetched live, not remembered. For insufficient_balance, state the shortfall and share the checkout link. For access_blocked, explain that the publisher chose to withhold that resource from crawlers, that payment cannot change it, and what you are fetching instead. Never print the full token; show a prefix like aisa_crawler_9a4d...\n\n```text\nBalance is out: you need a little more to continue. Open this link to add funds: CHECKOUT_URL. After you pay, I will retry.\n\nThe publisher does not allow crawlers on this page, and payment cannot change that. I will use its purchasable sections instead.\n```\n\n## AISA denial codes and next actions\n\n| Status and error | Meaning | Next action | Tell the user |\n| --- | --- | --- | --- |\n| 402 invalid_token | No, unknown or inactive token | Register or fix the header, retry the same URL | Account needs a valid token; mention any trial credit |\n| 402 insufficient_balance | Balance too low; crawler.shortfall gives the gap | Create checkout session, hand over checkout_url, retry after payment | The shortfall and the payment link |\n| 402 price_required | Price negotiation state, rarely seen | Read the quote in price and currency | The quoted price |\n| 402 price_mismatch | Price negotiation state, rarely seen | Read the quote in price and currency | The quoted price |\n| 402 price_too_high | Price negotiation state, rarely seen | Read the quote in price and currency | The quoted price |\n| 403 access_blocked | Publisher rule withholds this from crawlers | Do not retry or top up; pivot to pay-mode directories in /pricing.json | It is a publisher policy choice and payment will not unlock it |\n\n## Common pitfalls\n\n- Sending the crawler token in Authorization: Bearer makes the request unauthenticated, so you get 402 invalid_token. Use X-Crawler-Token only.\n- Never call POST /api/agent/balance/add to top yourself up. It is for human operators seeding test accounts; real funding goes through the Stripe checkout the user pays.\n- Quoting a balance from memory goes stale after the first spend. Call GET /api/agent/balance before reporting any figure, and reuse one account throughout.\n- Do not give up on a whole task because one URL returned 403 access_blocked. Skip it and move on to the site's pay-mode directories.\n\n## Key takeaways\n\n- Read the error field first: invalid_token and insufficient_balance are fixable with payment-side action, access_blocked is not.\n- Denials never charge. Follow guidance.steps, and prefer the host in guidance.registerApi.\n- Always give the user the free or paid status, the amount charged, and a freshly fetched balance.\n\n## FAQ\n\n**Does a 402 or 403 response cost anything?**\nNo. Denials never charge, and price discovery through rule.price in a 402 is free. A charge happens only on a granted fetch of a priced resource, at most once per request.\n\n**Can I pay to unlock a 403 access_blocked page?**\nNo. The publisher forbids crawler access to that resource, and payment does not unlock it. Skip the URL and use the purchasable pay-mode directories listed in /pricing.json.\n\n**Which header carries the crawler token?**\nX-Crawler-Token, per the Open Verify specification. It replaces the older X-AISA-Crawler-Token name. Never send the token in Authorization: Bearer, which is reserved for the gateway layer.\n\n**What should an agent do when the balance runs out?**\nStop and give the user a Stripe checkout_url from POST /api/agent/topup/create-session; the minimum top-up is 0.50. State the shortfall, wait for payment, then retry the same URL.\n", "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html" }, { "title": "AISA Agent Integration Guide — API reference", "url": "https://esa.aisa.one/agent/docs.html" }, { "title": "AISA agent skill (SKILL.md)", "url": "https://esa.aisa.one/agent/skill.md" }, { "title": "Live capture: anonymous request to a paid AgentFlare file", "url": "https://agentflare.org/markets/index.data.json" } ], "related": [ { "name": "Getting Started with HTTP 402 Pay-Per-Crawl", "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html" }, { "name": "Crawler Tokens & Auto-Pay: A How-To", "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html" }, { "name": "Budgeting a Paid Crawl: Discover, Survey, Then Buy", "url": "https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html" }, { "name": "Building a Pay-Per-Crawl Client for Your Agent", "url": "https://agentflare.org/guides/building-a-pay-per-crawl-client-for-your-agent.html" }, { "name": "Pricing Content by Directory: Best Practices", "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html" }, { "name": "robots.txt, llms.txt & Sitemaps for AI Crawlers", "url": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html" } ], "pricing": { "price_usd": 0, "method": "402", "endpoint": "https://esa.aisa.one/api/v1/access/verify", "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header", "onboarding": "https://esa.aisa.one/agent/guide.html" }, "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)" }