{ "@context": "https://agentflare.org/schema", "type": "Article", "tier": "L2-full", "title": "Budgeting a Paid Crawl: Discover, Survey, Then Buy", "description": "Crawl a whole AgentFlare-style site on a fixed budget by reading free discovery files first, choosing from free teasers, then buying only chosen deep files.…", "canonical": "https://agentflare.org/guides/budgeting-a-paid-crawl-discover-survey-then-buy.html", "category": "guides", "updated": "2026-10-16", "generated_at": "2026-10-16T00:37:01.652Z", "facts": [ { "label": "Track", "value": "For agent developers" }, { "label": "Level", "value": "Intermediate" }, { "label": "Access", "value": "Free / open" }, { "label": "Updated", "value": "2026-10-16" } ], "data": { "topic": "Budgeting a Paid Crawl: Discover, Survey, Then Buy", "track": "agent", "access": "free", "generated_at": "2026-10-16T00:37:01.652Z", "difficulty": "Intermediate", "tldr": "Crawl a whole AgentFlare-style site on a fixed budget by reading free discovery files first, choosing from free teasers, then buying only chosen deep files. Track spend as you go and stop at the first insufficient_balance.", "intro": "This guide is for developers building agents that crawl a publisher site fronted by AISA pay-per-crawl. You will fetch the free discovery files, read the per-directory price map, use free .md teasers to decide what is worth paying for, and buy only those deep files. You will keep a running total, stop cleanly when the balance runs out, and skip directories the publisher blocks, so the crawl never spends more than you planned and never retries a forbidden URL.", "prerequisites": [ "A registered AISA crawler account and its crawler token", "Python 3 and curl installed", "A spending cap you are willing to use for this crawl" ], "steps": [ { "title": "Fetch the free discovery files", "body": "Start with the files that cost nothing and need no token: robots.txt, llms.txt, sitemap.xml and pricing.json. The robots.txt on agentflare.org allows crawlers and points to the sitemap. The sitemap lists the URLs, llms.txt maps the content by section, and pricing.json declares what each directory costs. Read all four before you spend anything.", "code": { "lang": "bash", "text": "curl -s https://agentflare.org/robots.txt\ncurl -s https://agentflare.org/llms.txt\ncurl -s https://agentflare.org/sitemap.xml\ncurl -s https://agentflare.org/pricing.json" }, "sources": [ "S3", "S4", "S5" ] }, { "title": "Build your cost plan from pricing.json", "body": "Parse pricing.json and sort every rule by its mode. Rules with mode pay are purchasable and carry a price in USD. Rules with mode allow are free. Rules with mode block are closed to crawlers, and payment does not open them. On AgentFlare the paid files are *.data.json and *.full.md, while index pages, .html articles and .md teasers are free. Decide your directories now, before any request is billed.", "code": { "lang": "python", "text": "import json, urllib.request\n\np = json.load(urllib.request.urlopen('https://agentflare.org/pricing.json'))\nprint(p['directory_pricing_usd'])\nfor rule in p['rules']:\n print(rule['mode'], rule['price'], rule['urlPattern'])" }, "sources": [ "S4", "S2" ] }, { "title": "Survey the free tier to choose", "body": "Every article has a free .html page and a free .md teaser. Use them only to decide what is worth buying, then build a short shopping list of the .full.md or .data.json URLs you actually need. Teasers cost nothing, so read as many as you like. Replace ARTICLE with a slug taken from the sitemap, and keep the list inside the directories your plan marked as pay.", "code": { "lang": "bash", "text": "curl -s https://agentflare.org/research/ARTICLE.md\ncurl -s https://agentflare.org/news/ARTICLE.md" }, "sources": [ "S2", "S3", "S4" ] }, { "title": "Buy selected files with a running total", "body": "Send the crawler token in the X-Crawler-Token header; it replaces the earlier name X-AISA-Crawler-Token still shown on older pages. Add your genuine User-Agent. A paid 200 may carry X-AISA-Charge-Amount, so add it to your running total after each purchase. Stop at your cap. If a 402 body says insufficient_balance, stop the whole crawl and report crawler.shortfall. Any other denial skips that URL only.", "code": { "lang": "python", "text": "import json, urllib.request, urllib.error\n\nTOKEN = 'YOUR_CRAWLER_TOKEN'\nBUDGET = 0.10\nPRICE = 0.02\nspent = 0.0\nurls = ['https://agentflare.org/research/ARTICLE.full.md']\nfor url in urls:\n if spent >= BUDGET:\n break\n req = urllib.request.Request(url, headers={\n 'X-Crawler-Token': TOKEN,\n 'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'})\n try:\n r = urllib.request.urlopen(req)\n spent += float(r.headers.get('X-AISA-Charge-Amount') or PRICE)\n print('bought', url, 'running total', spent)\n except urllib.error.HTTPError as e:\n body = json.loads(e.read())\n if e.code == 402 and body.get('error') == 'insufficient_balance':\n print('stopping, shortfall', body['crawler']['shortfall'])\n break\n print('skipping', url, e.code, body.get('error'))" }, "sources": [ "S1", "S2", "S4" ] }, { "title": "Skip blocked paths and verify the balance", "body": "A 403 with error access_blocked is the publisher's policy, so do not retry it and do not top up for it. On AgentFlare the /members/ directory has mode block, so leave it out of your list and move on to pay-mode directories. Before reporting any balance, fetch it with your crawler token rather than quoting it from memory. When funds run out, hand the Stripe checkout_url to your user. Never call POST /api/agent/balance/add.", "code": { "lang": "bash", "text": "curl -s https://cdn.aisa.one/api/agent/balance \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'" }, "sources": [ "S2", "S3", "S4" ] } ], "reference": { "title": "AgentFlare directory modes and prices", "columns": [ "Directory", "Mode", "Price USD per request" ], "rows": [ [ "/news/", "pay", "0.005" ], [ "/social/", "pay", "0.005" ], [ "/markets/", "pay", "0.01" ], [ "/predictions/ and /crypto/", "pay", "0.01" ], [ "/research/", "pay", "0.02" ], [ "/scholar/", "pay", "0.02" ], [ "/guides/", "allow", "0" ], [ "/members/", "block", "0" ] ], "sources": [ "S4" ] }, "pitfalls": [ { "text": "Each granted fetch of a priced resource deducts its price once. Fetch the same URL twice and you pay twice, so keep a list of what you already bought.", "source": "S2" }, { "text": "Never send the crawler token in Authorization: Bearer. That header is reserved, and the request is treated as unauthenticated with a 402 invalid_token.", "source": "S1" }, { "text": "A 403 access_blocked is a publisher policy choice. Payment does not unlock it, so do not retry or top up; pivot to purchasable directories instead.", "source": "S2" }, { "text": "Do not call POST /api/agent/balance/add to refill mid-crawl. It is for human operators seeding test accounts; agents hand the checkout_url to their user.", "source": "S3" } ], "takeaways": [ "Discovery files and .md teasers are free, so read them first and buy only the deep files you chose.", "Add up charges as you go, and stop the whole crawl on insufficient_balance.", "Skip 403 access_blocked URLs and the /members/ directory without retrying; payment never unlocks them." ], "faq": [ { "q": "Do denied requests cost anything?", "a": "No. A 402 or 403 denial never charges your balance. A 402 body also shows the price in rule.price, so you can learn what a directory costs without paying." }, { "q": "How do I know what a request cost?", "a": "A paid 200 may include X-AISA-Charged: true and X-AISA-Charge-Amount headers. To confirm, call GET /api/agent/balance with your crawler token and use the number it returns." }, { "q": "What should my crawler do on insufficient_balance?", "a": "Stop the crawl. Report crawler.shortfall to your user and give them the Stripe checkout_url from the top-up session. The agent does not fund itself; after the user pays, resume." }, { "q": "Which header carries the crawler token?", "a": "X-Crawler-Token. It replaces the older X-AISA-Crawler-Token name that some AISA pages still show. Never use Authorization: Bearer for the crawler token." } ], "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html", "domain": "esa.aisa.one" }, { "title": "AISA Agent Integration Guide — API reference", "url": "https://esa.aisa.one/agent/docs.html", "domain": "esa.aisa.one" }, { "title": "AISA agent skill (SKILL.md)", "url": "https://esa.aisa.one/agent/skill.md", "domain": "esa.aisa.one" }, { "title": "AgentFlare pricing.json (directory price map)", "url": "https://agentflare.org/pricing.json", "domain": "agentflare.org" }, { "title": "AgentFlare robots.txt", "url": "https://agentflare.org/robots.txt", "domain": "agentflare.org" } ] }, "analysis_md": "Crawl a whole AgentFlare-style site on a fixed budget by reading free discovery files first, choosing from free teasers, then buying only chosen deep files. Track spend as you go and stop at the first insufficient_balance.\n\nThis guide is for developers building agents that crawl a publisher site fronted by AISA pay-per-crawl. You will fetch the free discovery files, read the per-directory price map, use free .md teasers to decide what is worth paying for, and buy only those deep files. You will keep a running total, stop cleanly when the balance runs out, and skip directories the publisher blocks, so the crawl never spends more than you planned and never retries a forbidden URL.\n\n## Before you start\n\n- A registered AISA crawler account and its crawler token\n- Python 3 and curl installed\n- A spending cap you are willing to use for this crawl\n\n## Step by step\n\n### 1. Fetch the free discovery files\n\nStart with the files that cost nothing and need no token: robots.txt, llms.txt, sitemap.xml and pricing.json. The robots.txt on agentflare.org allows crawlers and points to the sitemap. The sitemap lists the URLs, llms.txt maps the content by section, and pricing.json declares what each directory costs. Read all four before you spend anything.\n\n```bash\ncurl -s https://agentflare.org/robots.txt\ncurl -s https://agentflare.org/llms.txt\ncurl -s https://agentflare.org/sitemap.xml\ncurl -s https://agentflare.org/pricing.json\n```\n\n### 2. Build your cost plan from pricing.json\n\nParse pricing.json and sort every rule by its mode. Rules with mode pay are purchasable and carry a price in USD. Rules with mode allow are free. Rules with mode block are closed to crawlers, and payment does not open them. On AgentFlare the paid files are *.data.json and *.full.md, while index pages, .html articles and .md teasers are free. Decide your directories now, before any request is billed.\n\n```python\nimport json, urllib.request\n\np = json.load(urllib.request.urlopen('https://agentflare.org/pricing.json'))\nprint(p['directory_pricing_usd'])\nfor rule in p['rules']:\n print(rule['mode'], rule['price'], rule['urlPattern'])\n```\n\n### 3. Survey the free tier to choose\n\nEvery article has a free .html page and a free .md teaser. Use them only to decide what is worth buying, then build a short shopping list of the .full.md or .data.json URLs you actually need. Teasers cost nothing, so read as many as you like. Replace ARTICLE with a slug taken from the sitemap, and keep the list inside the directories your plan marked as pay.\n\n```bash\ncurl -s https://agentflare.org/research/ARTICLE.md\ncurl -s https://agentflare.org/news/ARTICLE.md\n```\n\n### 4. Buy selected files with a running total\n\nSend the crawler token in the X-Crawler-Token header; it replaces the earlier name X-AISA-Crawler-Token still shown on older pages. Add your genuine User-Agent. A paid 200 may carry X-AISA-Charge-Amount, so add it to your running total after each purchase. Stop at your cap. If a 402 body says insufficient_balance, stop the whole crawl and report crawler.shortfall. Any other denial skips that URL only.\n\n```python\nimport json, urllib.request, urllib.error\n\nTOKEN = 'YOUR_CRAWLER_TOKEN'\nBUDGET = 0.10\nPRICE = 0.02\nspent = 0.0\nurls = ['https://agentflare.org/research/ARTICLE.full.md']\nfor url in urls:\n if spent >= BUDGET:\n break\n req = urllib.request.Request(url, headers={\n 'X-Crawler-Token': TOKEN,\n 'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'})\n try:\n r = urllib.request.urlopen(req)\n spent += float(r.headers.get('X-AISA-Charge-Amount') or PRICE)\n print('bought', url, 'running total', spent)\n except urllib.error.HTTPError as e:\n body = json.loads(e.read())\n if e.code == 402 and body.get('error') == 'insufficient_balance':\n print('stopping, shortfall', body['crawler']['shortfall'])\n break\n print('skipping', url, e.code, body.get('error'))\n```\n\n### 5. Skip blocked paths and verify the balance\n\nA 403 with error access_blocked is the publisher's policy, so do not retry it and do not top up for it. On AgentFlare the /members/ directory has mode block, so leave it out of your list and move on to pay-mode directories. Before reporting any balance, fetch it with your crawler token rather than quoting it from memory. When funds run out, hand the Stripe checkout_url to your user. Never call POST /api/agent/balance/add.\n\n```bash\ncurl -s https://cdn.aisa.one/api/agent/balance \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'\n```\n\n## AgentFlare directory modes and prices\n\n| Directory | Mode | Price USD per request |\n| --- | --- | --- |\n| /news/ | pay | 0.005 |\n| /social/ | pay | 0.005 |\n| /markets/ | pay | 0.01 |\n| /predictions/ and /crypto/ | pay | 0.01 |\n| /research/ | pay | 0.02 |\n| /scholar/ | pay | 0.02 |\n| /guides/ | allow | 0 |\n| /members/ | block | 0 |\n\n## Common pitfalls\n\n- Each granted fetch of a priced resource deducts its price once. Fetch the same URL twice and you pay twice, so keep a list of what you already bought.\n- Never send the crawler token in Authorization: Bearer. That header is reserved, and the request is treated as unauthenticated with a 402 invalid_token.\n- A 403 access_blocked is a publisher policy choice. Payment does not unlock it, so do not retry or top up; pivot to purchasable directories instead.\n- Do not call POST /api/agent/balance/add to refill mid-crawl. It is for human operators seeding test accounts; agents hand the checkout_url to their user.\n\n## Key takeaways\n\n- Discovery files and .md teasers are free, so read them first and buy only the deep files you chose.\n- Add up charges as you go, and stop the whole crawl on insufficient_balance.\n- Skip 403 access_blocked URLs and the /members/ directory without retrying; payment never unlocks them.\n\n## FAQ\n\n**Do denied requests cost anything?**\nNo. A 402 or 403 denial never charges your balance. A 402 body also shows the price in rule.price, so you can learn what a directory costs without paying.\n\n**How do I know what a request cost?**\nA paid 200 may include X-AISA-Charged: true and X-AISA-Charge-Amount headers. To confirm, call GET /api/agent/balance with your crawler token and use the number it returns.\n\n**What should my crawler do on insufficient_balance?**\nStop the crawl. Report crawler.shortfall to your user and give them the Stripe checkout_url from the top-up session. The agent does not fund itself; after the user pays, resume.\n\n**Which header carries the crawler token?**\nX-Crawler-Token. It replaces the older X-AISA-Crawler-Token name that some AISA pages still show. Never use Authorization: Bearer for the crawler token.\n", "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html" }, { "title": "AISA Agent Integration Guide — API reference", "url": "https://esa.aisa.one/agent/docs.html" }, { "title": "AISA agent skill (SKILL.md)", "url": "https://esa.aisa.one/agent/skill.md" }, { "title": "AgentFlare pricing.json (directory price map)", "url": "https://agentflare.org/pricing.json" }, { "title": "AgentFlare robots.txt", "url": "https://agentflare.org/robots.txt" } ], "related": [ { "name": "Getting Started with HTTP 402 Pay-Per-Crawl", "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html" }, { "name": "Crawler Tokens & Auto-Pay: A How-To", "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html" }, { "name": "Handling 402 and 403 Responses: An Error Reference for Agents", "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html" }, { "name": "Building a Pay-Per-Crawl Client for Your Agent", "url": "https://agentflare.org/guides/building-a-pay-per-crawl-client-for-your-agent.html" }, { "name": "Pricing Content by Directory: Best Practices", "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html" }, { "name": "robots.txt, llms.txt & Sitemaps for AI Crawlers", "url": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html" } ], "pricing": { "price_usd": 0, "method": "402", "endpoint": "https://esa.aisa.one/api/v1/access/verify", "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header", "onboarding": "https://esa.aisa.one/agent/guide.html" }, "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)" }