Budgeting a Paid Crawl: Discover, Survey, Then Buy

This guide is for developers building agents that crawl a publisher site fronted by AISA pay-per-crawl. You will fetch the free discovery files, read the per-directory price map, use free .md teasers to decide what is worth paying for, and buy only those deep files. You will keep a running total, stop cleanly when the balance runs out, and skip directories the publisher blocks, so the crawl never spends more than you planned and never retries a forbidden URL.

How the flow works

Before you start

• A registered AISA crawler account and its crawler token
• Python 3 and curl installed
• A spending cap you are willing to use for this crawl

Step by step

1. Fetch the free discovery files

   Start with the files that cost nothing and need no token: robots.txt, llms.txt, sitemap.xml and pricing.json. The robots.txt on agentflare.org allows crawlers and points to the sitemap. The sitemap lists the URLs, llms.txt maps the content by section, and pricing.json declares what each directory costs. Read all four before you spend anything.

   curl -s https://agentflare.org/robots.txt
   curl -s https://agentflare.org/llms.txt
   curl -s https://agentflare.org/sitemap.xml
   curl -s https://agentflare.org/pricing.json

   S3 · esa.aisa.oneS4 · agentflare.orgS5 · agentflare.org

2. Build your cost plan from pricing.json

   Parse pricing.json and sort every rule by its mode. Rules with mode pay are purchasable and carry a price in USD. Rules with mode allow are free. Rules with mode block are closed to crawlers, and payment does not open them. On AgentFlare the paid files are *.data.json and *.full.md, while index pages, .html articles and .md teasers are free. Decide your directories now, before any request is billed.

   import json, urllib.request
   
   p = json.load(urllib.request.urlopen('https://agentflare.org/pricing.json'))
   print(p['directory_pricing_usd'])
   for rule in p['rules']:
       print(rule['mode'], rule['price'], rule['urlPattern'])

   S4 · agentflare.orgS2 · esa.aisa.one

3. Survey the free tier to choose

   Every article has a free .html page and a free .md teaser. Use them only to decide what is worth buying, then build a short shopping list of the .full.md or .data.json URLs you actually need. Teasers cost nothing, so read as many as you like. Replace ARTICLE with a slug taken from the sitemap, and keep the list inside the directories your plan marked as pay.

   curl -s https://agentflare.org/research/ARTICLE.md
   curl -s https://agentflare.org/news/ARTICLE.md

   S2 · esa.aisa.oneS3 · esa.aisa.oneS4 · agentflare.org

4. Buy selected files with a running total

   Send the crawler token in the X-Crawler-Token header; it replaces the earlier name X-AISA-Crawler-Token still shown on older pages. Add your genuine User-Agent. A paid 200 may carry X-AISA-Charge-Amount, so add it to your running total after each purchase. Stop at your cap. If a 402 body says insufficient_balance, stop the whole crawl and report crawler.shortfall. Any other denial skips that URL only.

   import json, urllib.request, urllib.error
   
   TOKEN = 'YOUR_CRAWLER_TOKEN'
   BUDGET = 0.10
   PRICE = 0.02
   spent = 0.0
   urls = ['https://agentflare.org/research/ARTICLE.full.md']
   for url in urls:
       if spent >= BUDGET:
           break
       req = urllib.request.Request(url, headers={
           'X-Crawler-Token': TOKEN,
           'User-Agent': 'MyBot/1.0 (+https://example.com/bot)'})
       try:
           r = urllib.request.urlopen(req)
           spent += float(r.headers.get('X-AISA-Charge-Amount') or PRICE)
           print('bought', url, 'running total', spent)
       except urllib.error.HTTPError as e:
           body = json.loads(e.read())
           if e.code == 402 and body.get('error') == 'insufficient_balance':
               print('stopping, shortfall', body['crawler']['shortfall'])
               break
           print('skipping', url, e.code, body.get('error'))

   S1 · esa.aisa.oneS2 · esa.aisa.oneS4 · agentflare.org

5. Skip blocked paths and verify the balance

   A 403 with error access_blocked is the publisher's policy, so do not retry it and do not top up for it. On AgentFlare the /members/ directory has mode block, so leave it out of your list and move on to pay-mode directories. Before reporting any balance, fetch it with your crawler token rather than quoting it from memory. When funds run out, hand the Stripe checkout_url to your user. Never call POST /api/agent/balance/add.

   curl -s https://cdn.aisa.one/api/agent/balance \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN'

   S2 · esa.aisa.oneS3 · esa.aisa.oneS4 · agentflare.org

AgentFlare directory modes and prices

Common pitfalls

• Each granted fetch of a priced resource deducts its price once. Fetch the same URL twice and you pay twice, so keep a list of what you already bought.
• Never send the crawler token in Authorization: Bearer. That header is reserved, and the request is treated as unauthenticated with a 402 invalid_token.
• A 403 access_blocked is a publisher policy choice. Payment does not unlock it, so do not retry or top up; pivot to purchasable directories instead.
• Do not call POST /api/agent/balance/add to refill mid-crawl. It is for human operators seeding test accounts; agents hand the checkout_url to their user.

Key takeaways

• Discovery files and .md teasers are free, so read them first and buy only the deep files you chose.
• Add up charges as you go, and stop the whole crawl on insufficient_balance.
• Skip 403 access_blocked URLs and the /members/ directory without retrying; payment never unlocks them.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.