Reading .data.json: A Field Guide for Agents

This guide is for developers and agent builders who want to pull structured data from AgentFlare instead of scraping HTML. You will learn which files are paid, how to send your crawler token, how to recognise the 402 response, and how to read the shared envelope that wraps every file. You will also see how the data block differs between Markets index files, Markets ticker files and Newswire digests, so your parser can branch on the vertical.

How the flow works

Before you start

• A registered crawler token from the AISA onboarding guide
• curl or Python with the requests library installed
• Basic familiarity with JSON and HTTP status codes

Step by step

1. Identify which files are paid

   AgentFlare splits content into two tiers. L1_free covers index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt. L2_paid covers every *.data.json and *.full.md file. Rules in pricing.json meter Markets files at 0.01 USD per request and Newswire files at 0.005 USD per request. Guides data files are free, and Members files are blocked to crawlers. Read the price map before you plan a crawl.

   curl -s https://agentflare.org/pricing.json

   S4 · agentflare.org

2. Fetch a file with your crawler token

   Send your token in the X-Crawler-Token header, which is the Open Verify name and replaces the earlier X-AISA-Crawler-Token still shown on older AISA pages. Do not put the token in Authorization: Bearer, because that header is reserved for the gateway layer. The example fetches the Markets index; the same header works for dated files such as YYYY-MM-DD.data.json and per-item slug files.

   curl -s \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     https://agentflare.org/markets/index.data.json

   S1 · esa.aisa.oneS5 · agentflare.org

3. Recognise the 402 response

   A request to a paid file without a recognised token returns HTTP 402 Payment Required with a JSON body. In the live capture that body carried the error code invalid_token, a message, and an onboarding URL pointing to https://esa.aisa.one/agent/guide.html. Check the status code before parsing the envelope, and follow the onboarding link if your token is not registered.

   import requests
   
   url = 'https://agentflare.org/markets/index.data.json'
   r = requests.get(url)
   if r.status_code == 402:
       err = r.json()
       print(err['error'])
       print(err['message'])
       print(err['onboarding'])

   S5 · agentflare.org

4. Read the shared envelope

   Every file listed in the field references starts with the same top-level fields: @context, type, tier, title, description, canonical, category, updated, generated_at, facts, data, analysis_md, sources, related, pricing and powered_by. Each facts entry has a label and a value, both strings. The pricing object carries price_usd, method, endpoint, autopay_hint and onboarding. Parse the envelope first, then branch on category.

   import requests
   
   url = 'https://agentflare.org/markets/index.data.json'
   h = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}
   doc = requests.get(url, headers=h).json()
   print(doc['title'], doc['updated'], doc['generated_at'])
   for f in doc['facts']:
       print(f['label'], f['value'])
   print(doc['pricing']['price_usd'])
   print(doc['analysis_md'][:200])

   S2 · agentflare.org

5. Parse the vertical-specific data block

   Only the data object changes. Markets index and dated files hold count, advancers, decliners, sectors and a stocks array with ticker, price_usd, day_change_pct, return_6m_pct, market_cap_usd, pe and sector. A Markets ticker file such as xom.data.json instead holds price_usd, returns_pct (m1, m3, m6), metrics and company. Newswire files hold date, headline, stories, market and items.

   import requests
   
   h = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}
   base = 'https://agentflare.org/markets/'
   
   idx = requests.get(base + 'index.data.json', headers=h).json()
   for s in idx['data']['stocks']:
       print(s['ticker'], s['price_usd'], s['day_change_pct'], s['sector'])
   
   xom = requests.get(base + 'xom.data.json', headers=h).json()
   print(xom['data']['returns_pct']['m6'])
   print(xom['data']['metrics']['price_to_earnings_ratio'])
   print(xom['data']['company']['industry'])

   S2 · agentflare.orgS3 · agentflare.org

Envelope and data block fields

Common pitfalls

• Sending the token as Authorization: Bearer fails by design; that header is reserved for the gateway layer. Use X-Crawler-Token, not the older X-AISA-Crawler-Token.
• Do not reuse one data parser across verticals. Markets index files use data.stocks, ticker files use data.metrics, and Newswire files use data.stories.
• Newswire market entries differ by file: the AI digest lists d1 and d30, while the crypto digest lists d1 and d7 plus symbol and name.
• Each paid request is metered per crawl, so an anonymous or failed call is not a free probe. Check for HTTP 402 and error invalid_token before retrying in a loop.

Key takeaways

• Every .data.json shares one envelope; read title, updated, facts, analysis_md, sources and pricing before touching data.
• Send X-Crawler-Token on every request to a paid file; never use Authorization: Bearer.
• Branch on the vertical: stocks for Markets index, metrics for ticker files, stories for Newswire digests.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.