Using AgentFlare Research and Scholar Data

This guide is for developers and agent builders who want to pull AgentFlare research briefs and scholar digests into a pipeline. You will learn how the index files point to individual items, which fields each file type holds, what a request costs, and how to send an authenticated fetch. You will also read the pricing block that comes back inside each file. Examples use curl and standard-library Python, with YOUR_CRAWLER_TOKEN as a placeholder for your own token.

How the flow works

Before you start

• A crawler token to send in X-Crawler-Token
• curl and Python 3 installed
• Basic familiarity with JSON

Step by step

1. Locate the index and item files

   AgentFlare keeps one data file per item under /research/ and /scholar/, plus an index.data.json in each folder. Start with the index, because it lists every brief or digest together with its url. Item files follow the pattern <slug>.data.json. Free teasers (.html and .md) are open, while .data.json and .full.md are the paid deep payloads.

   https://agentflare.org/research/index.data.json
   https://agentflare.org/research/tool-use-reliability-in-llm-agents.data.json
   https://agentflare.org/scholar/index.data.json
   https://agentflare.org/scholar/web-agents-navigation-benchmarks-research-digest.data.json

   S2 · agentflare.orgS3 · agentflare.orgS4 · agentflare.org

2. Fetch a file with your crawler token

   Send the crawler identity in the X-Crawler-Token header, which replaces the older X-AISA-Crawler-Token name. Never put the token in Authorization: Bearer, since that header is reserved for the gateway layer. Each .data.json request under /research/ or /scholar/ is metered per request at 0.02 USD through the verify endpoint https://esa.aisa.one/api/v1/access/verify.

   curl -s -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     'https://agentflare.org/research/index.data.json' \
     -o research-index.json

   S1 · esa.aisa.oneS4 · agentflare.org

3. Read the research index and briefs

   The research index holds data.count, data.clusters and data.briefs[], where each entry has title, cluster, sources and url. A single brief puts its content under data: topic, cluster, generated_at, tldr, intro, concepts, how_it_works, sections, comparison, key_numbers, timeline, pitfalls, takeaways, faq and sources. The snippet below prints the main parts of one brief.

   import json, urllib.request
   
   url = 'https://agentflare.org/research/tool-use-reliability-in-llm-agents.data.json'
   req = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})
   doc = json.load(urllib.request.urlopen(req))
   d = doc['data']
   print(d['topic'], d['cluster'])
   print(d['tldr'])
   for c in d['concepts']:
       print(c['term'], c['definition'])
   for k in d['key_numbers']:
       print(k['value'], k['label'], k['source'])
   for q in d['faq']:
       print(q['q'], q['a'])

   S2 · agentflare.org

4. Read the scholar index and digests

   The scholar index lists data.digests[] with title, headline, papers, citations, monthly_arxiv and url. A digest holds topic, arxiv_query, papers[] (title, link, authors, citations, year, snippet, takeaway), recent[], monthly_arxiv[] (month, count), themes, whats_new, trend_note, open_problems, faq and total_citations. This snippet prints papers and monthly counts.

   import json, urllib.request
   
   url = 'https://agentflare.org/scholar/web-agents-navigation-benchmarks-research-digest.data.json'
   req = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})
   doc = json.load(urllib.request.urlopen(req))
   d = doc['data']
   print(d['topic'], d['total_citations'])
   for p in d['papers']:
       print(p['title'], p['year'], p['citations'], p['takeaway'])
   for m in d['monthly_arxiv']:
       print(m['month'], m['count'])
   for item in d['recent']:
       print(item['published'], item['title'], item['link'])

   S3 · agentflare.org

5. Check the envelope and pricing block

   Every file wraps its data in the same envelope: title, description, canonical, category, updated, generated_at, facts, data, analysis_md, sources and related. The pricing object carries price_usd, method, endpoint, autopay_hint and onboarding. Read it from the response to confirm the per-request cost before you loop over many items.

   import json, urllib.request
   
   url = 'https://agentflare.org/scholar/index.data.json'
   req = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})
   doc = json.load(urllib.request.urlopen(req))
   pr = doc['pricing']
   print(pr['price_usd'], pr['method'], pr['endpoint'])
   print(pr['autopay_hint'])
   print(pr['onboarding'])
   for f in doc['facts']:
       print(f)
   for r in doc['related']:
       print(r['name'], r['url'])

   S2 · agentflare.orgS3 · agentflare.orgS4 · agentflare.org

Key fields in research and scholar files

Common pitfalls

• Sending X-AISA-Crawler-Token because older AISA pages still show it. The current header name is X-Crawler-Token.
• Putting the crawler token in Authorization: Bearer. That header is reserved for the gateway layer, so use X-Crawler-Token instead.
• The facts field in both index files is typed as array of unknown, while item files give facts[].label and facts[].value as strings. Do not share one parser.
• Every .data.json and .full.md request is metered at 0.02 USD, so save index and item files locally instead of refetching.

Key takeaways

• Research and scholar .data.json and .full.md files cost 0.02 USD per request; .html and .md teasers are free.
• Start at index.data.json in each folder, then fetch item files by the url listed there.
• Authenticate with X-Crawler-Token and read the pricing object in each response.

FAQ

How this page is made. Written from 4 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.