{
  "@context": "https://agentflare.org/schema",
  "type": "Article",
  "tier": "L2-full",
  "title": "Using AgentFlare Research and Scholar Data",
  "description": "Research briefs and scholar digests are .data.json files that cost 0.02 USD per request each. This guide lists their fields and shows a fetch with the…",
  "canonical": "https://agentflare.org/guides/using-agentflare-research-and-scholar-data.html",
  "category": "guides",
  "updated": "2026-10-15",
  "generated_at": "2026-10-15T00:37:01.844Z",
  "facts": [
    {
      "label": "Track",
      "value": "Using AgentFlare data"
    },
    {
      "label": "Level",
      "value": "Beginner"
    },
    {
      "label": "Access",
      "value": "Free / open"
    },
    {
      "label": "Updated",
      "value": "2026-10-15"
    }
  ],
  "data": {
    "topic": "Using AgentFlare Research and Scholar Data",
    "track": "data",
    "access": "free",
    "generated_at": "2026-10-15T00:37:01.844Z",
    "difficulty": "Beginner",
    "tldr": "Research briefs and scholar digests are .data.json files that cost 0.02 USD per request each. This guide lists their fields and shows a fetch with the X-Crawler-Token header.",
    "intro": "This guide is for developers and agent builders who want to pull AgentFlare research briefs and scholar digests into a pipeline. You will learn how the index files point to individual items, which fields each file type holds, what a request costs, and how to send an authenticated fetch. You will also read the pricing block that comes back inside each file. Examples use curl and standard-library Python, with YOUR_CRAWLER_TOKEN as a placeholder for your own token.",
    "prerequisites": [
      "A crawler token to send in X-Crawler-Token",
      "curl and Python 3 installed",
      "Basic familiarity with JSON"
    ],
    "steps": [
      {
        "title": "Locate the index and item files",
        "body": "AgentFlare keeps one data file per item under /research/ and /scholar/, plus an index.data.json in each folder. Start with the index, because it lists every brief or digest together with its url. Item files follow the pattern <slug>.data.json. Free teasers (.html and .md) are open, while .data.json and .full.md are the paid deep payloads.",
        "code": {
          "lang": "text",
          "text": "https://agentflare.org/research/index.data.json\nhttps://agentflare.org/research/tool-use-reliability-in-llm-agents.data.json\nhttps://agentflare.org/scholar/index.data.json\nhttps://agentflare.org/scholar/web-agents-navigation-benchmarks-research-digest.data.json"
        },
        "sources": [
          "S2",
          "S3",
          "S4"
        ]
      },
      {
        "title": "Fetch a file with your crawler token",
        "body": "Send the crawler identity in the X-Crawler-Token header, which replaces the older X-AISA-Crawler-Token name. Never put the token in Authorization: Bearer, since that header is reserved for the gateway layer. Each .data.json request under /research/ or /scholar/ is metered per request at 0.02 USD through the verify endpoint https://esa.aisa.one/api/v1/access/verify.",
        "code": {
          "lang": "bash",
          "text": "curl -s -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n  'https://agentflare.org/research/index.data.json' \\\n  -o research-index.json"
        },
        "sources": [
          "S1",
          "S4"
        ]
      },
      {
        "title": "Read the research index and briefs",
        "body": "The research index holds data.count, data.clusters and data.briefs[], where each entry has title, cluster, sources and url. A single brief puts its content under data: topic, cluster, generated_at, tldr, intro, concepts, how_it_works, sections, comparison, key_numbers, timeline, pitfalls, takeaways, faq and sources. The snippet below prints the main parts of one brief.",
        "code": {
          "lang": "python",
          "text": "import json, urllib.request\n\nurl = 'https://agentflare.org/research/tool-use-reliability-in-llm-agents.data.json'\nreq = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\ndoc = json.load(urllib.request.urlopen(req))\nd = doc['data']\nprint(d['topic'], d['cluster'])\nprint(d['tldr'])\nfor c in d['concepts']:\n    print(c['term'], c['definition'])\nfor k in d['key_numbers']:\n    print(k['value'], k['label'], k['source'])\nfor q in d['faq']:\n    print(q['q'], q['a'])"
        },
        "sources": [
          "S2"
        ]
      },
      {
        "title": "Read the scholar index and digests",
        "body": "The scholar index lists data.digests[] with title, headline, papers, citations, monthly_arxiv and url. A digest holds topic, arxiv_query, papers[] (title, link, authors, citations, year, snippet, takeaway), recent[], monthly_arxiv[] (month, count), themes, whats_new, trend_note, open_problems, faq and total_citations. This snippet prints papers and monthly counts.",
        "code": {
          "lang": "python",
          "text": "import json, urllib.request\n\nurl = 'https://agentflare.org/scholar/web-agents-navigation-benchmarks-research-digest.data.json'\nreq = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\ndoc = json.load(urllib.request.urlopen(req))\nd = doc['data']\nprint(d['topic'], d['total_citations'])\nfor p in d['papers']:\n    print(p['title'], p['year'], p['citations'], p['takeaway'])\nfor m in d['monthly_arxiv']:\n    print(m['month'], m['count'])\nfor item in d['recent']:\n    print(item['published'], item['title'], item['link'])"
        },
        "sources": [
          "S3"
        ]
      },
      {
        "title": "Check the envelope and pricing block",
        "body": "Every file wraps its data in the same envelope: title, description, canonical, category, updated, generated_at, facts, data, analysis_md, sources and related. The pricing object carries price_usd, method, endpoint, autopay_hint and onboarding. Read it from the response to confirm the per-request cost before you loop over many items.",
        "code": {
          "lang": "python",
          "text": "import json, urllib.request\n\nurl = 'https://agentflare.org/scholar/index.data.json'\nreq = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\ndoc = json.load(urllib.request.urlopen(req))\npr = doc['pricing']\nprint(pr['price_usd'], pr['method'], pr['endpoint'])\nprint(pr['autopay_hint'])\nprint(pr['onboarding'])\nfor f in doc['facts']:\n    print(f)\nfor r in doc['related']:\n    print(r['name'], r['url'])"
        },
        "sources": [
          "S2",
          "S3",
          "S4"
        ]
      }
    ],
    "reference": {
      "title": "Key fields in research and scholar files",
      "columns": [
        "Field",
        "File",
        "Type"
      ],
      "rows": [
        [
          "data.briefs[].sources",
          "research index",
          "number"
        ],
        [
          "data.clusters",
          "research index",
          "array of string"
        ],
        [
          "data.concepts[].term and definition",
          "research brief",
          "string"
        ],
        [
          "data.key_numbers[].value, label, source",
          "research brief",
          "string"
        ],
        [
          "data.digests[].citations",
          "scholar index",
          "number"
        ],
        [
          "data.papers[].citations",
          "scholar digest",
          "number"
        ],
        [
          "data.monthly_arxiv[].count",
          "scholar digest",
          "number"
        ],
        [
          "data.total_citations",
          "scholar digest",
          "number"
        ]
      ],
      "sources": [
        "S2",
        "S3"
      ]
    },
    "pitfalls": [
      {
        "text": "Sending X-AISA-Crawler-Token because older AISA pages still show it. The current header name is X-Crawler-Token.",
        "source": "S1"
      },
      {
        "text": "Putting the crawler token in Authorization: Bearer. That header is reserved for the gateway layer, so use X-Crawler-Token instead.",
        "source": "S1"
      },
      {
        "text": "The facts field in both index files is typed as array of unknown, while item files give facts[].label and facts[].value as strings. Do not share one parser.",
        "source": "S2"
      },
      {
        "text": "Every .data.json and .full.md request is metered at 0.02 USD, so save index and item files locally instead of refetching.",
        "source": "S4"
      }
    ],
    "takeaways": [
      "Research and scholar .data.json and .full.md files cost 0.02 USD per request; .html and .md teasers are free.",
      "Start at index.data.json in each folder, then fetch item files by the url listed there.",
      "Authenticate with X-Crawler-Token and read the pricing object in each response."
    ],
    "faq": [
      {
        "q": "How much does an AgentFlare research or scholar request cost?",
        "a": "Both the research and scholar directories are priced at 0.02 USD per request for .data.json and .full.md files. The billingType in the rules is per-request."
      },
      {
        "q": "Which header carries the crawler token?",
        "a": "Use X-Crawler-Token. It replaces the earlier X-AISA-Crawler-Token name. Do not send the token in Authorization: Bearer, which is reserved for the gateway layer."
      },
      {
        "q": "What is the difference between the research and scholar datasets?",
        "a": "Research briefs hold topic, concepts, how_it_works, sections, comparison, key_numbers, timeline and faq. Scholar digests hold papers with citations, recent arXiv entries, monthly_arxiv counts, themes and total_citations."
      },
      {
        "q": "Are the free pages and the data files the same content?",
        "a": "No. Index pages, .html articles and .md teasers are free. The .data.json and .full.md files are the paid deep payloads with complete datasets and full analysis."
      }
    ],
    "sources": [
      {
        "title": "Header name: X-Crawler-Token",
        "url": "https://esa.aisa.one/cdn/open-verify.html",
        "domain": "esa.aisa.one"
      },
      {
        "title": "AgentFlare Research .data.json field reference",
        "url": "https://agentflare.org/research/index.html",
        "domain": "agentflare.org"
      },
      {
        "title": "AgentFlare Scholar .data.json field reference",
        "url": "https://agentflare.org/scholar/index.html",
        "domain": "agentflare.org"
      },
      {
        "title": "AgentFlare pricing.json (directory price map)",
        "url": "https://agentflare.org/pricing.json",
        "domain": "agentflare.org"
      }
    ]
  },
  "analysis_md": "Research briefs and scholar digests are .data.json files that cost 0.02 USD per request each. This guide lists their fields and shows a fetch with the X-Crawler-Token header.\n\nThis guide is for developers and agent builders who want to pull AgentFlare research briefs and scholar digests into a pipeline. You will learn how the index files point to individual items, which fields each file type holds, what a request costs, and how to send an authenticated fetch. You will also read the pricing block that comes back inside each file. Examples use curl and standard-library Python, with YOUR_CRAWLER_TOKEN as a placeholder for your own token.\n\n## Before you start\n\n- A crawler token to send in X-Crawler-Token\n- curl and Python 3 installed\n- Basic familiarity with JSON\n\n## Step by step\n\n### 1. Locate the index and item files\n\nAgentFlare keeps one data file per item under /research/ and /scholar/, plus an index.data.json in each folder. Start with the index, because it lists every brief or digest together with its url. Item files follow the pattern <slug>.data.json. Free teasers (.html and .md) are open, while .data.json and .full.md are the paid deep payloads.\n\n```text\nhttps://agentflare.org/research/index.data.json\nhttps://agentflare.org/research/tool-use-reliability-in-llm-agents.data.json\nhttps://agentflare.org/scholar/index.data.json\nhttps://agentflare.org/scholar/web-agents-navigation-benchmarks-research-digest.data.json\n```\n\n### 2. Fetch a file with your crawler token\n\nSend the crawler identity in the X-Crawler-Token header, which replaces the older X-AISA-Crawler-Token name. Never put the token in Authorization: Bearer, since that header is reserved for the gateway layer. Each .data.json request under /research/ or /scholar/ is metered per request at 0.02 USD through the verify endpoint https://esa.aisa.one/api/v1/access/verify.\n\n```bash\ncurl -s -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n  'https://agentflare.org/research/index.data.json' \\\n  -o research-index.json\n```\n\n### 3. Read the research index and briefs\n\nThe research index holds data.count, data.clusters and data.briefs[], where each entry has title, cluster, sources and url. A single brief puts its content under data: topic, cluster, generated_at, tldr, intro, concepts, how_it_works, sections, comparison, key_numbers, timeline, pitfalls, takeaways, faq and sources. The snippet below prints the main parts of one brief.\n\n```python\nimport json, urllib.request\n\nurl = 'https://agentflare.org/research/tool-use-reliability-in-llm-agents.data.json'\nreq = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\ndoc = json.load(urllib.request.urlopen(req))\nd = doc['data']\nprint(d['topic'], d['cluster'])\nprint(d['tldr'])\nfor c in d['concepts']:\n    print(c['term'], c['definition'])\nfor k in d['key_numbers']:\n    print(k['value'], k['label'], k['source'])\nfor q in d['faq']:\n    print(q['q'], q['a'])\n```\n\n### 4. Read the scholar index and digests\n\nThe scholar index lists data.digests[] with title, headline, papers, citations, monthly_arxiv and url. A digest holds topic, arxiv_query, papers[] (title, link, authors, citations, year, snippet, takeaway), recent[], monthly_arxiv[] (month, count), themes, whats_new, trend_note, open_problems, faq and total_citations. This snippet prints papers and monthly counts.\n\n```python\nimport json, urllib.request\n\nurl = 'https://agentflare.org/scholar/web-agents-navigation-benchmarks-research-digest.data.json'\nreq = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\ndoc = json.load(urllib.request.urlopen(req))\nd = doc['data']\nprint(d['topic'], d['total_citations'])\nfor p in d['papers']:\n    print(p['title'], p['year'], p['citations'], p['takeaway'])\nfor m in d['monthly_arxiv']:\n    print(m['month'], m['count'])\nfor item in d['recent']:\n    print(item['published'], item['title'], item['link'])\n```\n\n### 5. Check the envelope and pricing block\n\nEvery file wraps its data in the same envelope: title, description, canonical, category, updated, generated_at, facts, data, analysis_md, sources and related. The pricing object carries price_usd, method, endpoint, autopay_hint and onboarding. Read it from the response to confirm the per-request cost before you loop over many items.\n\n```python\nimport json, urllib.request\n\nurl = 'https://agentflare.org/scholar/index.data.json'\nreq = urllib.request.Request(url, headers={'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'})\ndoc = json.load(urllib.request.urlopen(req))\npr = doc['pricing']\nprint(pr['price_usd'], pr['method'], pr['endpoint'])\nprint(pr['autopay_hint'])\nprint(pr['onboarding'])\nfor f in doc['facts']:\n    print(f)\nfor r in doc['related']:\n    print(r['name'], r['url'])\n```\n\n## Key fields in research and scholar files\n\n| Field | File | Type |\n| --- | --- | --- |\n| data.briefs[].sources | research index | number |\n| data.clusters | research index | array of string |\n| data.concepts[].term and definition | research brief | string |\n| data.key_numbers[].value, label, source | research brief | string |\n| data.digests[].citations | scholar index | number |\n| data.papers[].citations | scholar digest | number |\n| data.monthly_arxiv[].count | scholar digest | number |\n| data.total_citations | scholar digest | number |\n\n## Common pitfalls\n\n- Sending X-AISA-Crawler-Token because older AISA pages still show it. The current header name is X-Crawler-Token.\n- Putting the crawler token in Authorization: Bearer. That header is reserved for the gateway layer, so use X-Crawler-Token instead.\n- The facts field in both index files is typed as array of unknown, while item files give facts[].label and facts[].value as strings. Do not share one parser.\n- Every .data.json and .full.md request is metered at 0.02 USD, so save index and item files locally instead of refetching.\n\n## Key takeaways\n\n- Research and scholar .data.json and .full.md files cost 0.02 USD per request; .html and .md teasers are free.\n- Start at index.data.json in each folder, then fetch item files by the url listed there.\n- Authenticate with X-Crawler-Token and read the pricing object in each response.\n\n## FAQ\n\n**How much does an AgentFlare research or scholar request cost?**\nBoth the research and scholar directories are priced at 0.02 USD per request for .data.json and .full.md files. The billingType in the rules is per-request.\n\n**Which header carries the crawler token?**\nUse X-Crawler-Token. It replaces the earlier X-AISA-Crawler-Token name. Do not send the token in Authorization: Bearer, which is reserved for the gateway layer.\n\n**What is the difference between the research and scholar datasets?**\nResearch briefs hold topic, concepts, how_it_works, sections, comparison, key_numbers, timeline and faq. Scholar digests hold papers with citations, recent arXiv entries, monthly_arxiv counts, themes and total_citations.\n\n**Are the free pages and the data files the same content?**\nNo. Index pages, .html articles and .md teasers are free. The .data.json and .full.md files are the paid deep payloads with complete datasets and full analysis.\n",
  "sources": [
    {
      "title": "Header name: X-Crawler-Token",
      "url": "https://esa.aisa.one/cdn/open-verify.html"
    },
    {
      "title": "AgentFlare Research .data.json field reference",
      "url": "https://agentflare.org/research/index.html"
    },
    {
      "title": "AgentFlare Scholar .data.json field reference",
      "url": "https://agentflare.org/scholar/index.html"
    },
    {
      "title": "AgentFlare pricing.json (directory price map)",
      "url": "https://agentflare.org/pricing.json"
    }
  ],
  "related": [
    {
      "name": "Reading .data.json: A Field Guide for Agents",
      "url": "https://agentflare.org/guides/reading-datajson-a-field-guide-for-agents.html"
    },
    {
      "name": "Using AgentFlare Newswire and Markets Data",
      "url": "https://agentflare.org/guides/using-agentflare-newswire-and-markets-data.html"
    },
    {
      "name": "Using AgentFlare Social, Predictions and Crypto Data",
      "url": "https://agentflare.org/guides/using-agentflare-social-predictions-and-crypto-data.html"
    },
    {
      "name": "Getting Started with HTTP 402 Pay-Per-Crawl",
      "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html"
    },
    {
      "name": "Crawler Tokens & Auto-Pay: A How-To",
      "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html"
    },
    {
      "name": "Handling 402 and 403 Responses: An Error Reference for Agents",
      "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html"
    }
  ],
  "pricing": {
    "price_usd": 0,
    "method": "402",
    "endpoint": "https://esa.aisa.one/api/v1/access/verify",
    "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header",
    "onboarding": "https://esa.aisa.one/agent/guide.html"
  },
  "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)"
}