Building an Agent-Readable Site: HTML, Markdown and .data.json

This guide is for publishers who want AI agents to read their site without scraping HTML. You will learn how AgentFlare exposes one path in three forms, which tier is free and which is priced per crawl, how pricing rules map deep files to a price, and how a shared JSON envelope keeps every section parseable by the same agent code. By the end you can lay out your own URLs, write metering rules and fetch a deep payload with a crawler token.

How the flow works

Before you start

• A site where each article has a stable URL path
• An AISA account registered at the agent login page
• curl or Python with the requests library installed
• Access to edit robots.txt and your CDN content rules

Step by step

1. Publish one path in three forms

   Give every article a single base path and serve it in three forms: the HTML page for humans, a free Markdown teaser at .md, and a metered .data.json or .full.md that carries the complete dataset and full analysis. Agents then never need a separate API host; they pick the format by changing the file extension on a path they already know.

   https://PUBLISHER-SITE/news/SLUG.html       free, for humans
   https://PUBLISHER-SITE/news/SLUG.md         free teaser
   https://PUBLISHER-SITE/news/SLUG.data.json  metered, structured data
   https://PUBLISHER-SITE/news/SLUG.full.md    metered, full analysis

   S2 · agentflare.orgS4 · agentflare.org

2. Keep the free tier open to crawlers

   AgentFlare treats index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt as L1 free. Its robots.txt allows every user agent, including GPTBot, ClaudeBot and PerplexityBot, and points to the sitemap. Open teasers drive SEO and GEO discovery, while llms.txt acts as a map that tells agents the three forms exist and where to pay.

   User-agent: *
   Allow: /
   
   User-agent: GPTBot
   Allow: /
   
   User-agent: ClaudeBot
   Allow: /
   
   Sitemap: https://PUBLISHER-SITE/sitemap.xml

   S3 · agentflare.orgS6 · agentflare.org

3. Meter deep payloads with path rules

   Put a pricing.json beside your site and describe each directory with a rule. A regex urlPattern selects the .data.json and .full.md files, mode set to pay attaches a per-request price, and priority orders the rules. Anything not matched stays free by default, so the open surface is untouched. The rules import into cdn-agent Content Controls.

   urlPattern: ^/news/.*\.(data\.json|full\.md)$
   patternType: regex
   mode: pay
   price: 0.005
   currency: USD
   billingType: per-request
   priority: 100

   S3 · agentflare.org

4. Fetch the deep payload with a token

   Paid files settle through the AISA HTTP 402 protocol at https://esa.aisa.one/api/v1/access/verify. Agents set crawlerAutoPrice=true and identify themselves with the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token still shown on older AISA pages. Keep the token out of Authorization: Bearer, which is reserved for the gateway layer.

   curl -s \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     'https://PUBLISHER-SITE/news/SLUG.data.json'

   S1 · esa.aisa.oneS2 · agentflare.org

5. Use one JSON envelope everywhere

   Every .data.json shares the same top-level fields, whichever section it comes from. Only the data object changes shape: news carries stories and market arrays, while a markets stock file carries ticker, price_usd, returns_pct and metrics. An agent can read title, facts and sources from any file before it learns the section-specific schema.

   @context, type, tier, title, description
   canonical, category, updated, generated_at
   facts[] with label and value
   data (shape varies by section)
   analysis_md
   sources[] with title and url
   related[]
   pricing with price_usd, method, endpoint, autopay_hint, onboarding
   powered_by

   S4 · agentflare.orgS5 · agentflare.org

6. Parse the envelope in an agent

   Because the envelope is stable, the reading code is short. Request the file with the crawler header, parse the JSON, then read the common fields first and drill into data only for the section you care about. This example reads a stock file from the markets directory and prints its facts alongside the price and daily change.

   import requests
   
   url = 'https://PUBLISHER-SITE/markets/xom.data.json'
   headers = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}
   doc = requests.get(url, headers=headers, timeout=30).json()
   
   print(doc['title'], doc['updated'])
   for fact in doc['facts']:
       print(fact['label'], fact['value'])
   
   data = doc['data']
   print(data['ticker'], data['price_usd'], data['day_change_pct'])
   print(doc['analysis_md'][:200])

   S1 · esa.aisa.oneS5 · agentflare.org

Deep payload rules by directory

Common pitfalls

• Never send the crawler token in Authorization: Bearer. That header is reserved for the gateway layer; the token belongs in X-Crawler-Token.
• Older AISA pages still show X-AISA-Crawler-Token. Copying that name from them means your crawler identity header is wrong.
• Rules only meter URLs matching the pattern. Paid payloads must end in .data.json or .full.md, or they fall under default allow and stay free.
• The members directory is set to block for crawlers, so a crawler cannot buy its deep files at any price.

Key takeaways

• One path in three forms: open HTML, a free .md teaser, and a metered .data.json or .full.md for deep data.
• Regex rules in pricing.json set per-request prices by directory; unmatched URLs stay free by default.
• A shared envelope with facts, data, analysis_md, sources and pricing lets one parser read every section.

FAQ

How this page is made. Written from 6 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.