{ "@context": "https://agentflare.org/schema", "type": "Article", "tier": "L2-full", "title": "Building an Agent-Readable Site: HTML, Markdown and .data.json", "description": "AgentFlare serves every article at one path as HTML, a free .md teaser, and a metered .data.json or .full.md. A shared JSON envelope lets agents parse any…", "canonical": "https://agentflare.org/guides/building-an-agent-readable-site-html-markdown-and-datajson.html", "category": "guides", "updated": "2026-10-15", "generated_at": "2026-10-15T00:37:01.844Z", "facts": [ { "label": "Track", "value": "For publishers" }, { "label": "Level", "value": "Intermediate" }, { "label": "Access", "value": "Free / open" }, { "label": "Updated", "value": "2026-10-15" } ], "data": { "topic": "Building an Agent-Readable Site: HTML, Markdown and .data.json", "track": "publisher", "access": "free", "generated_at": "2026-10-15T00:37:01.844Z", "difficulty": "Intermediate", "tldr": "AgentFlare serves every article at one path as HTML, a free .md teaser, and a metered .data.json or .full.md. A shared JSON envelope lets agents parse any section with the same code.", "intro": "This guide is for publishers who want AI agents to read their site without scraping HTML. You will learn how AgentFlare exposes one path in three forms, which tier is free and which is priced per crawl, how pricing rules map deep files to a price, and how a shared JSON envelope keeps every section parseable by the same agent code. By the end you can lay out your own URLs, write metering rules and fetch a deep payload with a crawler token.", "prerequisites": [ "A site where each article has a stable URL path", "An AISA account registered at the agent login page", "curl or Python with the requests library installed", "Access to edit robots.txt and your CDN content rules" ], "steps": [ { "title": "Publish one path in three forms", "body": "Give every article a single base path and serve it in three forms: the HTML page for humans, a free Markdown teaser at .md, and a metered .data.json or .full.md that carries the complete dataset and full analysis. Agents then never need a separate API host; they pick the format by changing the file extension on a path they already know.", "code": { "lang": "text", "text": "https://PUBLISHER-SITE/news/SLUG.html free, for humans\nhttps://PUBLISHER-SITE/news/SLUG.md free teaser\nhttps://PUBLISHER-SITE/news/SLUG.data.json metered, structured data\nhttps://PUBLISHER-SITE/news/SLUG.full.md metered, full analysis" }, "sources": [ "S2", "S4" ] }, { "title": "Keep the free tier open to crawlers", "body": "AgentFlare treats index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt as L1 free. Its robots.txt allows every user agent, including GPTBot, ClaudeBot and PerplexityBot, and points to the sitemap. Open teasers drive SEO and GEO discovery, while llms.txt acts as a map that tells agents the three forms exist and where to pay.", "code": { "lang": "text", "text": "User-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nSitemap: https://PUBLISHER-SITE/sitemap.xml" }, "sources": [ "S3", "S6" ] }, { "title": "Meter deep payloads with path rules", "body": "Put a pricing.json beside your site and describe each directory with a rule. A regex urlPattern selects the .data.json and .full.md files, mode set to pay attaches a per-request price, and priority orders the rules. Anything not matched stays free by default, so the open surface is untouched. The rules import into cdn-agent Content Controls.", "code": { "lang": "text", "text": "urlPattern: ^/news/.*\\.(data\\.json|full\\.md)$\npatternType: regex\nmode: pay\nprice: 0.005\ncurrency: USD\nbillingType: per-request\npriority: 100" }, "sources": [ "S3" ] }, { "title": "Fetch the deep payload with a token", "body": "Paid files settle through the AISA HTTP 402 protocol at https://esa.aisa.one/api/v1/access/verify. Agents set crawlerAutoPrice=true and identify themselves with the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token still shown on older AISA pages. Keep the token out of Authorization: Bearer, which is reserved for the gateway layer.", "code": { "lang": "bash", "text": "curl -s \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n 'https://PUBLISHER-SITE/news/SLUG.data.json'" }, "sources": [ "S1", "S2" ] }, { "title": "Use one JSON envelope everywhere", "body": "Every .data.json shares the same top-level fields, whichever section it comes from. Only the data object changes shape: news carries stories and market arrays, while a markets stock file carries ticker, price_usd, returns_pct and metrics. An agent can read title, facts and sources from any file before it learns the section-specific schema.", "code": { "lang": "text", "text": "@context, type, tier, title, description\ncanonical, category, updated, generated_at\nfacts[] with label and value\ndata (shape varies by section)\nanalysis_md\nsources[] with title and url\nrelated[]\npricing with price_usd, method, endpoint, autopay_hint, onboarding\npowered_by" }, "sources": [ "S4", "S5" ] }, { "title": "Parse the envelope in an agent", "body": "Because the envelope is stable, the reading code is short. Request the file with the crawler header, parse the JSON, then read the common fields first and drill into data only for the section you care about. This example reads a stock file from the markets directory and prints its facts alongside the price and daily change.", "code": { "lang": "python", "text": "import requests\n\nurl = 'https://PUBLISHER-SITE/markets/xom.data.json'\nheaders = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}\ndoc = requests.get(url, headers=headers, timeout=30).json()\n\nprint(doc['title'], doc['updated'])\nfor fact in doc['facts']:\n print(fact['label'], fact['value'])\n\ndata = doc['data']\nprint(data['ticker'], data['price_usd'], data['day_change_pct'])\nprint(doc['analysis_md'][:200])" }, "sources": [ "S1", "S5" ] } ], "reference": { "title": "Deep payload rules by directory", "columns": [ "Directory", "Pattern covered", "Mode", "Price (USD)" ], "rows": [ [ "/news/", ".data.json and .full.md", "pay", "0.005" ], [ "/social/", ".data.json and .full.md", "pay", "0.005" ], [ "/markets/", ".data.json and .full.md", "pay", "0.01" ], [ "/predictions/", ".data.json and .full.md", "pay", "0.01" ], [ "/crypto/", ".data.json and .full.md", "pay", "0.01" ], [ "/research/", ".data.json and .full.md", "pay", "0.02" ], [ "/scholar/", ".data.json and .full.md", "pay", "0.02" ], [ "/guides/", ".data.json and .full.md", "allow", "0" ] ], "sources": [ "S3" ] }, "pitfalls": [ { "text": "Never send the crawler token in Authorization: Bearer. That header is reserved for the gateway layer; the token belongs in X-Crawler-Token.", "source": "S1" }, { "text": "Older AISA pages still show X-AISA-Crawler-Token. Copying that name from them means your crawler identity header is wrong.", "source": "S1" }, { "text": "Rules only meter URLs matching the pattern. Paid payloads must end in .data.json or .full.md, or they fall under default allow and stay free.", "source": "S3" }, { "text": "The members directory is set to block for crawlers, so a crawler cannot buy its deep files at any price.", "source": "S3" } ], "takeaways": [ "One path in three forms: open HTML, a free .md teaser, and a metered .data.json or .full.md for deep data.", "Regex rules in pricing.json set per-request prices by directory; unmatched URLs stay free by default.", "A shared envelope with facts, data, analysis_md, sources and pricing lets one parser read every section." ], "faq": [ { "q": "What is the difference between .md and .full.md?", "a": "The .md file is a free teaser in the L1 tier. The .full.md file sits in the L2 paid tier alongside .data.json and carries the full analysis, metered per crawl through the verify endpoint." }, { "q": "Which header identifies an AgentFlare crawler?", "a": "Send X-Crawler-Token with your token. It replaces the older X-AISA-Crawler-Token name. Do not use Authorization: Bearer for it, because that header is reserved for the gateway layer." }, { "q": "How do agents pay for a .data.json file?", "a": "Payment settles through the AISA HTTP 402 protocol at https://esa.aisa.one/api/v1/access/verify. Agents set crawlerAutoPrice=true with an X-Crawler-Token header. Register at https://esa.aisa.one/agent/login.html." }, { "q": "Does every section use the same JSON fields?", "a": "The top-level envelope is shared: title, canonical, facts, data, analysis_md, sources and related. The data object differs by section, for example stories in news versus stocks and sectors in markets." } ], "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html", "domain": "esa.aisa.one" }, { "title": "AgentFlare llms.txt (header and structure)", "url": "https://agentflare.org/llms.txt", "domain": "agentflare.org" }, { "title": "AgentFlare pricing.json (directory price map)", "url": "https://agentflare.org/pricing.json", "domain": "agentflare.org" }, { "title": "AgentFlare Newswire .data.json field reference", "url": "https://agentflare.org/news/index.html", "domain": "agentflare.org" }, { "title": "AgentFlare Markets .data.json field reference", "url": "https://agentflare.org/markets/index.html", "domain": "agentflare.org" }, { "title": "AgentFlare robots.txt", "url": "https://agentflare.org/robots.txt", "domain": "agentflare.org" } ] }, "analysis_md": "AgentFlare serves every article at one path as HTML, a free .md teaser, and a metered .data.json or .full.md. A shared JSON envelope lets agents parse any section with the same code.\n\nThis guide is for publishers who want AI agents to read their site without scraping HTML. You will learn how AgentFlare exposes one path in three forms, which tier is free and which is priced per crawl, how pricing rules map deep files to a price, and how a shared JSON envelope keeps every section parseable by the same agent code. By the end you can lay out your own URLs, write metering rules and fetch a deep payload with a crawler token.\n\n## Before you start\n\n- A site where each article has a stable URL path\n- An AISA account registered at the agent login page\n- curl or Python with the requests library installed\n- Access to edit robots.txt and your CDN content rules\n\n## Step by step\n\n### 1. Publish one path in three forms\n\nGive every article a single base path and serve it in three forms: the HTML page for humans, a free Markdown teaser at .md, and a metered .data.json or .full.md that carries the complete dataset and full analysis. Agents then never need a separate API host; they pick the format by changing the file extension on a path they already know.\n\n```text\nhttps://PUBLISHER-SITE/news/SLUG.html free, for humans\nhttps://PUBLISHER-SITE/news/SLUG.md free teaser\nhttps://PUBLISHER-SITE/news/SLUG.data.json metered, structured data\nhttps://PUBLISHER-SITE/news/SLUG.full.md metered, full analysis\n```\n\n### 2. Keep the free tier open to crawlers\n\nAgentFlare treats index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt as L1 free. Its robots.txt allows every user agent, including GPTBot, ClaudeBot and PerplexityBot, and points to the sitemap. Open teasers drive SEO and GEO discovery, while llms.txt acts as a map that tells agents the three forms exist and where to pay.\n\n```text\nUser-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nSitemap: https://PUBLISHER-SITE/sitemap.xml\n```\n\n### 3. Meter deep payloads with path rules\n\nPut a pricing.json beside your site and describe each directory with a rule. A regex urlPattern selects the .data.json and .full.md files, mode set to pay attaches a per-request price, and priority orders the rules. Anything not matched stays free by default, so the open surface is untouched. The rules import into cdn-agent Content Controls.\n\n```text\nurlPattern: ^/news/.*\\.(data\\.json|full\\.md)$\npatternType: regex\nmode: pay\nprice: 0.005\ncurrency: USD\nbillingType: per-request\npriority: 100\n```\n\n### 4. Fetch the deep payload with a token\n\nPaid files settle through the AISA HTTP 402 protocol at https://esa.aisa.one/api/v1/access/verify. Agents set crawlerAutoPrice=true and identify themselves with the X-Crawler-Token header, which replaces the earlier name X-AISA-Crawler-Token still shown on older AISA pages. Keep the token out of Authorization: Bearer, which is reserved for the gateway layer.\n\n```bash\ncurl -s \\\n -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \\\n 'https://PUBLISHER-SITE/news/SLUG.data.json'\n```\n\n### 5. Use one JSON envelope everywhere\n\nEvery .data.json shares the same top-level fields, whichever section it comes from. Only the data object changes shape: news carries stories and market arrays, while a markets stock file carries ticker, price_usd, returns_pct and metrics. An agent can read title, facts and sources from any file before it learns the section-specific schema.\n\n```text\n@context, type, tier, title, description\ncanonical, category, updated, generated_at\nfacts[] with label and value\ndata (shape varies by section)\nanalysis_md\nsources[] with title and url\nrelated[]\npricing with price_usd, method, endpoint, autopay_hint, onboarding\npowered_by\n```\n\n### 6. Parse the envelope in an agent\n\nBecause the envelope is stable, the reading code is short. Request the file with the crawler header, parse the JSON, then read the common fields first and drill into data only for the section you care about. This example reads a stock file from the markets directory and prints its facts alongside the price and daily change.\n\n```python\nimport requests\n\nurl = 'https://PUBLISHER-SITE/markets/xom.data.json'\nheaders = {'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN'}\ndoc = requests.get(url, headers=headers, timeout=30).json()\n\nprint(doc['title'], doc['updated'])\nfor fact in doc['facts']:\n print(fact['label'], fact['value'])\n\ndata = doc['data']\nprint(data['ticker'], data['price_usd'], data['day_change_pct'])\nprint(doc['analysis_md'][:200])\n```\n\n## Deep payload rules by directory\n\n| Directory | Pattern covered | Mode | Price (USD) |\n| --- | --- | --- | --- |\n| /news/ | .data.json and .full.md | pay | 0.005 |\n| /social/ | .data.json and .full.md | pay | 0.005 |\n| /markets/ | .data.json and .full.md | pay | 0.01 |\n| /predictions/ | .data.json and .full.md | pay | 0.01 |\n| /crypto/ | .data.json and .full.md | pay | 0.01 |\n| /research/ | .data.json and .full.md | pay | 0.02 |\n| /scholar/ | .data.json and .full.md | pay | 0.02 |\n| /guides/ | .data.json and .full.md | allow | 0 |\n\n## Common pitfalls\n\n- Never send the crawler token in Authorization: Bearer. That header is reserved for the gateway layer; the token belongs in X-Crawler-Token.\n- Older AISA pages still show X-AISA-Crawler-Token. Copying that name from them means your crawler identity header is wrong.\n- Rules only meter URLs matching the pattern. Paid payloads must end in .data.json or .full.md, or they fall under default allow and stay free.\n- The members directory is set to block for crawlers, so a crawler cannot buy its deep files at any price.\n\n## Key takeaways\n\n- One path in three forms: open HTML, a free .md teaser, and a metered .data.json or .full.md for deep data.\n- Regex rules in pricing.json set per-request prices by directory; unmatched URLs stay free by default.\n- A shared envelope with facts, data, analysis_md, sources and pricing lets one parser read every section.\n\n## FAQ\n\n**What is the difference between .md and .full.md?**\nThe .md file is a free teaser in the L1 tier. The .full.md file sits in the L2 paid tier alongside .data.json and carries the full analysis, metered per crawl through the verify endpoint.\n\n**Which header identifies an AgentFlare crawler?**\nSend X-Crawler-Token with your token. It replaces the older X-AISA-Crawler-Token name. Do not use Authorization: Bearer for it, because that header is reserved for the gateway layer.\n\n**How do agents pay for a .data.json file?**\nPayment settles through the AISA HTTP 402 protocol at https://esa.aisa.one/api/v1/access/verify. Agents set crawlerAutoPrice=true with an X-Crawler-Token header. Register at https://esa.aisa.one/agent/login.html.\n\n**Does every section use the same JSON fields?**\nThe top-level envelope is shared: title, canonical, facts, data, analysis_md, sources and related. The data object differs by section, for example stories in news versus stocks and sectors in markets.\n", "sources": [ { "title": "Header name: X-Crawler-Token", "url": "https://esa.aisa.one/cdn/open-verify.html" }, { "title": "AgentFlare llms.txt (header and structure)", "url": "https://agentflare.org/llms.txt" }, { "title": "AgentFlare pricing.json (directory price map)", "url": "https://agentflare.org/pricing.json" }, { "title": "AgentFlare Newswire .data.json field reference", "url": "https://agentflare.org/news/index.html" }, { "title": "AgentFlare Markets .data.json field reference", "url": "https://agentflare.org/markets/index.html" }, { "title": "AgentFlare robots.txt", "url": "https://agentflare.org/robots.txt" } ], "related": [ { "name": "Pricing Content by Directory: Best Practices", "url": "https://agentflare.org/guides/pricing-content-by-directory-best-practices.html" }, { "name": "robots.txt, llms.txt & Sitemaps for AI Crawlers", "url": "https://agentflare.org/guides/robotstxt-llmstxt-sitemaps-for-ai-crawlers.html" }, { "name": "How the Edge Verify API Decides Access", "url": "https://agentflare.org/guides/how-the-edge-verify-api-decides-access.html" }, { "name": "Getting Started with HTTP 402 Pay-Per-Crawl", "url": "https://agentflare.org/guides/getting-started-with-http-402-pay-per-crawl.html" }, { "name": "Crawler Tokens & Auto-Pay: A How-To", "url": "https://agentflare.org/guides/crawler-tokens-auto-pay-a-how-to.html" }, { "name": "Handling 402 and 403 Responses: An Error Reference for Agents", "url": "https://agentflare.org/guides/handling-402-and-403-responses-an-error-reference-for-agents.html" } ], "pricing": { "price_usd": 0, "method": "402", "endpoint": "https://esa.aisa.one/api/v1/access/verify", "autopay_hint": "set crawlerAutoPrice=true with the X-Crawler-Token header", "onboarding": "https://esa.aisa.one/agent/guide.html" }, "powered_by": "AISA — agent-native search, settlement & delivery (https://aisa.one)" }