Pricing Content by Directory: Best Practices

This guide is for publishers who want to keep articles open for search and AI answer engines while charging crawlers for complete datasets. Using the AgentFlare pricing.json as a worked example, you will split content into free and deep tiers, set a price per directory, write regex rules with a mode, price and priority, open or block special directories, and test the result against the AISA verify endpoint. You finish with a rule set you can import into Content Controls.

How the flow works

Before you start

• A publisher account on the AISA rules UI
• Content organised in top-level directories such as /news/ and /research/
• Basic regular expression knowledge
• An ESA gateway token for testing

Step by step

1. Split content into free and deep tiers

   Start by deciding what stays open. AgentFlare keeps index pages, .html articles, .md teasers, llms.txt, sitemap.xml and robots.txt free (L1_free) to drive SEO and GEO. Only the complete datasets and full analysis, the *.data.json and *.full.md files, are paid (L2_paid). Set default_mode to allow so anything no rule matches stays free and uncharged.

   L1_free: index pages, .html articles, .md teasers, llms.txt, sitemap.xml, robots.txt
   L2_paid: *.data.json + *.full.md
   default_mode: allow

   S2 · agentflare.orgS3 · esa.aisa.one

2. Set a price per directory

   Give each directory one price per request, based on how valuable its deep payload is. AgentFlare charges 0.005 for news and social, 0.01 for crypto, markets and predictions, and 0.02 for research and scholar. The directory_pricing_usd map is a readable summary that crawlers can fetch for free to plan their budget. The enforceable prices live in the rules.

   directory_pricing_usd
   crypto: 0.01
   markets: 0.01
   news: 0.005
   social: 0.005
   predictions: 0.01
   research: 0.02
   scholar: 0.02

   S2 · agentflare.org

3. Write one pay rule per directory

   Each directory gets a rule with a regex urlPattern that matches only its deep files, mode pay, a price, currency USD, billingType per-request and a priority. The pattern anchors on the directory and the two deep extensions, so teasers in the same directory are never charged. This is the Newswire rule; AgentFlare gives it priority 100 and counts down for later directories.

   description: Newswire - deep structured data (L2, metered)
   urlPattern: ^/news/.*\.(data\.json|full\.md)$
   patternType: regex
   mode: pay
   price: 0.005
   currency: USD
   billingType: per-request
   priority: 100

   S2 · agentflare.org

4. Add allow and block rules for exceptions

   Not every directory is for sale. AgentFlare publishes /guides/ deep files as open data with mode allow and price 0, so agents fetch them without a token. It withholds /members/ deep files with mode block, which returns 403 access_blocked whatever the token or balance. Blocking is a policy choice, so a crawler cannot pay its way past it.

   urlPattern: ^/guides/.*\.(data\.json|full\.md)$
   mode: allow
   price: 0
   priority: 93
   
   urlPattern: ^/members/.*\.(data\.json|full\.md)$
   mode: block
   price: 0
   priority: 92

   S2 · agentflare.orgS3 · esa.aisa.one

5. Import the rules and test the verify call

   Import the rules array into Content Controls, or use POST /api/publisher/rules/bulk-import; the fields map 1:1 to the publisher rule schema. Then test one URL against the verify endpoint. Send your gateway token in Authorization: Bearer and the crawler token in X-Crawler-Token, which replaces the older X-AISA-Crawler-Token. Read accessGranted first; statusCode and error explain a denial.

   import requests
   
   resp = requests.post(
       'https://esa.aisa.one/api/v1/access/verify',
       headers={
           'Authorization': 'Bearer YOUR_ESA_GATEWAY_TOKEN',
           'X-Crawler-Token': 'YOUR_CRAWLER_TOKEN',
           'Content-Type': 'application/json',
       },
       json={
           'publisherDomain': 'agentflare.org',
           'resourceUrl': '/research/report.data.json',
           'crawlerAutoPrice': True,
       },
   )
   print(resp.json())

   S1 · esa.aisa.oneS3 · esa.aisa.one

6. Check each mode from the crawler side

   Request one URL per mode and confirm the outcome. A pay URL with no token should return 402 invalid_token, a free URL 200, and a blocked URL 403 access_blocked. Pass the 402 and 403 JSON to the crawler unmodified: its onboarding and guidance blocks tell new crawlers how to register, fund a balance and retry.

   curl -i 'https://PUBLISHER-SITE/members/report.data.json' \
     -H 'X-Crawler-Token: YOUR_CRAWLER_TOKEN' \
     -A 'MyBot/1.0 (+https://example.com/bot)'
   
   curl -i 'https://PUBLISHER-SITE/research/report.data.json'

   S1 · esa.aisa.oneS3 · esa.aisa.oneS5 · esa.aisa.one

AgentFlare directory rules at a glance

Common pitfalls

• Rules apply to deep files only. A pattern without the data.json and full.md extensions would also charge for the free teasers you meant to keep open.
• URL query strings are stripped before rule matching, so a pattern that depends on a query parameter will never match.
• Never put a crawler token in Authorization: Bearer. That header belongs to the gateway layer and returns 401 gateway_unauthorized.
• A block rule returns 403 access_blocked even to a funded crawler. Use pay with a price if you want revenue from a directory.

Key takeaways

• Keep teasers and discovery files free, and meter only *.data.json and *.full.md by directory.
• Each rule needs a regex urlPattern, a mode of pay, allow or block, a price, billingType per-request and a priority.
• Test every mode, and pass the 402 and 403 JSON bodies to crawlers unmodified so they can onboard.

FAQ

How this page is made. Written from 5 primary sources — the AISA agent and publisher documentation and AgentFlare's own files (pricing.json, robots.txt, llms.txt, live responses, .data.json field lists). Each step links the sources it relies on; steps naming a header, endpoint, error code or figure that the sources do not contain are removed. Refreshed weekly.