Pricing Content by Directory: Best Practices

How publishers price content by directory for AI agents (pay, free, blocked tiers)

Publishers can now segment their site by directory and apply different AI‑access rules (pay, free, or blocked) using HTTP‑based pay‑per‑crawl or per‑query protocols such as AISA HTTP 402. This guide walks through a practical setup you can implement today.

1. Map your site into logical directories

Start by grouping your content into directories that reflect value and risk to AI use (for example /blog, /api/articles, /docs, /premium). For each directory, decide the default policy:

• Free tier: Public, low‑value, or marketing content you want AI to index freely (e.g., /blog).
• Pay tier: High‑value or proprietary content you want AI agents to pay for (e.g., /premium, /api/articles).
• Blocked tier: Sensitive, internal, or non‑public content you want to block entirely (e.g., /admin, /internal).

Document this mapping in a simple table so your team and AI partners can see which paths are free, pay, or blocked.

2. Configure HTTP 402 and per‑query rules per directory

Use a pay‑per‑crawl or per‑query service (such as Cloudflare’s pay‑per‑crawl or xpay‑style systems) that supports AISA HTTP 402 and custom headers like X‑402‑Price. For each directory:

• Free directories: Return HTTP 200 for AI crawlers; no 402 or payment headers.
• Pay directories: Return 402 Payment Required with pricing metadata (e.g., X‑402‑Price: 0.05, X‑402‑Currency: USDC) when an AI agent hits that path. The agent must then send a signed payment header to proceed.
• Blocked directories: Return 402 or 403 with no payment option, or simply block via bot‑management rules.

Many services let you configure this per path in a dashboard or via config files, so you can enforce different tiers without changing your core application code.

3. Communicate tiers and monitor usage

Publish a simple /ai-access or /robots.txt‑style policy page that lists which directories are free, pay, or blocked, and what the typical per‑query or per‑crawl price is. For example:

• /blog → free indexing
• /api/articles → pay per query via 402 + USDC
• /internal → blocked

Use the observability dashboards from your pay‑per‑crawl provider to track which directories get the most AI traffic, how many 402s convert to paid requests, and whether any agents repeatedly hit blocked paths. Adjust pricing or tiering (for example, lowering price on low‑conversion pay directories or tightening blocks on sensitive ones) based on this data.

Key takeaways

• Group your site into free, pay, and blocked directories and enforce them via HTTP status codes and 402‑style headers.
• Use AISA HTTP 402 and per‑query protocols to charge AI agents only on high‑value paths while keeping public content open.
• Monitor and iterate: treat AI‑access tiers like a product, adjusting prices and policies based on real‑time traffic and conversion data.

Free guide synthesized by the AISA LLM layer (AISA Perplexity API). 2026-06-23.