Research · Discovery & crawling · updated 2026-11-19
llms.txt: The Standard for AI-Readable Sites
llms.txt is a proposed Markdown file at a site's root that points AI systems to its most important content. Roughly ~950 domains publish one, but no major AI provider has confirmed using it, so keep robots.txt for access control.
You're reading the full report — free for humans and search engines.
The same content ships as a structured JSON dataset (normalized facts, sources and
citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open
HTTP 402 payment protocol.
crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify
Large language models face two problems when they read websites: context windows are too small for whole sites, and converting HTML full of navigation, ads, and JavaScript into clean text is difficult. The llms.txt proposal, put forward by Jeremy Howard, answers with a curated Markdown file served at the domain root. It sits alongside robots.txt and sitemap.xml but has a different job: it describes and prioritizes content rather than controlling access. Adoption is small and provider support is unconfirmed, yet the file takes little effort to write. This brief covers its structure, how it differs from robots.txt, the AI crawlers that robots.txt can block, and the risks of relying on it today.
Key concepts
- llms.txt
- A proposed standard: a Markdown file at the site root that gives AI systems a curated, structured map of the most important content on a website. S2 · blog.openreplay.comS3 · semrush.com
- llms-full.txt
- An optional companion file for comprehensive sites, holding extensive documentation, code examples, and deeper technical detail, while llms.txt stays a concise overview. S2 · blog.openreplay.com
- robots.txt
- A voluntary protocol at the domain root that uses User-agent, Allow, and Disallow rules to tell crawlers, including AI bots, which paths they may fetch. S1 · webless.ai
- Context window limit
- The core problem named in the proposal: context windows are too small to handle most websites in their entirety, and HTML conversion to plain text is imprecise. S5 · searchengineland.comS2 · blog.openreplay.com
- AI crawler user-agents
- Named bots that robots.txt can target, including GPTBot, Google-Extended, CCBot, anthropic-ai, ClaudeBot, PerplexityBot, Bytespider, and Applebot-Extended. S1 · webless.ai
Publishing an llms.txt file
- Create a lowercase llms.txt fileSave the file as llms.txt, the filename used by the official specification, and serve it from the domain root so it resolves at yourdomain.com/llms.txt.
- Add an H1 with the site nameStart with a single H1 heading naming the site or product, using the Markdown # syntax.
- Add a blockquote summaryFollow the heading with an optional blockquote giving a brief description of what the company or site does.
- List key links under H2 sectionsGroup links under headings such as Products, Documentation, and Resources. Add a colon and a short description after each link to explain what it leads to.
- Optionally add llms-full.txtFor comprehensive sites, publish a second file with detailed documentation and code examples, keeping the main file as the concise overview.
- Keep robots.txt for access rulesContinue using robots.txt to allow or block named crawlers by user-agent and path; llms.txt describes content rather than granting or denying access.
What llms.txt is and the problem it targets
llms.txt is a proposed standard, put forward by Jeremy Howard, for giving large language models a Markdown file at the root of a website. Howard's proposal states that context windows are too small to handle most websites in their entirety, and that converting complex HTML with navigation, ads, and JavaScript into LLM-friendly plain text is difficult and imprecise. Other guides add that most AI crawlers read only basic HTML and miss content loaded by JavaScript, and that crawlers cannot tell which pages matter, so they may spend effort on older blog posts or administrative sections. The file is meant to point crawlers to a curated set of the most important content. Because it is Markdown, it is readable by people and by LLMs, and its fixed format also allows classical parsers and regex to process it.
File structure and placement
Save the file as lowercase llms.txt at the domain root, for example https://yourdomain.com/llms.txt. Most files follow one pattern: an H1 with the site or product name, an optional blockquote description, then one or more H2 sections containing lists of links, each with a colon and a short description of what it leads to. A typical example groups links under headings such as Products, Documentation, and Resources, with entries like Getting Started: Quick setup guide. For larger sites, an additional llms-full.txt can carry detailed documentation, code examples, and deeper technical details, while the main file stays a concise overview. One guide describes a typical file as 10-50 lines of Markdown that can ship in under 10 minutes. Because the format is plain Markdown, no special parser is required for an LLM to read it.
Relationship to robots.txt and current support
robots.txt and llms.txt answer different questions. robots.txt uses User-agent, Allow, and Disallow rules at the domain root to control whether crawlers fetch paths, and it is a voluntary protocol that compliant crawlers honor. Webless lists AI bots that can be blocked this way: GPTBot, Google-Extended, CCBot, anthropic-ai, ClaudeBot, PerplexityBot, Bytespider, and Applebot-Extended. llms.txt does not grant or deny access; it curates and describes content. Support is the open issue: the file is described as a proposed standard adopted by ~950 domains, and OpenAI, Google, and Anthropic have not confirmed their crawlers use it. Semrush notes there is no proof it boosts AI visibility or referral traffic, though Google Lighthouse now checks for it in site audits. A practical setup keeps robots.txt as the gatekeeper and llms.txt as the briefing.
robots.txt vs llms.txt
| Aspect | robots.txt | llms.txt |
|---|---|---|
| Main purpose | Controls whether crawlers can access pages | Guides AI systems to the most important content and how to use it |
| Format | Key-value directives such as User-agent, Allow, Disallow | Markdown with H1, blockquote, H2 sections, and link lists |
| Location | Domain root, at /robots.txt | Domain root, at /llms.txt |
| Status | RFC standard, universally supported | Proposed standard; no major AI provider has officially confirmed support |
| Optional extension | Per-bot rules for named user-agents | llms-full.txt with extended documentation |
Timeline
- 1994Martijn Koster proposes robots.txt for web crawlers.dev.to
- 2024LLM crawlers become a significant new audience for robots.txt files.webless.ai
- 2024Developer community establishes the /llms.txt and /llms-full.txt convention in late 2024.hamidayub.com
Common pitfalls
- OpenAI, Google, and Anthropic have not confirmed their crawlers use llms.txt, so publishing it carries no guaranteed effect on how AI systems read your site.
- There is no proof that llms.txt boosts visibility in AI-generated responses or drives referral traffic; treat it as a policy surface, not a citation booster.
- robots.txt is voluntary: compliant crawlers honor it, but non-compliant third-party scrapers do not. Do not rely on any single file to protect sensitive content.
- Blocking a crawler in robots.txt stops active crawling of paths, but pages already known from backlinks or sitemaps may have been cached earlier.
Key takeaways
- Publish a concise Markdown llms.txt at the domain root; it costs little, but its benefits depend on future AI provider support.
- Use robots.txt for access control and llms.txt for describing priority content; the two files do different jobs.
- Do not expect measurable visibility gains yet: no major provider confirms support, and no proof of a citation boost exists.
FAQ
Where do I put the llms.txt file?
Place it at the root of your domain so it resolves at yourdomain.com/llms.txt, using the lowercase filename llms.txt. This follows the same convention as robots.txt and sitemap.xml.
What format does llms.txt use?
Markdown: an H1 with the site name, an optional blockquote description, then H2 sections of links, each followed by a colon and a short description. An optional llms-full.txt can hold more detailed content.
Does llms.txt replace robots.txt?
No. robots.txt controls which crawlers may fetch which paths, while llms.txt curates and describes important content. One guide frames robots.txt as the gatekeeper and llms.txt as the briefing document.
Do ChatGPT, Claude, or Google use llms.txt?
OpenAI, Google, and Anthropic have not confirmed that their crawlers use llms.txt files. Semrush reports no proof of a visibility boost, though Google Lighthouse now checks for the file in site audits.
How this page is made. Built from the full text of 8 sources found with the AISA Tavily API (advanced search), refreshed weekly. A language model organizes what those sources state into the blocks above; each block links the sources it draws on, and figures that do not appear in the sources are removed.
Sources & citations
- Robots.txt for LLMs: Controlling AI Crawlers on Your Site | Webless webless.ai
- llms.txt: A New Way for AI to Read Your Site blog.openreplay.com
- What is llms.txt & should you use it? semrush.com
- What Is LLMs.txt? Guide for AI Crawlers | Similar AI similar.ai
- Meet llms.txt, a proposed standard for AI website content crawling searchengineland.com
- LLMs.txt: Control How AI Crawlers Use Your Content royalplugins.com
- llms.txt vs robots.txt vs ai.txt: The Developer's Cheat Sheet - DEV Community dev.to
- Anatomy of llms.txt: Formatting Web Assets for AI Crawlers | Hamid Ayub — Insights hamidayub.com