llms.txt: What It Does, What It Doesn’t and Whether to Use It
Evaluate the llms.txt proposal as a lightweight content guide—not an access-control standard—and decide whether a low-cost implementation is worthwhile.
llms.txt is a proposed convention for publishing a concise Markdown index of a website’s important LLM-friendly resources at /llms.txt. It can be a useful curated map, especially for documentation sites. It is not a permissions file, a crawl directive, an official web standard or a documented ranking requirement for major AI search products.
Use it only after essential crawl, index and content work is sound.
What the proposal contains
The proposal by Jeremy Howard describes:
- an H1 naming the project or site;
- a short blockquote summary;
- optional explanatory text;
- H2 sections containing lists of links with descriptions;
- an optional section for secondary material.
It also proposes clean Markdown versions of detailed pages. The goal is to help inference-time systems find concise expert information without processing a complex website.
What llms.txt does not do
It does not:
- allow or block crawlers;
- override robots.txt;
- control training use;
- guarantee discovery, indexing or citations;
- replace an XML sitemap;
- repair inaccessible JavaScript content;
- prove that a provider consumes the file.
The name creates confusion with robots.txt. Treat llms.txt as a curated index, not access control.
Major platform guidance
OpenAI documents OAI-SearchBot, GPTBot and ChatGPT-User controls through robots.txt and published IP ranges. Perplexity documents PerplexityBot and Perplexity-User similarly.
Google explicitly says sites do not need new machine-readable AI files to appear in AI Overviews or AI Mode.
None of those statements makes llms.txt harmful; they mean it should not be sold as a required ranking factor.
When it may be useful
Consider it when:
- the site has extensive technical documentation;
- users or tools benefit from a curated resource map;
- important pages have clean Markdown equivalents;
- maintenance can be automated;
- the team wants to experiment with direct-agent workflows.
It may also serve human developers who want a compact inventory.
When to skip it
Skip or postpone it when:
- priority pages are blocked or not indexed;
- canonical URLs are inconsistent;
- content is thin or outdated;
- the file would be generated once and forgotten;
- the team expects measurable ranking improvement without evidence.
Fix robots.txt for AI crawlers and the technical crawler audit first.
A cautious implementation
```markdown
Official product documentation and implementation resources.
Core documentation
- Product overview: Scope, use cases and limitations.
- API documentation: Current integration reference.
Optional
- Release notes: Dated product changes.
```
Keep the file concise, canonical and free of private URLs. Link only to resources that are accessible and current.
Maintenance and measurement
Generate the file from a controlled content inventory where possible. Validate links on every deploy. Record access logs and referral or agent behavior, but do not interpret a fetch as proof that the file influenced an answer.
Run a simple test:
- define the agent or workflow expected to use the file;
- record the baseline;
- publish and validate;
- observe retrieval;
- keep it only if maintenance cost and utility are justified.
Frequently asked questions
Is llms.txt an official standard?
It is a public proposal and emerging convention, not a ratified internet standard.
Is it robots.txt for AI?
No. robots.txt controls crawler access. llms.txt proposes a content map and controls nothing.
Will Google use it for AI Overviews?
Google says no new AI text file is required for its Search AI features.
Can it contain every page?
It should be curated. A sitemap is a better exhaustive URL inventory.
