What Is llms.txt? A Practical Guide Without the Hype

llms.txt is a community proposal for publishing a concise, Markdown-formatted map of a website's important content at /llms.txt. It can help a system that chooses to fetch the file find human-selected pages and descriptions. It is not a web standard, an access-control file, or a guaranteed ranking or citation signal.

The proposal is documented at llmstxt.org. Its purpose should be described precisely: it suggests a format that may help language models use a site at inference time. Major AI and search platforms do not universally promise to fetch it, respect its ordering, index the linked pages, or cite the site because the file exists.

What an llms.txt File Contains#

A useful file is a short guide to canonical public resources:

• A clear site or product name.
• A one-paragraph description.
• Links to the most important documentation, product, tool, or policy pages.
• A short explanation of what each linked page contains.
• Optional facts that are stable, sourced, and maintained.

Example:

# Example Product

> A concise description of the product and who it serves.

Start here#

• Overview: Product and documentation overview.
• API reference: Current endpoints and schemas.
• Pricing: Current plans and usage charges.

Key resources#

• Tool catalog: Canonical list of available tools.
• Changelog: Dated product updates.

Keep the file small enough to be useful as a map. Link to canonical pages rather than copying the full site into one very large response.

What llms.txt Does Not Do#

llms.txt does not replace:

• robots.txt: crawler access instructions.
• XML sitemaps: a standard search-engine discovery mechanism for canonical URLs.
• Canonical tags: duplicate-URL consolidation signals.
• Structured data: machine-readable facts attached to a page.
• Good content: the answer, evidence, and user experience on the linked page.

It also does not grant permission to ignore robots.txt, authentication, copyright, privacy, or site terms.

Should You Publish One?#

Publish llms.txt when all of these are true:

1. You can identify a small set of canonical public pages.
2. The descriptions can be kept accurate.
3. The file is generated or reviewed whenever those pages change.
4. Maintaining it costs little.

Do not prioritize it over crawlability, indexability, accurate documentation, internal links, and a valid sitemap. If the file becomes a stale second catalog, it creates conflicting information instead of clarity.

Why We Removed llms-full.txt from ToolRouter#

The proposal also describes an optional llms-full.txt variant that can inline more content. That is not automatically useful. ToolRouter's version had grown to roughly 1.6 MB and repeated content already available in the tool catalog, per-tool Markdown pages, and documentation.

We removed that duplicate endpoint and kept:

• A curated /llms.txt index.
• /tools.md as the catalog map.
• Per-tool .md pages for tool details.
• Canonical documentation pages.

This keeps each response focused and gives every item one maintained source of truth.

How to Create llms.txt#

1. Choose the canonical pages#

Start with pages that explain the entity, product, documentation, catalog, pricing, policies, and meaningful updates. Exclude sign-in, account, checkout, dashboard, filter, duplicate, and action URLs.

2. Write factual descriptions#

Describe what a reader will find on the linked page. Avoid promotional superlatives, hard-coded counts that drift, and unsupported statistics.

3. Publish it at the root#

Serve the file from:

https://example.com/llms.txt

Return a normal 200 response with a plain-text content type. Do not hide the only copy of important information in this file; the linked HTML pages remain the canonical user-facing sources.

4. Validate the links#

Check that every URL resolves directly to a public canonical page. Remove redirect chains, deleted articles, private routes, and query combinations that are not intended as standalone resources.

5. Monitor without assuming causality#

Log requests to the file if your infrastructure supports it, but do not interpret a fetch as a citation or a ranking improvement. Measure AI referrals, citations for a fixed prompt set, non-brand search visibility, and conversions separately.

Common Mistakes#

• Publishing thousands of low-value URLs instead of a curated map.
• Copying the full site into a multi-megabyte file.
• Treating the proposal as an official standard adopted by every platform.
• Claiming citation gains without a controlled measurement method.
• Letting product counts, commands, prices, or routes become stale.
• Linking to private, redirected, noindex, or deleted pages.