llms.txt

Do AI search engines use llms.txt?#

No major AI search engine has confirmed using llms.txt to find, rank or cite content as of September 2026: Google says Search doesn’t use it, and OpenAI, Anthropic and Perplexity publish llms.txt files for their own developer docs without saying their crawlers read anyone else’s.

• Google Search ignores it

  Official

  Google’s AI optimization guide says “Google Search itself doesn’t use them,” and that creating one “will neither harm nor help your site’s visibility or rankings in Google Search.”

• AI search bots barely fetch it

  Observed

  In Ahrefs’ study, the search crawlers of OpenAI, Perplexity and Anthropic combined made only a couple of hundred llms.txt fetches across the whole sample.

• Coding agents do read it

  Observed

  The proposal’s August 2026 revision says “coding agents use them reliably.” That is its authors’ report rather than an independent measurement, but it matches where the file was designed to help.

Google’s John Mueller put the skeptical case in 2025, calling llms.txt “comparable to the keywords meta tag”: a site owner’s claim about itself, which a crawler could check by reading the site directly.

llms.txt vs robots.txt: what's the difference?#

llms.txt is a reading guide that suggests which pages a language model should read first and grants or blocks nothing, while robots.txt is a set of access rules telling crawlers which paths they may fetch, which reputable search and AI bots obey.

The practical consequence runs both ways. If you want to keep AI bots out, llms.txt can’t do it; if you want them in, robots.txt and your CDN decide that, whatever llms.txt says. The two files don’t interact, and neither replaces an XML sitemap, which lists every canonical URL rather than a curated few. See robots.txt and AI crawlers.

How do you create an llms.txt file?#

Create an llms.txt file by writing a Markdown document with an H1 naming the site, a one-paragraph blockquote summary, and H2 sections listing your most useful pages as links with a short note on each, then serving it at /llms.txt.

# Plannora
 
> Plannora is project management software for small teams: boards,
> automations and a free plan for up to five users.
 
## Product
- [Pricing](https://plannora.io/pricing): plans, limits and what's free
- [Integrations](https://plannora.io/integrations): Slack, Google Workspace, GitHub
 
## Docs
- [API reference](https://plannora.io/docs/api): REST endpoints and webhooks
 
## Optional
- [Changelog](https://plannora.io/changelog)

• Only the H1 is required. The summary, detail and link sections are optional in the spec.
• Optional is a convention for secondary links that a model short on context can skip.
• Markdown versions of pages help agents. The proposal suggests serving a clean .md copy of each page at the same URL with .md added or swapped in, and its 2026 revision adds link relations so agents can find them.
• llms-full.txt is a companion convention: documentation platforms such as Mintlify generate one that compiles all docs text into a single Markdown file.

The llms.txt generator builds the file from a short form. Keep it curated, not exhaustive, and update it when key pages move.

Should I add an llms.txt file?#

Add an llms.txt file if you publish developer docs, an API or anything coding agents read, since that’s where the file is reported to be used; for AI search visibility, treat it as harmless but optional, and fix crawler access and answer-first pages first.

Developer docs, APIs, SDKsHelps

Coding agents fetch llms.txt and Markdown page versions to load documentation efficiently, the use the 2026 revision reports as reliable.

Google AI Overviews & AI ModeNo effect

Google says Search doesn’t use llms.txt; it neither helps nor hurts.

ChatGPT, Claude and Perplexity searchUnconfirmed

None has said its search crawler reads other sites’ llms.txt, and log studies show almost no fetches.

Replacing robots.txt or a sitemapNo effect

It controls no access and lists only a curated subset of pages.

How do you measure whether llms.txt is used?#

Measure llms.txt use from your server logs: count requests to /llms.txt by user agent, verify any big names against their published IP ranges, and compare AI citations before and after publishing the file.

# Who requests llms.txt, by user agent (combined log format)
grep '"GET /llms.txt' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -20

Expect mostly SEO tools and unidentified bots: in Ahrefs’ study, audit tools were the largest group of requesters and AI search crawlers barely appeared. A request isn’t proof of use either — a fetch shows a bot read the file, not that any answer drew on it. Citations are the outcome that matters, and they’re measured by running a fixed set of buyer prompts on a schedule, which is prompt tracking, not by reading logs. If you publish a file, give it a fair test: note the date, keep the prompt set fixed, and compare the months before and after. AI answers vary from run to run, so judge the trend across dozens of prompts, never a single answer.

Related terms#

• AI crawlersrobots.txtA plain-text file at a site’s root that tells crawlers which paths they may fetch, rule by rule for each user agent — a voluntary standard that reputable search and AI bots follow, which controls crawling but not whether a URL gets indexed.Read the entry
• AI crawlersAI crawlersAutomated bots run by AI companies that fetch web pages for one of three jobs — training models, building an AI search index, or retrieving a page live for a user’s question — and because each job uses its own user agent, each can be allowed or blocked separately.Read the entry
• Technical SEOXML sitemapA machine-readable file listing a site’s canonical URLs, optionally with last-modified dates, so search engines can discover pages and prioritize recrawling the ones that changed — a discovery aid, not a ranking factor or a guarantee of indexing.Read the entry
• GEOAI search & GEOGenerative engine optimizationThe practice of making content easy for AI answer engines such as ChatGPT, Perplexity and Google AI Overviews to retrieve, quote and cite — where SEO competes for a ranked link, GEO competes to be a named source inside the generated answer.Read the entry
• Technical SEOSchema markupStructured data, usually written as JSON-LD with the Schema.org vocabulary, that labels what a page contains — an article, a product, an organization, a local business — so search engines can interpret it unambiguously and show it as rich results.Read the entry