llms.txt: What It Is and Whether Your Website Needs One
Short answer: llms.txt is a plain markdown file at the root of a website that lists the pages an AI model should read, proposed by Jeremy Howard in September 2024. Google states its Search and AI features do not read it, no major AI company documents reading it when answering questions, and a 300,000-domain study found no link between having one and being cited. It costs an hour to add and does no harm; it is not a substitute for crawlable pages.
Sources are llmstxt.org, Google Search Central, OpenAI and Anthropic documentation, Search Engine Journal, Search Engine Roundtable and SE Ranking, checked 18 October 2026.
What does the file contain?
The llms.txt proposal was published by Jeremy Howard on 3 September 2024 and updated on 10 August 2026. It asks for a markdown file at /llms.txt with:
- One H1 with the site or project name. This is the only required line.
- A blockquote with a short summary of what the site is.
- Optional paragraphs or lists with context a model needs.
- H2 sections, each a list of markdown links to pages, with an optional one-line note per link.
- A section headed "Optional" for pages a model can skip when short on context.
The idea is that a model with a limited context window gets a curated index instead of a 200-page crawl. Some sites also publish llms-full.txt, a single file with the full text of every page, though the proposal page does not define it.
Two real examples. Anthropic's developer documentation llms.txt has one H1, one blockquote and about 15 H2 sections linking several hundred pages. OpenAI's crawler documentation points to its own llms.txt as the index of its docs. Both are documentation sites for developers, which is the use case the proposal was written for.
Which AI systems read it?
Only what is documented.
| System | Documented position | Source and date |
|---|---|---|
| Google Search, AI Overviews, AI Mode | Does not use it. No new machine-readable files, AI text files, markup or Markdown are needed | AI optimisation guide, May 2026, updated July 2026 |
| Google (John Mueller) | No AI service has said it uses the file, and server logs show they do not fetch it | Search Engine Journal, 17 April 2025; Search Engine Roundtable, 17 June 2025 |
| OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User) | Controlled by robots.txt. OpenAI publishes an llms.txt for its own docs but does not document reading other sites' files | OpenAI bots documentation |
| Anthropic crawlers (ClaudeBot, Claude-User, Claude-SearchBot) | Controlled by robots.txt. The crawler page does not mention llms.txt | Anthropic support |
The evidence on outcomes: SE Ranking analysed about 300,000 domains and, as reported by Search Engine Journal on 20 November 2025, found 10.13% had the file and no relationship between having it and how often the domain was cited in AI answers.
What AI systems do read is what Google reads: HTML pages that are crawlable, with clear headings, dated facts and the answer near the top. That is covered in how AI search reads your website.
How do you write one for a 10-page site?
About 20 lines. The structure, described in prose because the file is only markdown:
- Line 1: a hash, a space and the business name.
- Line 3: a greater-than sign, then one sentence. What the business does, for whom, where. "Corporate secretarial firm in Singapore for private limited companies, founded 2015."
- Line 5: a short paragraph with the facts a model should not guess. Address, hours, prices, licences.
- A section headed Services, with one link per service page and a one-line note: the service and the starting price.
- A section headed Company, with links to the about, contact and pricing pages.
- A section headed Optional, with links to articles and the privacy policy.
Rules:
- Every link is an absolute URL to a live page that says the same thing. The file is a map, not a second copy of the site.
- Keep the notes factual. A model that reads "from S$600 a year" and finds S$600 on the page has a consistent source.
- Update it when a page or price changes. A stale file contradicts the site.
- Serve it as plain text or markdown at the root. It does not go in the sitemap and does not need a link from the site.
Total effort: under an hour for 10 pages.
What does it not do?
- It does not get the site into Google's AI Overviews or AI Mode. Google says its features ignore it.
- It does not block crawlers. That is robots.txt, with GPTBot, ClaudeBot and Google-Extended as the user agents.
- It does not replace structured data for products, local business details or articles.
- It does not fix a page that is slow, hidden behind JavaScript or missing a clear answer.
- It is not an official standard. No standards body has adopted it, and no AI company has committed to reading it.
Mueller's comparison, in the April 2025 Reddit thread, was to the keywords meta tag: a file the site owner controls, which a system would have to verify against the real pages before trusting, and which can be used to say things the pages do not.
Should you add one?
| Situation | Decision |
|---|---|
| Developer documentation with hundreds of pages | Yes. This is the use case it was designed for, and the tools that read it are coding assistants |
| Business site of 5 to 30 pages | Optional. One hour, no risk, no documented benefit |
| Site that is not yet crawlable, is slow, or is missing prices and facts on its pages | No. Fix the pages first; the file points at pages |
| A vendor selling llms.txt as an AI ranking service | No. Nothing published supports the claim |
The order of work for AI visibility is unchanged by this file: crawlable pages, facts with dates, structured data, speed, then a Search Console check of which queries now show AI features. The file can be the last 20 lines of that list. For the wider method, see AI search optimisation Singapore.
What cense does
cense's marketing AI agent runs a daily GEO and SEO schedule: an original article drafted nightly from the client's industry and competitor posts, with AI answers checked weekly and the client approving before publishing. From S$800 a month. See SEO and AI search.
What is llms.txt?+
A markdown file at the root of a website that gives AI models a short summary and a curated list of links to the pages worth reading. Jeremy Howard proposed it on 3 September 2024; the only required line is an H1 with the site name.
Does Google read llms.txt?+
No. Google's AI optimisation guide, published May 2026, states that no new machine-readable files, AI text files, markup or Markdown are needed for Google Search, AI Overviews or AI Mode, and that Search ignores them.
Does llms.txt improve AI citations?+
No evidence that it does. SE Ranking's analysis of about 300,000 domains, reported 20 November 2025, found 10.13% had the file and no relationship between having it and how often a domain was cited in AI answers.
Does llms.txt block AI crawlers?+
No. Crawler access is controlled by robots.txt, using user agents such as GPTBot, OAI-SearchBot, ClaudeBot and Google-Extended. llms.txt is a reading list, not an access rule.
Should a small business website add llms.txt?+
Optional. It takes under an hour for a 10-page site, does no harm and has no documented benefit. Crawlable pages with dated facts, prices and structured data come first; the file points at those pages.
