llms.txt: what it is and whether your website needs it
Translated from the Spanish original. Leer en español
You've been told your website needs a file called llms.txt so artificial intelligence reads it better. A thread says so, a course repeats it and someone has already sold you the service of creating it. Before doing anything, it is worth knowing what it is and what Google says.
Short answer. llms.txt is a Markdown text file, at the root of your website, that summarizes your site and links to its key pages so AI agents can read it. It is a proposal, not a Google standard. Google Search doesn't use it: creating it neither helps nor harms your ranking there.
What is llms.txt and who proposed it?
It is a proposal by Jeremy Howard, published in September 2024. The idea is simple: web pages are made for people, with menus, ads and JavaScript, and a language model has a hard time extracting clean text from them. A file called /llms.txt gives it, in one place, a summary and a list of links to what matters. The proposal itself explains it at llmstxt.org.
It is not an official standard of any search engine or any body. It is a convention each site decides to adopt or not. Nor is it a robots.txt for AI: it doesn't block or allow anything, it only guides. If you want to review what each control file does, I explained it in robots.txt is not a padlock.
What does an llms.txt file look like inside?
According to the specification, it is a Markdown file with a fixed structure, in this order:
- An H1 title with the site's name. It is the only mandatory part.
- A blockquote with a short summary.
- Optional text with details on how to interpret the rest.
- Sections with an H2 title that contain lists of links, each with its optional note.
There is a convention: the section called "Optional" is for secondary things, what an agent can skip if it needs to save space.
A made-up example, for a spare parts shop that doesn't exist:
# Example Spare Parts> Online shop of spare parts for cars over ten years old, with 48-hour delivery across Spain.## Catalog- [Brakes](https://ejemplo.com/frenos/): Pads, discs and fluids.- [Filters](https://ejemplo.com/filtros/): Oil, air and fuel.## Help- [Shipping and returns](https://ejemplo.com/envios/): Deadlines and conditions.## Optional- [Blog](https://ejemplo.com/blog/): Maintenance guides.
The proposal also suggests offering a Markdown version of each important page, with the same address and the .md ending. It is the part that takes the most work, and it isn't necessary to have the file.
What does Google say about llms.txt?
Here you have to separate two things. For Google Search, the answer is clear. In its guide on optimizing for generative AI features, Google puts llms.txt files among the things you can ignore. In my words: you don't need to create machine-readable files, text files for AI, special markup or Markdown to appear in Google Search, because Google Search doesn't use them. The guide is at developers.google.com.
That guide adds one more nuance: Google can discover, crawl and index many types of files besides HTML, but that doesn't mean it treats them specially. It also adds that it is perfectly valid to create and maintain an llms.txt for other services or systems that do use it. In that case, in Google Search it will neither harm nor help your visibility.
Put another way: Google doesn't read it to rank you or to choose what to show in AI Overviews or AI Mode. Its guide AI Features and Your Website says the same: you don't need text files for AI to appear in those features. If you want to see what does count, you have it in how to appear in AI Overviews.
Does any AI use it?
This is the question that matters and the answer is: there is no public evidence of general use. In June 2025, Google's John Mueller wrote that no AI system was using llms.txt at that time, and that it showed in server logs: chatbots request your pages, but not the llms.txt. Search Engine Roundtable picked it up. Months earlier, Mueller had compared it to the meta keywords tag, according to Search Engine Journal.
But there is a nuance. The proposal itself says thousands of sites publish it, that several documentation platforms generate it automatically and that AI labs publish it in their developer documentation. Where it seems to make the most sense is in technical documentation, where coding agents follow those links. A blog or a services website is a different case.
My reading, and it is my own judgment: that some agents consult it in technical documentation doesn't prove it matters for your business website. Without measuring it in your logs, you don't know.
How does it differ from robots.txt and the sitemap?
The three files live at the root of your website, but do different things. This table sums it up:
| File | For whom | What it does | Format |
|---|---|---|---|
| robots.txt | Crawlers | Indicates which paths they can request | Text with rules |
| sitemap.xml | Search engines | Lists the URLs you want them to know | XML |
| llms.txt | AI agents | Summarizes the site and links to what matters | Markdown |
If you cannot see all the columns, scroll the table sideways.
The llms.txt proposal frames it as a complement: while the sitemap lists all pages, llms.txt offers an annotated selection. But it doesn't replace either of the other two. If your sitemap isn't right, fix it before thinking about llms.txt.
What mistakes repeat when creating it?
These are the ones I see most often in files of this kind. It is what I would review, not an official list:
- Putting all the website's URLs in. An llms.txt isn't a sitemap: if you fill it with links, it loses the point of being a selection.
- Linking to pages that redirect, return 404 or are blocked. An agent that follows the link meets an error.
- Promising Markdown versions that don't exist. It is better to link to the real HTML page.
- Writing a summary full of slogans. The summary should say what the site is, who it serves and what it contains, in a few sentences.
- Creating it once and forgetting it. When you change the website's structure, the file falls behind.
Should I create an llms.txt on my website?
This is my rule, and it is my own judgment, not a Google rule. Ask yourself three questions:
- Is your website technical, product or API documentation? Then it makes more sense, because it is the use it was born for.
- Can you generate it from your CMS or your sitemap with no manual work and keep it up to date? Then it costs little.
- Do you expect it to give you traffic, citations or rankings? Then don't do it for that. Google has said it doesn't use it and there is no public evidence from the others.
If the answer is "I don't know", prioritize what you do know works: useful content, a clean technical structure, a correct sitemap.xml and Google being able to index what you want. Also check in Search Console that what matters is indexed.
My own website has one, at castillocanton.com/llms.txt, with the main pages and the Desayuno SEO articles.
How do I create it and check it?
There are four steps:
- Write the file following the structure: H1, summary, sections with links.
- Link only to pages that exist and that you want known. If you don't have Markdown versions, link to the HTML ones. Don't announce what isn't there.
- Upload it to the root, so it opens at yourdomain.com/llms.txt.
- Check that it loads with a 200 code and that the links aren't broken. A crawler like Screaming Frog works for that: you can feed it the list of links in List mode.
Update it when you publish or delete important pages. An llms.txt with broken or old links is worse than not having one.
Frequently asked questions
Does llms.txt improve my ranking on Google?
No. Google says Google Search ignores these files: they neither help nor harm your visibility or rankings. It is in its guide on optimizing for generative AI features.
Is it the same as robots.txt or a sitemap?
No. The robots.txt tells crawlers what they can request. The sitemap lists URLs for search engines. The llms.txt is a Markdown summary with links, meant to be read by an AI agent, and it blocks nothing.
Do I have to create Markdown versions of my pages?
It isn't mandatory. The proposal recommends it for pages with information an agent may need, but the file works by linking to HTML pages.
Can having it harm me?
According to Google, in Google Search it neither helps nor harms. The real risk is maintenance: broken links or outdated content.
Sources
- Google Search Central, Guide to optimizing for generative AI features (in Spanish).
- Google Search Central, AI Features and Your Website.
- Jeremy Howard, The /llms.txt file.
- Search Engine Roundtable, Google Says No AI System Currently Uses LLMs.txt.
- Search Engine Journal, Google Says LLMs.txt Comparable To Keywords Meta Tag.
To finish your coffee
llms.txt is a sensible, cheap proposal, but with scarce public evidence of use so far. If it costs you nothing, add it. If it is sold to you as what will make AI cite you, be wary: nobody can promise that.
If you doubt what to apply on your website, you can see what my technical SEO service includes.
Tomorrow, another topic with much more pull than any technical file: Google reviews. How to ask for them without breaking Google's rules and what the platform prohibits. See you at Desayuno SEO.
