The New Technical Signals Lesson 17 of 27

llms.txt: what it is and whether you need it

What you'll learn
  • What llms.txt is and how it works
  • The honest state of adoption — who's using it, who isn't, and why
  • Whether you actually need one on your site right now

What llms.txt is

llms.txt is a file you place at the root of your website — at yoursite.com/llms.txt — that gives AI systems a structured summary of what your site contains and where the most important content lives.

It works roughly the way robots.txt works for search engines, but with a different purpose. Where robots.txt is mostly about permissions (which crawlers can access which parts of your site), llms.txt is about orientation — pointing AI systems at the content you’d most like them to find and use when answering questions about you or your topics.

A typical llms.txt file is short and human-readable. It usually contains:

  • The name of your site and what it’s about
  • A brief description of who you are and what you do
  • A list of the most important pages on your site, often grouped by category
  • Optional links to plain-text or Markdown versions of those pages, which AI systems can read more easily than HTML

It’s plain text. There’s no schema, no code, no formal validation step. You write it the way you’d write a short site map for a friend.

How it works in practice

When an AI system visits your site — particularly one trying to answer a question about your topic or your business — it can choose to look for /llms.txt and read it before looking at anything else. If the file is there, well-structured, and useful, the AI gets a fast, accurate overview of what you publish and where to find the substantive content.

The intended benefit is twofold. First, AI systems waste less time crawling pages that aren’t relevant to the question they’re trying to answer. Second, you get to nudge them toward your strongest content — the pages you’d most like to be cited from.

In theory, this is genuinely useful. In practice, the benefit depends entirely on how widely llms.txt is supported by the AI systems that matter to you. And that’s where things get more complicated.

The honest state of adoption

llms.txt is a proposed standard, not an established one. It was introduced in 2024 by a developer named Jeremy Howard, who wanted to give websites a way to communicate their structure to AI systems more effectively. The idea was good. The reception has been mixed.

As of the time this lesson is written, some AI systems do read llms.txt files. Others don’t. The major AI companies — OpenAI, Anthropic, Google — haven’t formally committed to honouring the standard. Some smaller AI tools and search products have. Adoption is real, but it’s partial and uncertain.

This makes llms.txt an unusual kind of recommendation. It’s:

  • Cheap to implement. A small file at the root of your site, written in plain text. No code, no developer required.
  • Risk-free. If AI systems ignore it, nothing breaks. There’s no downside to having one.
  • Not yet a guaranteed benefit. You can’t point to specific traffic, citations, or visibility that come from llms.txt alone. You’re investing in a standard that may or may not become important.
  • Potentially useful for entity clarity. Even if AI systems don’t formally read the file, having a well-structured summary of your site at a predictable URL contributes to the overall picture an AI builds of you.

The honest framing is this: llms.txt is a small bet on the future, made with very little cost. It’s not a current-state SEO requirement. It’s not a magic bullet. But for a site that’s already done the other GEO work well, adding llms.txt is one of those low-effort, low-risk moves that costs almost nothing and might quietly help.

Whether you need one

Here’s a simple way to decide.

You probably want llms.txt if:

  • Your site has more than 10–15 pages of substantive content
  • You publish regularly on a defined topic and want AI systems to find the strongest pages first
  • You’ve already done the other GEO work — entity clarity, structured data, content patterns — and want to add the next layer
  • You care about being among the early adopters of emerging web standards

You can probably skip it for now if:

  • Your site is very small (a five-page brochure site doesn’t benefit much from a sitemap-style file)
  • You haven’t done the more foundational GEO work yet — there’s no point pointing AI systems at content that isn’t ready to be cited
  • You’re not in a topic area where AI-driven traffic or citation is a meaningful concern

In neither case is llms.txt the most important thing on your list. It’s a sensible addition once the bigger pieces are in place, and a distraction before then.

What a useful llms.txt looks like

If you decide to add one, the structure is simple. A working file might look something like this:

Warren Groom — Freelance WordPress Developer

Toronto-based freelance WordPress developer, working with marketing and PR agencies across Canada, the US, and the UK. Specialising in custom hand-coded themes, white-label delivery, SEO, and AI visibility (GEO).

Services

About

Learning resources

Insights

That’s the whole file. Short, structured, written for a human to read. No code, no special characters, no required syntax beyond clear headings and clean links.

You can extend it with category descriptions, longer summaries, or links to plain-text versions of key pages — but the simple version above is enough to be useful if AI systems choose to read it.

A useful mindset

llms.txt is a small investment in a future that might happen. It costs almost nothing, breaks nothing, and might quietly help. That’s enough reason to consider it — but not enough to call it essential.

If you do the other GEO work well, you’re already winning the game. llms.txt is a finishing touch, not a foundation.

Coming up in the next lesson: AI crawlers — GPTBot, ClaudeBot, PerplexityBot. The named bots that visit your site on behalf of AI systems. We’ll look at what each one is, what it does, and what decision (if any) you need to make about it.

← Previous lesson How to know if your schema is working Structured Data, Made Simple Next lesson → AI crawlers: GPTBot, ClaudeBot, PerplexityBot The New Technical Signals