The only real way to work with generative AI is to provide text or base64 information. Everything else is inference, and inference is unreliable. LLMs.txt was proposed as a way to give AI systems that text: a signpost file at the root of your domain, telling models what your site offers and how to read it. It sounds reasonable. It is also solving a problem that has already moved on.

LLMs.txt files are not likely to be seen

If and when they are seen, they are not observed. There are no teeth or binding actions making LLMs.txt a requirement. There is no aggregator service pushing for adoption the way Google did for robots.txt in the early 2000s. Without that enforcement mechanism, compliance is optional, and optional standards on the open web tend to be ignored by the parties with the most incentive to ignore them.

Model makers want to scrape every page. They ignored robots.txt where it suited them, and they will ignore llms.txt for the same reasons. The data hunger of frontier models does not respect polite suggestions.

AI agents already read the live page directly

AI agents now have a generalized understanding of what pages we offer, so they try to read the live webpage directly rather than consulting a manifest file. The capability that did not exist when LLMs.txt was proposed is now standard. Agents can navigate, parse, and extract from rendered pages in real time. A static index file is, at best, redundant for the systems that matter most.

We can add LLMs.txt files because the cost is low. But the traffic they attract will be niche tools, employees testing the file, and model makers scraping it for training data alongside everything else. It is not a discovery channel. It is a courtesy.

The internet has shifted from human-mediated to bot-mediated

The deeper structural change is that the internet has shifted from human-mediated to bot-mediated. Two choices follow from that shift.

The first is to keep human websites and try to tell AI how to navigate them. This is the LLMs.txt approach. It is ineffective because it asks models to follow instructions they have no reason to follow, and because the websites themselves were built for human eyes, not agent parsing.

The second is to make AI equivalent websites via MCP. This is effective because it gives agents a purpose-built interface that returns content designed by us, facilitating the user's request with no context bloat. In the MCP model, guidance to AI is enforced by us in the tools we design. The agent does not guess. It calls a function with defined inputs and defined outputs.

Three models of discovery

The shift can be mapped across three models:

  • Before: Humans interacted with search engines, which interacted with webpages.
  • Now A: Humans interact with AI agents, which interact with search engines and webpages.
  • Now B: Humans interact with AI agents, which interact with MCP services.

Now A is where most of the web sits today. Now B is where the architecture is heading. LLMs.txt belongs to Now A. It tries to make the old web legible to the new agents. MCP belongs to Now B. It builds the new web for them.

What changes the calculus

Model providers may seek to monetize MCP discovery. If they do, that would counter LLMs.txt adoption further, because the economic incentive would be to route agents through controlled discovery layers rather than open manifest files. Either way, the direction is away from passive signposting and toward active, structured interfaces.

LLMs.txt was proposed before AI agents could effectively scrape and before MCP existed. Both have changed. The problem it solved was real at the time. The solution it offered has been overtaken by capability and protocol. Add the file if you want. Just know what it is, and what it is not.