# Agents fetch pages, not llms.txt · Onto Blog
> Ahrefs logged 137k domains in May 2026. 97% of published llms.txt files got zero requests. Agents fetch the page they were sent to — serve Markdown there.

**Source:** /blog/agents-fetch-pages-not-llmstxt
**Extracted:** 2026-09-04T20:32:43.759Z

---
Engineering · 8 min read

## Agents fetch pages, not llms.txt

Ahrefs logged 137k domains in May 2026. 97% of published llms.txt files got zero requests. Agents fetch the page they were sent to — serve Markdown there.

[All posts](/blog)

The web spent a year arguing about a file.

Publish `/llms.txt`. Put a summary at the root. Link the important pages. Agents will orient themselves without crawling the HTML. The SEO industry attached “AI visibility” to that filename before any major platform committed to reading it.

### The logs

In June 2026, Ahrefs published the log data. They looked at 137,210 domains in Ahrefs Web Analytics that received traffic in May. They checked each root for an `llms.txt` that returned HTTP 200 and was actually Markdown, not an HTML 404 in disguise.

28% of that set published the file. Ahrefs is explicit: their customers skew more technical than the web at large, so treat 28% as an upper bound. Of the ~38,000 domains with a valid file, 97% saw zero requests in May. No bots. No humans. Nothing.

If you published `llms.txt` this year, the most likely outcome is that nothing fetched it.

### The 3% that got hit

The 3% that did get hit are more interesting than the headline. 96% of those requests were bots. Named AI tools were 19.5% of that tiny pool. GPTBot led. Claude-Code was second, ahead of every AI search and assistant crawler. AI retrieval bots — the ones that answer live questions in ChatGPT, Perplexity, AI Overviews — were 1.1%. Slackbot fetched `llms.txt` more often than PerplexityBot.

SEO audit tools were 21.7%. GEO/AEO checkers, `llms.txt` validators, and research crawlers together were about 12%. A supply chain formed around a file before the readers showed up.

### Agents do not go looking

The cleanest split in the study: AI bots never requested `/llms.txt` on domains that did not have one. The 404s were almost all humans typing the path. Agents do not go looking for the file. They fetch it when a link, an index, or a user instruction tells them it exists.

Google already said the quiet part. Search Central: machine-readable files like `llms.txt` are not required to appear in generative AI features. John Mueller: the file is “not done for search.” A temporary crutch, maybe, for coding tools parsing developer docs.

Chrome shipped an experimental Lighthouse check for the same file in the same week. Ahrefs counted that audit at roughly 1 in 1,000 fetches. A check is not a reader.

### The sidecar is the wrong layer

None of this means Markdown for agents is a dead idea. It means the sidecar is the wrong layer.

Agents already request URLs. Pricing. Docs. Changelog. They do not fail loudly when the response is 800 KB of React. They parse what they can, fill the gaps, and answer anyway. The hallucination is often in the payload, not the model.

The useful move is not a second document at a path nobody asks for. It is the same URL, a different representation. `Accept: text/markdown`. Humans keep the page. The agent gets structure instead of markup. That is content negotiation. The web already knows how to do this. Almost no site does it for agents.

### Same URL, a different representation

Vercel and Cloudflare already proved Markdown-for-agents on their own customers. The gap is a Next.js app that is not on those platforms. [Onto Serve](/serve-sdk) is that layer: one line of middleware, App Router, Next.js 14+. It is shipping. It is not “any site.” It is not a Control Plane. It is not a waitlist.

[Onto Read](/read-api) is the other side of the same engine: when an agent (or a developer building one) needs a clean read of a URL that does not serve Markdown itself. [API](https://docs.buildonto.dev) and [MCP](/mcp-server). Score is free. A read costs 1 credit.

Do not confuse either with an `llms.txt` generator. A catalog at `/llms.txt` can still be useful as an index if you link it and if coding agents are actually your readers. Ahrefs' own numbers point that way: the agentic bucket out-fetched retrieval. Mueller's “docs crutch” matches the logs. It does not match the pitch that this file will get you cited in ChatGPT.

### If you want a file

If you want a file, treat it like code. Version it. Do not put instructions in it. Ahrefs found a research crawler named `prompt-injection-survey` already studying the path. Agents are built to trust what they ingest.

The line is still the same one on the [homepage](/): the web is written for eyes. Agents do not have any. They are already at your URL. Give them Markdown there.

### Sources

Louise Linehan, [Ahrefs llms.txt study](https://ahrefs.com/blog/llmstxt-study/), 15 Jun 2026. See also the [homepage](/) and [docs.buildonto.dev](https://docs.buildonto.dev).

Published 3 September 2026 · 8 min read

Keep reading

### More from the blog.

[OlderOnto has no AI inside — and that's the whole pointEngineering](/blog/no-ai-inside)

Get started

### Read the web the way an agent does.

[Get started](https://app.buildonto.dev/signup) [Scan your site](/scanner)

Free tier · 1,000 reads / month · no card