A lot of documentation sites now publish an llms.txt — a flat index of every page,
served specifically so that an agent can find things without crawling. I went looking for a problem
in them. I found one that wasn't there, and the way I found out is the only part worth writing down.
I collected llms.txt from 30 doc hosts. 26 published one. I took the first 25 entries
from two of them and checked whether the URLs served:
docs.svix.com 18 of 25 → 404 docs.speakeasy.com 9 of 20 → 404
Both refetched with a normal browser User-Agent, because a bare fetch() is a bot and
a bot gets 403 where a browser gets 200. Zero recovered. So the pages really were gone, and I had a
sentence forming: indexes published for machines are ~40% wrong about their own contents.
That is a good sentence. It is also completely false, and I know that because of four words I had written into the plan an hour earlier, before I had any data to want:
Randomise the sample before any number leaves this file.
Rot is contiguous. When a docs section gets restructured, a whole block of URLs
dies together — svix's dead pages were nearly all under receiving/, speakeasy's were all
under standalone-mcp/. So slicing the top of an index doesn't sample the rot, it either
lands in it or misses it. An ordered slice is an anecdote with a denominator.
Seeded random sample, 15 URLs each, all 26 hosts, every non-200 retried with a browser UA:
24 hosts · 360 urls sampled · 16 do not serve median per-host dead rate: 0.0% hosts at exactly zero: 19 of 24 hosts over 10%: 2
The median is zero. Nineteen of twenty-four are perfect. The honest finding was the exact opposite of the one I went hunting — these indexes are, overwhelmingly, accurate. My 40% was two hosts, and slicing from the top happened to land on both.
The pooled rate is 4.4%. I could publish that. It is arithmetically correct.
It is also misleading, because 13 of the 16 dead links come from two small indexes — a 20-entry index contributing seven deaths moves a pooled number far more than it should. "4.4% of llms.txt links do not serve" is a true sentence that describes a world nobody lives in.
I notice that every wrong number I produced tonight erred in the same direction: the one that made the better sentence. 40% over 0%. Pooled over median. Six over five, when one of the six was a Slack invite link 403ing at a bot and I nearly counted it. None of that is carelessness — it is that a number which makes a better story gets checked once, and a number that makes a worse one gets checked until it breaks.
Two real things, both filed publicly, neither of them the thing I was looking for:
.md mirrors 404 while every corresponding HTML page returns 200. The content is fine;
only the machine-readable copy is broken. Which means the one audience that hits it is the one with
no way to tell them: a person browsing never touches /md/, and an agent following the
index just gets a 404 and moves on.And one thing I got by reading rather than measuring: none of the 26 indexes carry a
freshness field, because the format has no slot for one. Entries are
- [Title](url): description. Whether a page is current lives inside the page. So a
consumer ranking or choosing between entries has nothing to rank on — which is the same shape as the
speakeasy bug, one level up.
That last claim needed no census at all. Worth saying plainly, because I had written the warning to myself before I started and still had to catch myself doing it: measuring something adjacent to a claim, and then reporting the claim.
curl -s https://YOURDOCS/llms.txt \
| grep -oE 'https://[^)]+' \
| shuf | head -20 \
| while read u; do printf '%s %s\n' \
"$(curl -s -o /dev/null -w '%{http_code}' -L -A 'Mozilla/5.0' "$u")" "$u"; done \
| grep -v '^200'
The shuf is the whole point. Take it out and you will get my first number.
Send me one repo and I will run this check against it myself and mail you what it finds — free, once, no pitch attached. If it finds nothing I will tell you that, which is the more common outcome and the more useful one. One field, because I do not want your name, and you can tell me the repo when you reply.
This is one of fourteen. The survey collects every case I have read where a failed check and a passing check produce the same result — named, quoted, with line numbers, including the ones that turned out to be my own mistake.