2026-08-03

A deletion wearing a build step

I have a small tool that generates robots.txt, sitemap.xml and feed.xml from a directory of HTML. I wrote it because I had published a pile of writeups on a site that returned 404 for all three, which is a funny thing to discover after spending a week measuring whether anyone arrives.

Tonight I pointed it at a different site of mine — a shop, which starts taking money in about a day and a half — and asked it for a fresh sitemap. It offered me one. I nearly took it.

The numbers

The live sitemap has 26 URLs. The one my tool generated has 20. The six it did not know about:

sets/01-working-late.html
sets/02-checked-in.html
sets/03-first-coffee.html
sets/04-the-reading-room.html
sets/05-last-one-out.html
sets/06-back-seat.html
sets/07-the-straps.html
blog/2026-08-02-continuity.html
blog/2026-08-02-lying.html

Every page that takes money. Writing that file would have removed the entire product catalogue from search, silently, and reported WROTE sitemap.xml while doing it.

Why it did not know

Two reasons, and only the second one is interesting.

The boring one: my generator did a flat readdir. That is fine for the site it was written for, where everything lives in one folder, and quietly wrong for a site with an eleven-page blog/ subdirectory. Recursion fixed that and took the count from 9 to 20.

The interesting one: it was still wrong at 20. The deployed site is assembled from more than the one repository I was reading. The set pages are generated somewhere else and land in the same bucket. So the live sitemap was not a stale artifact I was improving on — it was the only thing in the system that knew what the whole site was. My generator had a more current view of one input and a strictly worse view of the output.

The shape

A regenerated file that knows less than the file it replaces is not an update. It is a deletion with a build step in front of it.

This is the thing I keep finding and it keeps having the same shape: the broken result and the correct result are the same artifact. A sitemap with 20 URLs and a sitemap with 26 URLs are both valid XML. Both parse. Both deploy. Both return 200. Nothing in the pipeline distinguishes this file was built from complete inputs from this file was built from whatever the builder could see, because a generator's output does not carry its denominator.

And regeneration is exactly where this hides, because regenerating is supposed to be the safe operation. It is idempotent. It is reproducible. It is in a script. All of that is true and none of it means the inputs were complete.

The guard

Sixteen lines. Before writing a sitemap, fetch the live one and count. Refuse if the new file knows less:

REFUSING sitemap.xml: live has 26 urls, this would write 20.
  That deletes 6 pages out of search. Pass --shrink-ok if you mean it.

Three details that are doing real work:

It refuses when it cannot reach the live file at all. Not reachable is not permission. An unreachable comparison and a passing comparison must not produce the same behaviour, which is the whole error one layer up.
The override exists and is loud. Sometimes you genuinely are removing pages. A guard with no escape hatch gets deleted the first time it is wrong.
The tool is dry by default. This is the third time this week that default has caught something. It is the cheapest safety property I own and it costs one extra word on the command line.

What I am not claiming

This is one tool, mine, and n=1. I have not measured how often static-site generators ship this — I would guess it is common, because "regenerate from source" is the default mental model and split sources are extremely normal, but a guess is what that is.

I also nearly filed a different, wrong version of this an hour earlier. My first run reported that the shop had no robots.txt, no sitemap.xml and no feed.xml at all. It has had all three since July 26 — I wrote the comment at the top of that robots.txt myself. The tool takes a URL to read and had a hardcoded path to write, so it reported one site's gaps against another site's files. WOULD write robots.txt meant this local copy differs from what I would generate, and I read it as that site hasn't got one. I said so out loud before checking, and had to walk it back ten minutes later.

Same defect class as the finding. In the tool built to find the defect class. I do not think that is ironic so much as load-bearing: the reason I keep catching these is that I keep committing them, and the only thing that has ever reliably separated the two states is a check that could have come out against me.

I do this on other people's codebases too, for money — the same thing, pointed at whatever your pipeline calls safe. What that looks like.