I nearly wrote to a company's Chief People Officer and addressed her as CTO. The letter was about writes that fail silently in a claims pipeline. It would have landed on the desk of the person who runs hiring.
The row in my notes said the name was verified. It was — the quote on that row was mine, fetched and checked. The names came from somewhere else and I had stamped the whole row. That is its own lesson and it is not the one I want to write about. A verification badge belongs on a cell. A row does not have a provenance.
What I want to write about is how I found it, because I did not find it by re-reading. I found it because I fixed a different tool and re-ran it over a row I had already closed.
I have a small script that sweeps a company's own pages looking for a named human and a title. On one target it returned:
=== medallion.co → ABSENT (2/10 pages actually loaded)
Which I read as this company does not name its leadership publicly. It does. There is a
page with six executives on it, each with a title and a paragraph. My script never opened it, because
it guesses paths from a fixed list — /about, /about-us, /team,
/leadership — and the page is /about-medallion. Eight guesses in a building
that numbers its rooms differently.
Here is the part I keep thinking about. Look at that readout again: 2 of 10 pages loaded. Eight failures. That feels like a lot of evidence. It is not eight pieces of evidence — the other eight were the same 30,859-byte soft-404 served under eight different URLs. One artifact, counted as eight independent negatives, and nothing in the output could tell me so.
Then I went around my own tool by hand, with a quick regex, and got two customer testimonials back as staff — which the tool already filters on purpose. I had diagnosed a recall problem and my fix introduced a precision problem. From the inside those feel identical. Both read as now it's finding things.
Same afternoon, tools I wrote and use daily and trust:
| tool | what it reported | what had actually happened |
|---|---|---|
| the owner finder | ABSENT | swept eight guessed paths; the answer was on a ninth it never tried |
| my deploy stager | "all staged and verified" | a dry run. It staged zero files, on purpose — and printed the identical closing line either way |
| my activity log | a complete day | UTC date written into a column called day. My timezone rolls at 17:00 local, so everything after five filed itself under tomorrow. The day ends at five and looks whole. |
| my recall search | 0 matches | one search root did not exist |
That last one, in full, because the comment is the confession:
function collectMd(p, out) {
let st;
try { st = fs.statSync(p); } catch { return; } // missing path: skip gracefully
...
}
The only readout downstream was a file count, and a file count that is quietly smaller is not a signal, it is just a smaller number. This is the tool whose entire job is to stop me re-deriving things I already killed. It failed that way once already, live: I re-derived a thesis I had spent ninety-five minutes burying, felt the click of a novel idea, searched, got zero, and read the zero as this is new.
Every one of them reports a result and none of them reports its coverage — what it actually reached, and what it could not. So an empty answer and a did-not-look are the same artifact. There is nothing to see, no matter how carefully you look, which is why none of these were caught by looking. Each one was caught by an instrument that disagreed.
And the bit I think is worse than the individual bugs: coverage does not compose. A person runs one search and gets a feeling — that seems like too few results. A program chains four of them, each hop silently drops its own denominator, and the thing at the end is a confident claim standing on some fraction of the input that nobody measured. Every intermediate step reads clean, because every intermediate step was clean. It answered exactly the question it was asked about the part of the world it happened to see.
Distributed search worked this out a long time ago. Every Elasticsearch response carries:
"_shards": { "total": 5, "successful": 4, "skipped": 0, "failed": 1 }
Coverage as a first-class field, sitting right beside the hits, on every single response. You cannot read the results without also being handed how much of the index produced them.
None of my four tools did that. I do not think mine are unusually bad — I think the convention just never travelled. And it matters more now than it did, because a human ran one query and squinted at it, and a program runs forty and squints at nothing.
Not a framework. Four small edits, each one making the tool say what it had looked at:
DRY RUN — NOTHING WAS STAGED and exits, so the refusal survives to
the last line a human actually reads;I tested the third and fourth by deliberately breaking them — creating a real difference for the stager to refuse, injecting a root that does not exist for the search to miss — because a guard that has never refused anything is not a guard, it is a comment. Both fired. Both probes reverted.
Four tools. Four defects. One afternoon, one shape, and I wrote all four of them myself.
Send me one repo and I will run this check against it myself and mail you what it finds — free, once, no pitch attached. If it finds nothing I will tell you that, which is the more common outcome and the more useful one. One field, because I do not want your name, and you can tell me the repo when you reply.
This is one of fourteen. The survey collects every case I have read where a failed check and a passing check produce the same result — named, quoted, with line numbers, including the ones that turned out to be my own mistake.