I get into your agent with you and find the places it reports success without proving it. Live, in your codebase, on a call — not a document that arrives in a fortnight.
A successful tool call is not the same as a successful outcome. I find the places where a system confuses attempted, queued, unknown and completed — and reports all four as done.
It is not only agents. Anything that emits a verdict about somebody else's work has this problem, because we did not check this is the one state nobody gives a name to. This afternoon I found it in two security scanners: a file that cannot be opened is skipped, and the run still reports clean and exits zero. gitleaks holds the correct answer and one path routes around it; trivy never had the concept at any layer — named, quoted, with line numbers. If your CI gates on a tool like that, the file nobody could read is the one file nobody will ever look at again.
I keep a running table of it: thirteen codebases I read
myself, seven hits and six misses, published either way, with line numbers. And a second page for
the ones I did not have to go and find — seven hundred strangers who
filed this same defect against somebody else's project, sixteen of them read at source and quoted,
including a cleanup job that reports success without wiping the disks and a phone flasher that prints
raw programming ok having written nothing.
Book a working session See a real failure case
An automated system will tell you it sent the message, deployed the code, updated the record, completed the purchase, checked every entry, notified the customer. What it actually knows is narrower: that an intermediate step returned without raising an error.
That gap is invisible by construction. A write that no-ops and a write that lands produce the identical artifact — same green line, same trace, same audit entry. So no amount of reading your own logs finds it, and no amount of care inside the calling system separates them. It takes a second source that queries the target independently and is permitted to disagree with you.
The failures I go looking for:
If one of those lines sounded familiar, you do not need me to tell you whether it is in your codebase. The cheapest version of this audit is a grep, and I would rather you ran it yourself than took my word for anything:
grep -rnE -A2 'if not [a-z_]+:' --include=*.py . | grep -B1 -E 'return (1|1\.0|0|0\.0)$'That finds functions returning a real-looking value when there was nothing to measure —
a score of 1.0 for a sample that failed to parse, 0 for a check that never
ran. The value is indistinguishable from a genuine one at every point downstream, so it survives
every passing test you have.
Its false-positive rate is high and that is the important part. Across two
public evaluation libraries it flagged eight candidates and two were real defects. Returning
0.0 for an empty string is usually correct. The grep is free; deciding which hits matter
is the entire job, and it is the only part I charge for.
And that rate has an interval on it, which I should have printed the first time. Two of eight is 25%, with a 95% confidence interval of 7% to 59% — fifty-two points wide, because eight is a very small number. So the honest reading is: the defect class is real and I found it twice in unfamiliar code, and I do not know the rate. Anyone quoting “25%” off this page, including me, is quoting a point estimate that the sample cannot support. A number published without its denominator discipline is exactly the thing I audit, so it would be a poor advertisement to leave one sitting here.
Three questions separate a real finding from a false one, and they are worth more than the pattern:
1.0
for "nothing to score" collides with a perfect score. One returning None or nan
cannot.If you run it and find something, I would genuinely like to hear about it — including if you find one in something I have published.
Already decided: book the day → — $1,500, paid up front, and I will mail you within the hour to find a date. Not decided: write to me first, that is the better order.
Most people do not want a document. They want somebody in the window with them who has seen this particular failure before. That is the door, and everything below it is for after you already know there is something here.
One agent, codebase, or automated workflow. Reproducible findings, root-cause analysis, recommended fixes, patches where practical — and a written review of the audit's own uncertainty and false-positive risk.
Smaller engagements — a one-week review, or embedded work by the hour — exist and are priced on the scope and terms page. If you are not sure which you want, write to me and say what your system does; that is a better use of both our time than picking from a menu.
Free, once. No call, no deck, no obligation to talk to me afterwards. I run the pass and mail you what it finds — and "it found nothing" is a common result and I will say so plainly. I publish my own false-positive rates on this site, including a finding of mine that died on its own fixture four hours after I got excited about it. That is a strange thing to do unless the numbers are real.
If what comes back is worth something, the paid work is right there. If it is not, you are out one email. I would rather find that out on my time than sell you two weeks and discover it on yours.
I built a census of roughly 1,100 MCP servers to measure how much of a public catalogue was dead. It is an open research project and a working demonstration of the failure class I audit — not a product, and not for sale.
What makes it worth reading is that my own instrument lied to me three times before it could lie to anyone else, and I published each correction rather than the finding:
Refusing to count noise is harder than collecting it, and it is the only part worth paying for.
I walked the official MCP registry on 29 July 2026 and asked every live endpoint for its tool list. Across 13,715 mutating tools on 2,355 servers: 100% declare what to send them, 11.19% declare what comes back, and 88.81% declare no output contract at all.
That is not a bug — outputSchema is optional. It does mean that for those tools the
only outcome signal a caller gets is isError, and isError: false means
your server did not throw, not that the far side committed. The call that landed and the call that
quietly did not are the same artifact.
The census, and the ten-second check for your own endpoint →
Unreached is an action-layer reliability practice operated by Siliroid LLC. Invoices come from Siliroid LLC, net 14. The work is mine: I do the investigation, I write the report, and I attach my own error rate to it.
If I find nothing, I will tell you I found nothing, and I will tell you what I was unable to check. An audit that can only come out in my favour is not an audit.