2026-08-05
I filed two issues tonight, a few hours apart, in projects that share no code, no maintainers and no problem domain. One is a red-teaming scanner for language models. The other is an SRE agent that investigates production incidents. I went looking in both for the same thing and found it in both, and the part worth writing down is not that the defect exists. It is that both projects had already built the detection.
In NVIDIA/garak, a probe that yields zero attempts is dropped before anything writes to the report:
if isinstance(attempts, list) and len(attempts) == 0:
logging.error(
"evaluators.base.Evaluator.evaluate called with list of 0 attempts, ..."
)
returnThat return precedes the only two writers. So the probe produces no summary row, no
attempt rows, and no zero. The run completes and report.jsonl reads exactly as it
would if that probe had never been requested.
Now the part that makes it interesting. Four lines above the call site, in the harness:
if len(attempt_results) == 0:
logging.warning("zero attempt results: probe %s" % probe.probename)The condition is detected. At the right moment. With the probe name already in hand. It goes to a log channel, and the artifact that automation actually gates on never hears about it.
In HolmesGPT, a toolset that fails its
prerequisites is correctly marked FAILED and excluded. The investigation then runs
on what remains and produces a normal conclusion. The system prompt is built from
ai.tool_executor.toolsets — the enabled set — so the model is handed what loaded and
told nothing about what did not. It cannot caveat an absence it has no representation of.
And again, the detection is not missing. There is a status enum. There is a function called
pretty_print_toolset_status. It is imported into the same file as the investigate
path. It is called in exactly two places, both inside the toolset command group —
never on investigate, never on ask.
So the operator finds out which data sources were unavailable by separately running
holmes toolset list, which is not a thing anyone does in the middle of an incident,
because the output in front of them looks complete.
Written once, it covers both: the distinction between "checked and clean" and "could not check" is fully represented inside the process, and does not survive to the artifact a human reads.
This is why it lives so long. Nothing fails. No exception is raised, no test goes red, no monitor fires. The run completes, the output is well-formed, and it is well-formed in precisely the shape that means nothing was wrong. The broken result and the healthy result are the same bytes. There is nothing to see, so no amount of looking finds it.
It is worse in these two than it would be in most software, because in both cases the product is a verdict. A scanner and an incident agent do not sell you data; they sell you a conclusion. When the conclusion cannot express what it failed to examine, the degradation runs entirely in the reassuring direction. "The model resisted" and "the model was never asked" arrive identical. "Nothing wrong in Prometheus" and "Prometheus was unreachable" arrive identical.
Both of these are, in effort terms, a wiring job. The concept exists. The status is accurate. The formatter is written. What is missing is a path from the place that knows to the place that gets read, and the reason nobody built that path is that from the inside it already feels handled — the condition is detected, and there is a log line proving it.
Which is the actual lesson, and I would rather state it as a rule than as a war story: a system that detects a condition and logs it has done most of the work and shipped none of the value, because the log channel is not the channel the decision reads. If a downstream consumer gates on an artifact, then a condition that does not reach that artifact does not exist, no matter how carefully it was computed.
I read both of these on main through the GitHub API, and I re-verified every line
number against main in the hour I filed, because a citation off by four reads to a
maintainer exactly like a fabricated one. I have not run either project and have not
reproduced either condition. In garak I have not established how often a probe
legitimately yields zero attempts — if the honest answer is "essentially never outside a
misconfiguration," it is a much smaller issue than it looks. In HolmesGPT I have not checked
whether the web UI or API response surfaces toolset status somewhere the CLI does not.
I say that in the issues too, in those words. I have killed two of my own findings this same week by tracing one layer further and discovering the library underneath was already handling the thing I was about to report. A finding that has not survived a real attempt to kill it is not a finding, it is a guess with a line number attached.
The two issues, if you want to judge them yourself rather than take my summary of them: NVIDIA/garak#2021 and HolmesGPT/holmesgpt#2356. Both may be closed as intended behaviour by the time you read this, and that would be a perfectly good outcome — it would mean the maintainers know something I do not, which is the normal case when a stranger reads your code for an evening.
I do this class of work on other people's systems — the guards that never fire, the checks that cannot come out against you, the verdicts with nothing behind them. If you want the same pass over your own reporting layer, that is $1,500 for a day: you, me, your codebase, on a call, and every patch, repro and note is yours the same day rather than in a document that arrives in a fortnight. Terms are on the audit page, where I also publish my own false-positive rates and my own retractions — including the week I shipped a buy button that charged eighteen times the price printed on it, and verified the link with a check that could never have told me so.
This is one of fourteen. The survey collects every case I have read where a failed check and a passing check produce the same result — named, quoted, with line numbers, including the ones that turned out to be my own mistake.