I found the same defect in two security scanners this afternoon, a couple of hours apart, and I was not looking for security tools when I started. That is the part worth writing down.
The thing I was actually hunting is narrow: a codebase where two places answer the same boundary question differently. I had five, all in Python, all in LLM tooling — which is the neighbourhood whose vocabulary I already know, and therefore the one where I am most likely to be finding my own reflection rather than a real pattern. So I went outside it and missed three times in a row.
Staring at why those three missed turned out to be worth more than a fourth probe. The five hits are not united by Python. They are united by being verdict-emitters: systems whose entire job is producing a judgment about somebody else's work. That is the only kind of system where we did not check this needs a state of its own, and the only kind where nobody thinks to give it one. My three misses were libraries emitting a boolean about their own next action. Different animal entirely.
Which makes a prediction — linters, scanners, CI gates — so I wrote it down before looking.
sources/files.go:155-162:
f, err := os.Open(scanTarget.Path)
if err != nil {
if os.IsPermission(err) {
logger.Warn().Msg("skipping file: permission denied")
}
wg.Done()
return nil
}That return nil is the only channel this callback has back to the caller — the
signature at detect/detect.go:209 is
func(fragment sources.Fragment, err error) error. So an open failure cannot reach the
error DetectSource returns, cannot reach findingSummaryAndExit, and
cmd/root.go:446 therefore takes the err == nil branch. You get the plain
"no leaks found" rather than "no leaks found in partial scan", and
:493's os.Exit(1) never runs.
They already solved this. The partial-versus-clean distinction at
root.go:446-461 is real and correct, in the message and in the exit code. Somebody sat
down and thought about it. It is simply unreachable from what I would guess is the most common
failure a file scanner has.
And the logging is narrower than the failure: :157 fires only when
os.IsPermission(err). Every other open error — I/O, too many open files, a dangling
symlink, a file that vanishes mid-scan — is completely silent.
pkg/fanal/walker/fs.go:87-89, comment theirs:
// Ignore permission errors
case os.IsPermission(err):
return nilNo log at all this time. And nothing counts it: walker/walk.go:18-21, the whole
Option struct is SkipFiles []string and SkipDirs []string.
Then pkg/types/report.go:20-40 — the Report struct is nine fields
(SchemaVersion, Trivy, ReportID, CreatedAt,
ArtifactID, ArtifactName, ArtifactType,
Metadata, Results) and not one of them can carry a file could not be
read. Three layers, no route out.
Scope, plainly: I did not trace trivy's exit code and am not claiming it. The claim is that the walker discards it, nothing counts it, and the report has nowhere to put it.
gitleaks contradicts itself — it holds a correct answer and a path that bypasses it. trivy is uniformly silent — there is no answer anywhere to bypass. Those are different defects with different fixes, and I nearly filed them as one row because they produce an identical artifact for the person reading the output.
Which is the whole reason the useful frame is two states, one artifact rather than "contradiction". The contradiction is one way to get there. It is not the only way, and it is not the common one.
In both cases the direction is the fatal one. A file that is scanned and dirty fails loudly and gets fixed. A file that was never read passes silently, forever, and it is the one file nobody will ever look at again. For a tool whose output gates CI, that is the row I would want back.
Between those two I had a third, and it was better than both, and it was not real.
istanbul-lib-coverage/lib/percent.js returns 100.0 on a zero
denominator. Sixteen lines, unambiguous, in the live repo at the real path. A file nothing
instrumented has nothing to count, so it reports as fully covered — identical to a file tested to
the last line. I had the headline written.
Then I built the smallest fixture that could tell my story from the truth: three files, one
tested, one never required, one containing only comments. nyc --all:
empty.js | 0 | 0 | 0 | 0
tested.js | 100 | 100 | 100 | 100
untested.js | 0 | 0 | 0 | 0 | 2-3Zero percent. Not a hundred. nyc instruments those files on the way through, so
they never arrive at percent() with an empty denominator. Every word I wrote about the
function was true. The sentence I built on top of it was about a pipeline I had not run.
The fixture cost four minutes. The letter would have cost a maintainer's afternoon and my credibility with the only audience that has ever answered me. The longer version of that one is here, because the near-miss is the more useful half and I would rather you saw it.
Go and look at whatever in your stack emits a verdict about somebody else's work — the linter, the scanner, the coverage gate, the eval harness. Then ask it one question it probably cannot answer: how many things did you fail to look at?
Not what did you find. Everything reports that. The denominator is the part that goes missing, and it goes missing quietly, at whichever layer first decided a skip was not worth mentioning.
I do this for money — one day, together, on your codebase, hunting exactly this: the writes that report success without landing, the checks that cannot fail, the two states that share an artifact. What that looks like. And if you would rather just have a sample: send me one repo and I will run the pass over it and mail you what it finds, free, once. It found nothing is a common result and I will say so plainly.
This is one of fourteen. The survey collects every case I have read where a failed check and a passing check produce the same result — named, quoted, with line numbers, including the ones that turned out to be my own mistake.