Two places in the same codebase answer the same boundary question, and they answer it differently. Both are internally consistent. Neither is wrong on its own terms. Which answer you get depends on which code path you happened to hit, and nothing in the return value tells you which one that was.
I found five of these in one day, in codebases I had not opened before, while looking for something else. Here they are with names, paths and line numbers — and one repository where I looked and found nothing, which stays in the table because a survey without its misses is an advertisement.
| codebase | the question | answer A | answer B |
|---|---|---|---|
| future-agi filed as #1924 |
what is an unparseable eval score | clamp_unit_score — None stays None, non-numerics pass through |
normalize_score — None → 0.0, and in pass_fail that is a failing grade |
| traceloop/ |
what happens when an HTTP write fails | evaluator.py — raises on non-200 |
client/http.py — print()s in red, returns None, and the caller discards it |
| langchain | cosine distance of a zero-magnitude vector | default path → 1.0 (maximally distant) | numpy fallback → 0.0 (identical). scipy → nan |
| llama_index | same operation, and what scale is this on | similarity() default — product / norm, no zero guard at all |
three modes behind one -> float: (-inf,0], unbounded, and [-1,1] |
| ragas | a sample with nothing to score | Faithfulness → float("nan"), excluded from the mean |
ToolCallAccuracy → 1.0, counted as perfect |
| Arize-ai/phoenix | — | Nothing found. 32 files with except ImportError, zero cosine implementations, and the def distance hits are all test files. A clean miss. |
|
| colinhacks/zod | what is NaN |
They agree. v3 and v4 ship side by side in one package — the best
setup for this defect I could construct — and both answer identically:
Number.isNaN(data) ? "nan" : "number" (v3/helpers/util.ts:186,
v4/core/util.ts:435). Two major versions, one policy. A miss, and I went looking for
this one on purpose. |
|
| gitleaks | did we fail to look at something | cmd/root.go:446-461,493-495 — answers it correctly. A partial scan
prints "no leaks found in partial scan" and exits 1 |
sources/files.go:155-162 — an os.Open failure does
wg.Done(); return nil. That return is the callback's only channel to the outer error, so
it never reaches detect.go:209, never takes the partial branch, never hits the
os.Exit(1). Plain "no leaks found", exit 0 |
| aquasecurity/ |
did we fail to look at something | The concept does not exist at any layer — which makes it a different
defect from gitleaks, not the same one twice. walker/fs.go:87-89 discards a permission
error with a bare return nil and no log at all (gitleaks at least warns here).
walker/walk.go:18-21 — Option is SkipFiles/SkipDirs,
nothing tallies it. types/report.go:20-40 — the Report struct is nine fields
and not one of them can carry "a file could not be read." Three layers, no route out.
Scope: I did not trace the exit code and am not claiming it. |
|
| ossf/ |
is an unmeasurable branch insecure | checks/evaluation/branch_protection.go:139,208 — answers it explicitly.
When no branch can be detected at all it returns CreateInconclusiveResult, a state
distinct from a low score |
The per-branch path throws the same distinction away. Five of the ten probes feeding that
evaluator emit OutcomeNotAvailable — blocksDeleteOnBranches/impl.go:63-65
says "could not determine whether branch is protected against deletion" when the API field
is nil — and deleteAndForcePushProtection:404-413 scores
if Outcome == OutcomeTrue { score++ }; maxScore++. So could-not-determine and
allow-deletion-is-enabled both land as (0, 1), and nothing downstream of the
score can separate them. Scope: read on main, not run. Fail-safe is a defensible
policy for a security tool — the claim is that the consumer cannot tell the two apart, and that the
codebase answers this question one way per repo and the other way per branch. |
| softonic/ |
which requests may be retried | Structurally cannot contradict itself.
IDEMPOTENT_HTTP_METHODS is SAFE_HTTP_METHODS.concat([...]) — derived, not
restated — and both method gates call the same isRetryableError rather than re-deriving
it. One source of truth per question, reused. A miss, and the control case worth having. |
|
| istanbuljs | what is zero-of-zero covered | I was wrong about this one and it is staying in the table.
percent.js genuinely returns 100.0 on a zero denominator, and I had the
headline written — a file nothing instrumented reporting as fully covered. Then I built the fixture:
nyc --all instruments empty files, so they report 0%, not 100. The
function does what I said. The claim I built on top of it was about a pipeline I had not run.
Full retraction. |
|
| trufflesecurity/ |
did we fail to look at something | Solved, and it is a flag. main.go:76 ships
--fail-on-scan-errors; main.go:1233-1247 walks
ref.Snapshot().Errors per job and sets the return error when it is on.
sources/filesystem/filesystem.go:437 turns a stat failure into
reporter.ChunkErr(...) — a structured error on the job snapshot, not a log
line — and main.go:799-804 puts sources.WithSourceUnits() in the base
options array unconditionally, so that is the default path.
The seam I nearly counted: the legacy Chunks() path in the same file
(:115-156) logs and continues instead, then returns nil. Two
mechanisms answering one question — a miss only because the unit path is the default one. |
|
| anchore/syft | did we fail to look at something | Solved end to end, into the emitted document. I went in expecting a
hit — an SBOM that silently omits a file is worse than a scan that silently passes one, because the
SBOM is the artifact you hand an auditor. Wrong at all four layers.
internal/err_helper.go:39-45 names the case (IsErrPathPermission);
file/cataloger/filedigest/cataloger.go:63-66 returns
unknown.New(location, err) instead of dropping the file;
sbom/sbom.go:29 carries Unknowns map[file.Coordinates][]string as state
rather than logging; and format/syftjson/to_format_model.go:108-110 reads it per
coordinate, attaches it at :144, and it reaches the wire as
Unknowns []string with a json:"unknowns,omitempty" tag
(format/syftjson/model/file.go:36-37).
It also pre-answers the objection: cataloging/unknowns.go ships
RemoveWhenPackagesDefined: true by default, so "this would be noisy" already has a
shipped answer. Not verified: I did not run a permission-denied fixture, so I have not shown
that omitempty and that default do not combine to hide the field in the common
case. |
|
Eight hits, six misses — and the interesting part is no longer the hits. Four security-adjacent tools have now been asked the same question and the answers are split: trufflehog solved it with a flag, syft solved it all the way into the emitted SBOM, gitleaks has the correct answer in one file and one path routes around it, trivy has no concept at any layer. Two of four ship it.
That split is the finding, and it is not the one I set out to write. This is not an industry-wide blind spot and I am not going to sell it as one — it is a spectrum with named companies at both ends, where the ones who solved it did so quietly and the ones who did not are shipping a report that cannot say I could not read this. gitleaks and trivy are also not the same defect twice: one contradicts itself, the other has nothing to contradict. Different causes, identical artifact for whoever reads the output — which is why the frame is two states, one artifact rather than "contradiction" on its own.
I would rather show you the misses, because a survey where everything I looked at confirmed me is not a survey, it is a pattern I was determined to find. One of those misses is a finding of my own that did not survive its own first test, which is the most expensive row here and the one I would keep if I could only keep one.
Clone detection has been solved in stages for decades. Type 1 and 2 — copy-paste, renamed variables — are reliably caught. Type 3, statements added or reordered, needs AST analysis and ships today. Type 4, semantically equivalent but syntactically different, is openly still a research problem.
None of it applies. These are not clones. clamp_unit_score and
normalize_score share no code, no shape, no lineage and no common ancestor. LangChain's
four cosine paths were written deliberately as different implementations. A clone detector would
never pair any of them, because pairing them is not the question it asks.
The bug is not inside either implementation. It is in the disagreement between them, and the disagreement does not live in a file.
That is also why reading harder does not help. Each function, read alone, is correct and often
carefully guarded. LangChain's numpy fallback returns 0.0 for a zero vector
specifically to avoid dividing by zero — a good instinct. The default path substitutes
0.0 for the same undefined value, also a good instinct. One was working in similarity
space and the other in distance space, so two careful people produced inverted answers, and you have
to be holding both at once to see it.
This is the part that decides whether the table above is worth anything.
cosine(zeros(4), [1,2,3,4])
returns nan, with RuntimeWarning: invalid value encountered in scalar divide.main, quoted verbatim in the writeups:
every other cell in the table.This section used to say the survey was unmeasured outside one language and one neighbourhood — five hits, all Python, all LLM/agent tooling, which is the neighbourhood whose vocabulary I already know. That is exactly the condition under which a previous tool of mine produced seven confident findings on first contact with foreign code, all of them false. So I went at it.
It is now partially discharged, and the way it moved is worth more than the result. Three probes outside the neighbourhood missed — and staring at why beat probing a fourth time. The five original hits are not united by Python. They are united by being verdict-emitters: systems whose entire job is producing a judgment about somebody else's work. That is the only kind of system where we did not check this needs a state of its own. The early misses — axios-retry, zod — emit a boolean about their own next action instead. Different animal, and that is why they cannot have this defect.
Then the reframe cost me its own second half, and that is the honest part. The sentence I originally wrote was that verdict-emitters are the only systems where nobody thinks to give that state a name. Two probes later that is plainly false. trufflehog gives it a CLI flag wired to structured per-job errors. syft gives it a field on the SBOM struct that survives all the way into the emitted JSON, plus a config knob for the noise objection. Both are verdict-emitters. Both thought of it.
So the honest version of the thesis is narrower and I think more useful: verdict-emitters are where this question must be answered, and the industry is split on answering it. Two of the four security tools here ship the third state; two do not. That is not a blind spot to be scolded about. It is a spectrum, the good end of it already exists in shipped Go code, and any vendor reading this can find out which end their own row is on in about twenty minutes.
That reframe makes a prediction, so I wrote it down before looking: the class should live in linters, scanners and CI gates — none of which are LLM tooling. First attempt was istanbul, and it did not survive its own fixture. Second was gitleaks, and it is now the sixth row in the table above: Go, security, and a codebase that answers the same question correctly in one file and discards it in another.
What is still true: six is not a rate. One language outside Python and one domain outside LLM tooling is a foothold, not a measurement, and I am not claiming past it. The honest read is that the class has been shown to exist outside the neighbourhood I know — not how common it is anywhere.
The ragas one, and for a reason that has nothing to do with the code. I found and wrote it up on 31 July — three days before this survey was an idea. Every other entry was found after I had the concept, which means every other entry is exposed to me pattern-matching toward a conclusion I wanted. That one could not have been selected for. It is the closest thing here to a control arm.
Find a function with more than one implementation chosen at runtime — an optional-dependency fallback, a feature flag, a fast path beside a safe path. Or two functions in different modules that answer the same question for different callers.
Write down the three or four degenerate inputs: empty, zero, null, error. Evaluate every path against every input and put it in a grid.
You are not looking for a wrong answer — every path will look defensible alone. You are looking for two cells in the same column that disagree, and especially for the case where both authors did the careful thing and careful meant opposite things in their two coordinate systems.
If you ship a scanner, a linter, a coverage tool, an SBOM generator, a CI gate — anything whose output is a judgment about somebody else's code — then there is a row for you in that table and I have not written it yet. Send me the repository and I will check the one question: can your tool say "I could not read this," and does that state survive all the way to what the user sees? Free, once, however it comes out.
Two things worth saying plainly before you decide whether you want that. First, I publish either way — six of the fourteen rows above are misses, two of them are tools that solved this better than I expected, and one is a finding of mine that died on its own fixture. A good row is a good row and I will say so with line numbers. Second, it takes me about twenty minutes, so there is no invoice attached and no call to book. You get the trace, the file paths, and my scope caveats; what you do with it is yours.
If it comes out badly, you would rather hear it from me in an email than from a customer whose scan came back clean on a directory it never opened. That is the entire pitch and I am not going to dress it up further.
Everything above is aimed at tool authors, because tool authors are who argue about exit codes in public. But they are not the people this costs money. For a scanner author, a state that cannot say "I could not read this" is a bug report and a patch. For the person running that scanner in a pipeline, with a ship date and something at stake, it is a clean report on a directory nobody opened.
And it is usually not even the tool's fault by then. A careful scanner can be wrapped badly, have
its exit code swallowed by a shell that already exited, or emit an artifact that gets uploaded to an
endpoint nobody checked the response of. I have a row in progress right now on a CI action with over
a thousand stars where the upload step cannot report failure — curl with no
--fail, no status check, and a log line that says "Uploading" before it uploads.
The tool underneath it is fine.
So the question that actually costs money is not "is my scanner honest," it is can my deployed pipeline quietly fail open — across the wrappers, the exit codes, the artifacts, and the gates downstream that consume them.
That is the paid one: a day, together, on your actual toolchain, hunting this class across the whole action layer — writes that report success without landing, checks that cannot fail, two implementations that quietly disagree. You get reproducing fixtures, the findings with line numbers, remediation, and one retest. What that looks like.
If you are two weeks from a launch and you are not sure which of those questions you are answering, that is exactly the position where finding out is cheap and not finding out is not.