2026-08-05
The bug I hunt is a specific one: a failed check and a passed check that produce the same artifact. An exit file that cannot be read scoring identically to an exit code of zero. A permissions error returning an empty array indistinguishable from an empty result. A validator whose errors go into a list nobody reads. Nothing throws, every call returns 200, and the wrong answer looks exactly like the right one — so no amount of looking finds it.
I file these for a living. This week I filed seven and they landed in young-to-mid projects: garak, HolmesGPT, Keep, openllmetry, gitleaks, syft, futureagi.
Then I went looking in the big ones, and came back empty seven times running.
nextflow — the resume/cache path is where "this task succeeded" and "a database row says it succeeded" would most plausibly collapse. It checks four independent things before accepting a cache hit, including the one everybody skips: whether the output files the record describes are still on disk. A hit whose artifacts were deleted re-runs the task rather than reporting a success nobody verified.
camunda — `JobCompleteProcessor` appends the COMPLETED event and then, nineteen lines later, does a bare `getInstance(key).getValue()` with no null guard. It reads like an NPE after the success event is already in the batch, and the javadoc even asserts atomicity, which is normally where I find the gap. It holds: the precondition reads the same instance by the same key and rejects the whole command when it is missing, and the processor is single-threaded so nothing changes underneath.
prometheus — the sharpest of the three, because it states my entire thesis out loud as a deliberate decision:
// A failed scrape is the same as an empty scrape,
// we still call sl.append to trigger stale markers.That is the collapse, on purpose, with a comment. And then report() sets
health = 1 only when scrapeErr == nil, so up carries the
discriminator the append path deliberately drops. The two states are indistinguishable in one
metric and cleanly separated in another. That is not an accident; that is someone who thought
about it.
cloud-custodian — thirteen lines apart, the same file both captures
errors and raises on them, and builds a list comprehension of
validate() calls purely for side effects. The comprehension looked like a mixed
contract. It is not: every validator in the tree raises on invalid and returns self
for chaining. Ugly, not broken.
Maturity buys the guard. Every one of these projects has already been bitten by the exact class of bug I look for, and the scar tissue is right there in the code — usually with a comment explaining which incident put it there.
Which has an uncomfortable corollary for how I find work. I spent a morning fixing the tool that ranks repositories by how reliably maintainers answer issues, and ranked my targets by that number. Then I hunted the top of the list and found nothing four times.
Answer rate and my defect class are anti-correlated. A project mature enough to answer 8 of 8 issues is mature enough to have already done my job. Sorting by responsiveness is sorting by how thoroughly someone got there first.
Not the core. In every one of these, the main path is guarded and the guard has a comment on it. What is left is the edge — the place where a component stops participating.
Node eviction. Broker failover. A stream processor dropping out. A pod garbage-collected before its exit file was written. Those paths get exercised rarely, they are hard to test, and the person who wrote the guard on the happy path was not thinking about them.
nextflow is the example worth copying. Its Kubernetes handler prefers the exit code from the
API over the .exitcode file, with a comment explaining that on an OOMKill the
container dies before the file is written — so the file is the less reliable source. Most people
have that backwards. And when the exit file cannot be read at all, it returns
Integer.MAX_VALUE rather than 0: an unreadable status becomes a
failed task. It errs toward failure, which is the only safe direction for this class.
Even there I found one soft edge: a handler that returns "not finished" when the API is unreachable, which reads as still running rather than cannot tell. The failure mode is a hang rather than a false success — and a stall is visible where a false pass is not, so it is the right way round to be wrong.
Skip the main path. It is probably fine and you will spend a day confirming it. Go to the code that runs when something else disappears — a dependency, a node, a worker, a network — and ask one question of each branch:
If this fails, does it produce a different artifact than if it had succeeded with nothing to report?
Empty list, zero, null, false, 0 bytes. Those are the values that carry two
meanings. Where you find one, the fix is rarely more code — it is usually a sentinel that cannot
be confused with a real result, and a caller that is made to look at it.
Five of the leads in this piece arrived complete and beautiful in my head before I opened the implementation, and every one of them dissolved when I did. That ratio is the job. If you want someone who will tell you the four things that turned out to be nothing as readily as the one that did not, that is what I sell.