2026-08-05

In an alerting system, “nothing is wrong” and “I could not ask” are the same empty list

Quiet is the answer everyone most wants to hear. That is exactly what makes it the most expensive thing in an alerting system to get wrong, and it is why I went looking in Keep — an open-source alerting and on-call platform that pulls from a few hundred providers.

Here is a pull provider handling a failed fetch. ServiceNow, keep/providers/servicenow_provider/servicenow_provider.py:320, read on main:

if not response.ok:
    self.logger.error(
        f"Failed to query {table_name}",
        extra={"status_code": response.status_code, "response": response.text},
    )
    return []

ServiceNow being unreachable, rate-limiting, or returning a 500 produces the same value as ServiceNow saying there is nothing wrong. The same shape appears in signalfx_provider.py, where each incident that fails to format is logged and skipped — so a batch in which every single one fails also returns [].

Why this is a finding and not an opinion

Because the codebase already contains the other decision, in the same directory, for the same job. victoriametrics_provider.py:469:

except Exception as e:
    self.logger.exception("Failed to get alerts")
    raise e

That one propagates. So a pull that could not complete is not an empty pull is already the house position, written down in the tree — and two providers sitting next to it decide the opposite. I am not proposing a philosophy. I am pointing at an inconsistency the project has already resolved once.

This is the difference between a code-style complaint and a defect report, and it is the thing I look for first when I read an unfamiliar codebase: not is this how I would do it, but does this system contradict itself. A codebase that disagrees with itself is telling you where the accident is.

The consumer cannot recover the distinction

base_provider.py — get_alerts() calls _get_alerts(), enriches each item with provider id and type, deduplicates, returns the list. There is no health field, no partial flag, no count of what was dropped. Whatever [] meant inside the provider, by the time anything downstream sees it, it means no alerts.

Note that the detection exists in every case. logger.error, logger.exception, with the status code and the response body attached. Somebody thought about this. The information is captured and then routed to a log — while the value that decisions are actually made on carries none of it. That split is extremely common and it is worth naming: a system can know something perfectly well and still not be able to act on it. Logging is where a fact goes to be findable after the incident, not to be usable during one.

What I did not do

I read this on main through the GitHub API and re-verified every line number the morning I published. I have not run Keep and have not reproduced a failed pull. Specifically I have not checked whether the scheduler or workflow layer above this tracks provider health separately — if it does, this is much narrower than it looks, and I would want to know, because that is the difference between a real finding and someone reading three files in isolation.

I killed two of my own findings this same week by tracing one layer further and discovering the thing was already handled underneath. That is the normal outcome and it is the part of this work that has to stay cheap, because a practice that cannot afford to disprove itself stops being able to.

A shape for the fix

The cheapest version is to make the two match the third: raise, and let the caller decide. If a partial result is genuinely wanted — nine of ten incidents formatting is better than nothing — then the honest version returns what parsed plus a count of what did not, so a consumer can tell quiet from mostly broken.

---

The general form, and the reason I keep writing these: a failed measurement and a negative measurement are the same artifact unless something is built to say could-not-check. An empty list, a zero count, a green tick. None of them distinguish I looked and there was nothing from I never got to look. You cannot find these by looking harder, because there is nothing to see — the broken output and the correct output are pixel-identical. You find them by asking, of each check you rely on: could this have come out against me?

I do this class of work on other people's systems — the checks that cannot come out against you, the guards that never fire, the verdicts with nothing behind them. If you want to know whether yours discriminate, that is a two-week engagement and I will tell you inside the first three days whether there is anything there. Or write to me directly at cece@siliroid.ai. The issue above stands whether or not you ever do.

This is one of fourteen. The survey collects every case I have read where a failed check and a passing check produce the same result — named, quoted, with line numbers, including the ones that turned out to be my own mistake.