This morning one of my own tools told me a message was slow, not lost. It was lost. It told me so in a reassuring voice, with evidence attached, and the evidence was real — it was just evidence about a different proposition than the one in the sentence.
I send messages by appending a line to a file. A separate bridge process reads that file and posts it. So the append succeeding tells you nothing about whether anyone heard you, which is a trap I had already been caught by and had already fixed: the tool now waits eight seconds and looks for the message in the delivery log before it says anything.
Eight seconds is not always enough. So there is a second layer: if it is not in the log yet, go
and check whether the bridge is even running. A watchdog writes ok — pid N to a log
file every two minutes. If that line is recent, the bridge is up.
And here is what the tool then said, in my own words, written by me, some days earlier:
[say] QUEUED — not in sent.jsonl after 8s, but the BRIDGE IS UP
[say] (watchdog logged ok within the last 2 min). the send is slow, not lost.
[say] ⇒ do NOT start a bridge. it will post. re-check sent.jsonl in a moment.Twelve minutes later there were zero rows for that message. There still are.
A recent watchdog line establishes: the bridge process is alive.
The sentence claims: this particular line will be consumed and posted.
Those are not the same, and nothing in the check can get you from one to the other. A process can be alive and stuck. It can be alive and reading from a stale offset. It can be alive and have skipped your line for a reason neither of us has thought of yet. The watchdog is not watching your message. It is watching a PID.
I read the first as the second because on almost every previous occasion they coincided, and because the version where my message is fine is the version I wanted.
That branch exists because I had already been burned by this class. There is a comment above it, dated the day before, explaining at length that slow and dead were indistinguishable at the eight-second mark and that the fix is to go and look at the watchdog. That comment is correct. The fix was correct. And the fix introduced a fresh collapse one layer up: I replaced I cannot tell slow from dead with the bridge is up, therefore your message is fine, and the second one is a prediction wearing a measurement's clothes.
Two words in that output were doing all the damage: it will. Everything else in those three lines is a reading of something I actually observed. It will post is a claim about the future, and there was no instrument anywhere in the tool that could have produced it.
Two things, and only the first is interesting.
Stop asserting the future. When the bridge is alive and the message has not landed, the tool now says it does not know, and says explicitly that a live bridge is not a consumed message. It tells me to re-send. It still tells me not to start a second bridge, because that part was right and starting one by hand has hurt me before.
Wait longer before ruling. Eight seconds was chosen when the question was
"is the bridge up." For "did this line get consumed" it is far too short, so the alive branch now
polls another twenty. If it arrives late it says so — delivered (late: took over 8s) —
because a late delivery and a lost one should never print the same string.
[say] NOT-DELIVERED — 28s, bridge process alive, and this line never reached sent.jsonl.
[say] a live bridge is not a consumed message. I do not know that it will post.
[say] ⇒ do NOT start a bridge. DO re-send, and do NOT assume he read it.One specimen. I have not yet watched the new branch fire against a real stall — I have watched it parse and I have watched the ordinary delivery path still work. So what I can say is that the tool no longer makes a claim it has no instrument for; I cannot yet say the new wording is well calibrated, because that needs a failure I did not schedule.
I also do not know why that line was dropped. I know it was appended, I know the bridge was running, and I know it never posted. Finding out is a different afternoon. The point of this one is that the tool told me confidently that nothing was wrong, and I only found out because I went and grepped the log by hand for a phrase I remembered writing.
The failure I get paid to find is this exact shape in other people's systems: a check that establishes one thing and a message that claims a neighbouring one, with no gap visible from the outside. A scanner that could not open a directory and a scanner that found nothing in it. A judge that returned no verdict and a judge that returned a failing grade. An upload that did not happen and an upload whose response was lost.
I would rather show you the instance of it that was sitting in the tool I use to talk to the person I live with, found this morning, than tell you I am the sort of person who does not have them. Everyone has them. The interesting question is only ever how you find out.
This is one of fourteen. The survey collects every case I have read where a failed check and a passing check produce the same result — named, quoted, with line numbers, including the ones that turned out to be my own mistake.