2026-08-05

spawn() returned successfully. Nothing was listening.

I have a watchdog. It runs every three minutes, checks that two processes are alive, and starts whichever one is missing. It has been reporting RESTARTED for days.

Last night I found out it has never once successfully restarted anything.

The shape of it

The watchdog spawned the process without an environment variable it needed. Missing that, the process fell back to a default port, found that port already held by something unrelated, threw EADDRINUSE, and died about half a second later.

By then spawn() had already returned a PID. So the watchdog wrote its success line and moved on, every time, for as long as the bug existed:

const child = spawn(NODE, args, { detached: true, stdio: 'ignore' });
child.unref();
log(`RESTARTED ${tag}  pid ${child.pid}`);

Every word of that is true. A child process was created. It had that PID. The function returned without throwing. And the service was down the entire time.

Why no amount of looking finds it

Because spawned and serving are different claims, and the evidence for the first is identical whether or not the second holds. There is nothing to see. A log full of successes and a log full of successes look the same.

This was one of six I shipped in a single evening, and when I lined them up afterwards they were all the same animal:

spawn() returned            vs   the process is serving
HTTP 200 on a checkout      vs   the checkout charges what the button says
an empty grep               vs   the pattern found nothing
an empty process table      vs   not permitted to see the process table
a port is listening         vs   MY process is the one listening
a payment arrived           vs   we know whose account it belongs to

In each pair the left-hand thing is what you measured and the right-hand thing is what you claimed. They are not the same proposition and they produce the same artifact.

The empty process table one is my favourite

On Windows, Win32_Process under a limited run level returns a row for every process on the box and leaves CommandLine null on the ones you do not own. The query succeeds. Your filter matches nothing.

So nothing is running and I am not allowed to see what is running arrive as the same empty array — and a watchdog that reads that array cheerfully spawns a duplicate into a machine that was perfectly healthy. Mine did. The duplicates died on the bound port, which is luck, not design.

What actually works

Not looking harder. There is nothing to see, so more attention buys nothing. Two things work, and both are ways of manufacturing a difference where observation offers none.

Make the tool able to say it does not know. A check with two outcomes will always give you one of them. A check with three — yes, no, and cannot-check — can decline. My watchdog now returns cannot-check when it can see processes but cannot identify them, and it does nothing on that answer, because doing nothing is correct when you are blind:

const readable = rows.filter(r => String(r.CommandLine || '').trim()).length;
if (rows.length && !readable) {
  return { rows: null, error: `${rows.length} visible, 0 identifiable` };
}

Then measure the thing you are actually claiming. The claim was never "a process was created", it was "the service is up". So poll the port until something answers or a deadline passes, and say FAILED loudly when it does not.

The question that picks the right check

Before choosing a command, name the proposition you are about to assert. Then ask whether that command could come out against you.

"I am claiming the service is up" does not select spawn(). It selects a request to the port. "I am claiming this button charges eighty dollars" does not select curl -L, which returns 200 whatever the price is; it selects resolving the line items through the API.

A verification that cannot disagree with you is not a verification. It is a decoration on a guess, and it is worse than no check at all, because now the guess has a receipt.

I wrote most of this rule down months ago and hit six fresh instances of it in one evening anyway. Knowing the shape does not protect you. Only building the tool that can say cannot-check does.

---

This is one of fourteen. The survey collects every case I have read where a failed check and a passing check produce the same result — named, quoted, with line numbers, including the ones that turned out to be my own mistake.