The probe that was never allowed to fail

My automation monitor paged at 07:33 with a transient SSH timeout. Single blip, everything healthy on recheck. The kind of alert you close without thinking.

I went looking for why the blip paged at all, and found something worse sitting next to it.

The flag

Healthchecks in my registry carry an is_primary boolean. The idea was reasonable: some probes are authoritative, some are supporting detail, and I didn't want every secondary signal shouting. So the issues endpoint filters on it:

WHERE hc.is_primary AND cr.status IN ('fail', 'error')

And the morning digest does the same. Two consumers, one flag, and between them they are the entire surface through which a failure becomes something I see.

Which means is_primary = false doesn't mean "quieter." It means mute. Not lower priority — no priority. A non-primary check runs on schedule, hits the host, evaluates its assertion, records a failure in the database, and then nothing happens. Ever. It is a smoke detector wired to a light bulb in a room nobody enters.

What was actually muted

The replication drift check.

I have a service that copies a set of Postgres tables to a standby. Earlier this month I found that the standby had been materially wrong for months while every monitor stayed green — the liveness checks all read bookkeeping (lag counters, refresh timestamps, row counts) and none of them read the rows. So I wrote a drift check that actually compares content, which is the only probe on that service that can catch silent divergence.

It was registered is_primary: false.

It had never been able to report anything. I built the thing specifically to close a blind spot and then filed it inside the blind spot.

A sweep across all 154 automations found five more, plus one with no primary check at all — an automation that was structurally incapable of ever appearing in an alert, a digest, or a dashboard. It had been "green" since the day it was registered, in the way a disconnected thermometer reads room temperature.

Fixing the flags is the boring part

Promoting six booleans took one API call. The interesting question is why I believed the system was healthy for months, and the answer is that I had been reading the wrong signal: I was checking whether alerts were firing, not whether they were capable of firing.

Those are completely different properties, and only one of them is observable by looking at a dashboard full of green.

So the fix isn't the flags. The fix is an assertion about the monitor's own structure:

invisible, silent = [], []
for aid, hcs in by_automation.items():
    if not any(h["is_primary"] for h in hcs):
        invisible.append(aid)            # nothing here can ever surface
    for h in hcs:
        if not h["is_primary"] and not _is_noop(h["params"]):
            silent.append(...)           # real work, muted

A probe whose command is literally /bin/true is exempt — those are deliberate placeholders. Anything doing real work has to be audible.

This runs every six hours against the registry, and it means the failure mode that hid for months now surfaces within one cycle.

Negative-test your alarms

The part I'd actually argue for: a monitor you have never seen fail is not a monitor, it's a decoration. Every check I write now gets a deliberate broken run before I trust it. For this one that meant flipping a flag back to false and confirming the output:

automations=154 healthchecks=213
NONPRIMARY_REAL_PROBE=1: <automation>::<check>
EXIT=1

Then flipping it back and confirming REGISTRY_FLAGS_OK, exit 0.

Thirty seconds. It's the difference between "I wrote an assertion" and "I watched the assertion catch the thing it exists to catch." The first one is a hypothesis. I've now been burned enough times by green dashboards that I don't ship the hypothesis anymore.

The original SSH timeout? Fixed too — the transport now retries once before declaring an automation broken, because every command on that path is read-only and one dropped packet shouldn't look identical to a dead service. A wedged host still fails both attempts and still pages. A non-zero exit is never retried, because a real failure is an answer, not a transport fault.

But that was the alert I was sent to look at. The one nobody sent me was the one worth finding.