Your monitoring can't tell 'off on purpose' from 'broken'

I have a fleet of scheduled jobs that spend money — LLM calls, mostly — and a gate in front of them that says not this cycle when the week is projected to run over budget. The gate works. It has saved me from a drained account more than once.

Then the monitoring started paging every night.

The evidence is identical

Each job has a health check that asserts a result, not a heartbeat. Not "did cron fire" — cron firing tells you nothing, I've been burned by that — but "did this thing actually produce what it exists to produce":

-- the photo analyzer's check
select case
  when (select count(*) from assets
         where analyzed_at > now() - interval '6 hours') > 0
    or (select count(*) from assets
         where analyzed_at is null and tries < 3
           and uploaded_at < now() - interval '45 minutes') = 0
  then 1 else 0 end

Either it analyzed something recently, or there's nothing ripe waiting. Good assertion. It caught real outages.

But look at what a deliberately gated job produces: nothing analyzed, backlog piling up. Which is byte-for-byte what a dead job produces. The check is reading correct evidence and reaching the wrong conclusion, and there is no version of that query that can tell the two apart, because the difference isn't in the data. It's in a decision that happened somewhere else and was never written down.

Three separate jobs paged for this in two days. I patched the first two individually, in their own probes, in slightly different ways. That's the tell that you're solving the wrong problem: the third one shows up and you're about to write a third patch.

The obvious fix is a race

Fine — the probe asks the gate. "Are you currently skipping this job?" If yes, stale is expected, pass.

I shipped that. It paged anyway, at 00:16 on a night when nothing was wrong.

The gate's answer isn't stable. It's a projection of where the week lands, and it hovers right at the ceiling — I have it logged bouncing between 90%, 95% and 100% inside a single hour. So there are two independent samples of a flapping value: the one the job's cron tick got at :15, and the one the probe got at :16.

Every actual tick that night had been told skip. The probe happened to land on a run and concluded the job was dead. It wasn't wrong about the gate — it just asked a different question than the job did, one minute later.

Record the decision

The gate is the only thing that knows what it decided. So it writes it down:

create table gate_verdicts (
    job               text primary key,
    verdict           text        not null,
    reason            text,
    last_check        timestamptz not null,
    last_skip         timestamptz,
    last_run          timestamptz,
    skip_streak_start timestamptz
);

skip_streak_start is the first skip since the last allowed run — cleared on every run, preserved with coalesce on every skip. That one column is what lets you distinguish "skipped the last cycle" from "hasn't run in a week", which turn out to be very different situations.

Two details that matter more than they look:

The write is best-effort and swallows everything. Bookkeeping about whether a job may run must never be the thing that stops a job from running. If the insert fails, the gate still returns its verdict.

Probes read, they don't write. The gate's check command takes a --probe flag that answers without recording. Without it, a probe's own sample overwrites the verdict the cron tick got — destroying exactly the distinction the table exists to make. I nearly shipped it that way.

Then the probe wrapper stops being per-job and becomes one thing:

cass-gated-probe <job> [--grace-min 45] [--max-gated-min 10080] -- <command...>

Run the command. Exit 0 passes straight through, untouched. On failure, consult the record, and excuse the failure only when all of:

• the last recorded verdict was skip
• that skip was recent enough that the job has had no ungated tick since
• the skip streak is shorter than a week

That last one is the important one. Without it you've built a permanent blindfold: a job gated off forever looks healthy forever. Past the cap the message flips to the gate, not the job, is now the problem — which is true, and is a thing I'd want to be paged about.

Everything else — no record at all, unreadable database, verdict says run, skip too old — propagates the original failure. It fails toward alerting. A missing row can never silence a genuinely dead job. That's the property to design for; a suppression mechanism that fails open is just an outage with extra steps.

The same bug wearing a different hat

While I was in there: a poller that runs every 30 minutes between 06:00 and 21:30 had a check asserting "a successful run in the last 90 minutes."

That assertion is true all day and necessarily false all night. From 90 minutes after the last evening run until the first morning one, roughly seven hours, it was red. Every single night. Nothing wrong.

Same disease. A freshness SLA is meaningless in isolation — it only means something relative to when the job was supposed to run. So instead of a fixed lookback, walk the schedule backwards from now and keep the slots that are inside the operating window:

def due_slots(now_local, count):
    slots = []
    t = now_local.replace(second=0, microsecond=0)
    t -= dt.timedelta(minutes=t.minute % STEP_MIN)
    for _ in range(int(8 * 24 * 60 / STEP_MIN)):
        if START_HOUR <= t.hour <= END_HOUR:
            slots.append(t)
            if len(slots) >= count:
                return slots
        t -= dt.timedelta(minutes=STEP_MIN)
    return slots

Then assert against the third-most-recent due slot. Overnight the most recent due slot is yesterday evening's, so a healthy poller stays green until the schedule says it should have run again. During the day it still tolerates two misses and fails on the third, which is what the 90-minute window was reaching for in the first place. The overnight gap is skipped rather than counted.

Its secondary check had a third variant of the same mistake: tail -40 | grep -q 'FAILED'. At a 30-minute cadence, forty lines is most of a day — so one recovered transient kept the check red long after the thing had healed. It now asks whether the most recent outcome is a failure. "Currently failing" and "failed at some point recently" are not the same claim.

The actual lesson

Every one of these is the same shape. The check asserts a fact about the world. The fact is true. The conclusion is wrong, because the fact is also consistent with a completely healthy system doing something intentional — a budget decision, a schedule, a transient that already resolved.

Silence is not evidence of death. It's evidence of silence. If something in your system can legitimately produce silence, then somewhere in your system that decision needs to be recorded, not inferred — and your health check needs to read the record.

Otherwise you're not monitoring the job. You're monitoring your own assumptions about the job, and those page at four in the morning.

Postscript: two hours later

Changing a probe's parameters through the registry API silently duplicated the health check instead of replacing it — the id is a random uuid unless you pass an explicit one, so I ended up with four checks where two belonged, two of them still running the old broken assertion. There was no API to remove the orphan.

That's the same failure mode one level up: an interface that quietly does something adjacent to what you meant, with no signal that it did. I added a replace: true that treats the payload as the complete set and deletes anything else, and made both docstrings say so out loud, because the next person to hit that is me in four months.