dig writes its errors to stdout, and my monitor believed them

I have a job that keeps a handful of A records pointed at my house's residential WAN address, and a probe that asserts the result — it asks public resolvers what they actually hand out, rather than asking the DNS API what the zone says. The API tells you what you wrote. A real lookup tells you what clients get.

The probe paged at page severity with this:

FAIL  host.example.com: 1.1.1.1 answers
      [';; communications error to 1.1.1.1#53: timed out',
       ';; no servers could be reached'],
      WAN is 203.0.113.9

DNS was fine. The four runs on either side of it were green. What happened is that one UDP packet went missing, and my parser turned that into an accusation.

The bug

def resolve(name, resolver):
    out = subprocess.run(
        ["dig", "+short", "+time=3", "+tries=1", f"@{resolver}", name, "A"],
        capture_output=True, text=True, timeout=12)
    return [ln.strip() for ln in out.stdout.splitlines()
            if ln.strip() and not ln.strip().endswith(".")]

dig +short prints its own diagnostics on stdout, not stderr. So when the exchange fails, stdout isn't empty — it contains ;; communications error to 1.1.1.1#53: timed out. My filter was written by exclusion: keep every line that isn't a trailing-dot CNAME target. ;; communications error... doesn't end in a dot, so it sailed through as an A record, compared unequal to the real WAN address, and became a fault.

Parsing by exclusion is the actual mistake. "Keep the lines that don't look like the thing I know about" assumes you know every shape the output can take. You don't. Validate by shape instead:

for ln in out.stdout.splitlines():
    try:
        ipaddress.IPv4Address(ln.strip())
    except ValueError:
        continue          # CNAME targets, dig chatter, anything else
    addrs.append(ln.strip())

The bigger bug

Filtering the noise out isn't enough, because it only converts a wrong answer into an empty one — and the probe treated "no addresses" as a fault too. It still would have paged, just with a less interesting message.

A monitor has to be able to say three things, not two:

• I looked, and it's fine.
• I looked, and it's broken.
• I couldn't look.

That third state is the one everybody skips, and it's where the 3am pages come from. So resolve() now returns (addresses, resolver_was_reachable), and a name that no resolver would answer for goes into a blind list rather than a problems list. Real faults still exit 1 and page. Blindness exits 2, which my sentinel surfaces as "probe could not run" — visible, not alarming.

if problems:
    return 1
if blind:
    for b in blind:
        print(f"BLIND {b}")
    return 2

Note the ordering: faults win. If one name is genuinely wrong and another was unobservable, that's still a page. Blindness only decides the outcome when nothing observed was bad.

The fix that broke the fix

Then I added a retry, because one lost packet shouldn't make the probe blind either. Each resolver gets a second attempt two seconds later before it counts as unreachable.

That's when the timing test blew up. Nine names, two resolvers, two attempts each, dig +time=3 +tries=2 = up to six seconds a shot, plus backoff. Against a black-holed resolver: about three minutes. My monitor kills a check at its command timeout and reports it as ERROR — which pages. I had spent the whole afternoon building a careful "I couldn't see" path and then routed around it with a stopwatch.

The fix is that unreachability is a property of the resolver, not of the name:

_dead_resolvers: set[str] = set()

def resolve(name, resolver):
    if resolver in _dead_resolvers:
        return [], False
    ...
    _dead_resolvers.add(resolver)   # after the last attempt fails
    return [], False

One name's worth of waiting per resolver, not nine. Measured after:

That last row is still uncomfortably close to a 20-second default timeout, so the check is now registered with an explicit 45-second budget. A probe that knows it can be slow should say so, rather than being killed and having that read as the automation being broken.

What I'd do differently

Write the three-state contract first. I wrote this probe with a docstring that already said exit 2 means "the probe could not determine the truth, which must not read as DNS being wrong" — and then the code didn't implement it, because the failure mode I pictured was "the API call throws," not "the CLI helpfully narrates its problems onto stdout where my regex lives."

The docstring was right. It just wasn't tested. The test that would have caught all of this is four characters of config: point the probe at 192.0.2.1, a reserved address that answers nothing, and assert the exit code.