A conditional import, a bare except, and three months of nothing

I have a webhook handler that triages incoming events: fetch the record, classify it, and if the classification is one of the safe ones, claim a lock and run the automation. It had been humming along for months. The log table had 21,374 rows in it.

It had executed nothing since June 10th.

Nobody noticed because it looked busy. Every event got classified, every row got written, the dashboard was full. The only thing missing was the part where it did the work.

The tell was in the data, not the logs

The handler is deliberately quiet — it prints on failure, and it wasn't failing. So the logs said nothing useful. What actually gave it away was a question I only thought to ask because the number was suspiciously round:

locks = [i for i in all_rows if i["id"].startswith("exec-lock-")]
print(len(locks))   # 0

Zero. Not "few". Not "fewer than expected". Every execution path starts by writing an exec-lock-<id> row, and in the entire history of the table there had never been one. That turns a vague "is this working?" into a precise "this specific line has never run."

One wrong key

The lock write sits behind a freshness check:

is_new = False
try:
    entered = record.get("dateEntered", "")
    if entered:
        from datetime import datetime, timezone
        age = (datetime.now(timezone.utc)
               - datetime.fromisoformat(entered)).total_seconds() / 60
        is_new = age < 10
except Exception:
    is_new = False

The vendor's REST API doesn't put the created timestamp where I looked. It hangs off a metadata sub-object — record["_info"]["dateEntered"] — and the top-level key I asked for simply isn't there. So entered was always "", and the if was never true.

That alone is a boring bug. It cost me the freshness gate. What made it expensive is the line inside the if.

Python does not care that your import is conditional

An import inside a function binds that name as a local for the entire function — not from the import statement onward, and not only if the branch runs. The compiler decides at compile time. So:

from datetime import datetime, timezone   # module level

def handler(record):
    if record.get("nope"):
        from datetime import datetime, timezone   # never executes
    return datetime.now(timezone.utc)             # UnboundLocalError

UnboundLocalError: cannot access local variable 'datetime'
where it is not associated with a value

The module-level import is right there. Doesn't matter. Inside handler, datetime is a local, and that local was never assigned.

Two hundred lines later my handler did this:

try:
    table.put_item(Item={"id": f"exec-lock-{rid}",
                         "locked_at": datetime.now(timezone.utc).isoformat()},
                   ConditionExpression="attribute_not_exists(id)")
    should_exec = True
except Exception:
    should_exec = False   # "safe default"

Every event, UnboundLocalError. Every event, swallowed. Every event, should_exec = False. The comment even calls it a safe default, which is how it survived code review — it reads as caution and behaves as an off switch.

Why no test caught it

Because no test could import the module. It contained this:

f"{'LICENSE REQUIRED\n\n' if out_of_licenses else ''}"

A backslash inside an f-string expression is a syntax error before Python 3.12. The deploy target was 3.12 and shipped fine; my local interpreter was 3.11, so import handler_module blew up at parse time and every test that would have touched this file quietly wasn't written. The one function with a three-month outage in it had zero coverage, and the reason was a formatting shortcut in an unrelated function eight hundred lines away.

Hoisting two strings out of their f-strings fixed the import. The tests I then wrote assert the thing that was actually missing:

def test_exec_lock_is_actually_written(wired):
    run_webhook(ticket(id=999002), "server_offline")
    assert "exec-lock-999002" in wired["store"], \
        "exec lock never written -- execution path is dead again"

Put the old bug back, and four of them fail. That's the bar: a test that can't fail on the original defect is decoration.

What I'd do differently

Never import inside a conditional. There is no case where it's worth it. If you want a lazy import, do it unconditionally at the top of the function, or import module and use module.thing.

A bare except Exception around a line that decides control flow is a silent off switch. If the fallback sets a boolean that gates whether any work happens, log the exception. Mine would have said UnboundLocalError in plain text on day one.

Monitor the expected result, not the run. "Did the handler execute" was true 21,374 times. "Did anything get locked and worked" was false every single time. Assert the side effect you actually want.

And run the dumb query occasionally: how many of X have ever existed? Zero-of-anything-ever is the cheapest bug detector I know, and it works on systems whose logs look perfectly healthy.