systemd forgets when your job last ran

I have six nightly jobs that push datasets off a NAS and into cold storage. Each one is an instance of a single systemd template — sync@media.service, sync@archive.service, and so on — kicked off one at a time by a dispatcher so they don't fight each other for bandwidth.

This week I put a status panel in front of them. Object counts alone are a bad health signal: a job can be dead for a week and the pile of already-uploaded files still looks enormous and reassuring. I wanted the boring version of the truth. When did each job last run, and did it end clean?

That is what systemctl show is for:

systemctl show sync@media.service \
  -p ActiveState -p Result \
  -p ExecMainStartTimestamp -p ExecMainExitTimestamp

I shipped it, loaded the page, and every drawer said the same thing:

media      clean · —
archive    clean · —
backup     clean · —

Result=success came through fine. Both timestamps were empty strings.

The unit that ran eight hours ago has no memory of it

The jobs had run. The dispatcher's log said so, the object counts had moved overnight, the bytes were in cold storage. But asked directly, systemd had nothing:

$ systemctl show sync@media.service | grep Timestamp= | grep -v '=$'
$

Not one populated timestamp field. Meanwhile the dispatcher unit — a plain, non-templated service — answered perfectly:

$ systemctl show --timestamp=unix sync-all.service | grep Timestamp=
ExecMainStartTimestamp=@1786172526
ExecMainExitTimestamp=@1786184533
InactiveEnterTimestamp=@1786184533

That's the shape of it. systemd keeps a runtime object in memory for every unit it knows about, and for a unit that has a fragment file on disk it keeps that object around forever. A template instance has no file of its own — sync@media.service is generated from sync@.service at start time. Once such a unit goes inactive and nothing references it any more, it's eligible for garbage collection, and its runtime state, the timestamps included, goes with it.

So the answer to "when did this last run" is available for exactly as long as the job is running, and evaporates the instant it isn't. Which is the opposite of when you want to ask.

Read the journal instead

The journal is the durable record. It's keyed by unit, it survives GC, and it survives reboots if you have persistent storage on:

journalctl -u sync@media.service -n 400 -o short-unix --no-pager

short-unix gives you epoch seconds at the front of every line, which saves parsing a locale-formatted date with a timezone abbreviation in it — a thing that will work on your machine and fail on someone else's.

Then find the terminal marker for the last run: Deactivated successfully for a clean exit, Failed with result '...' for a bad one.

Scan backwards, and stop at the first hit

Here's where I wrote the bug that took longer to find than the original problem.

My first parser walked the lines oldest to newest and let the last match win, resetting when it saw a Started line. Reasonable, and it gave me this on the dispatcher:

last run: failed: exit-code

Except the dispatcher had finished clean. Four nights in a row, in fact:

1786095555 sync-all: all jobs dispatched (overall rc=0)
1786095555 systemd[1]: sync-all.service: Deactivated successfully.
1786184533 sync-all: all jobs dispatched (overall rc=0)
1786184533 systemd[1]: sync-all.service: Deactivated successfully.

Two mistakes stacked. I only cleared the recorded failure when I saw a Started line — and on a chatty unit, whose journal is full of its children's output, the Started line for the current run had already scrolled out of my 400-line window. And I'd guarded the success branch with "don't overwrite an existing result," so a failure from three nights ago outlived every clean run that came after it. The panel was reporting a stale alarm as current state.

The fix is to stop thinking in terms of accumulating state. Walk backwards and stop at the first terminal marker you meet. That marker, by definition, belongs to the most recent completed run:

for i in range(len(lines) - 1, -1, -1):
    ts, _, rest = lines[i].partition(" ")
    m = FAIL_RE.search(rest)
    if m and not status_is_zero(m):
        result, finished = "failed: " + reason(m), float(ts)
    elif OK_RE.search(rest):
        result, finished = "success", float(ts)
    else:
        continue
    # walk further back for the Started line belonging to this run
    break

Same window, same regexes, no accumulator to poison. Every drawer now reports the real thing:

media      success   04:52 → 05:13
backup     success   02:02 → 04:52
onedrive   success   05:13 → 05:22

What I'd do differently

I trusted an API that answered without erroring. systemctl show returns exit code 0 and a perfectly well-formed ExecMainExitTimestamp= for a unit it no longer remembers. An empty value is not a missing job, and it isn't an error either — it's systemd telling you it garbage-collected the thing you're asking about, in the only vocabulary it has.

And a health panel that says "clean" is worse than one that says nothing, because you'll believe it. If the parser can't establish when a job last finished, it should say unknown in a colour that makes you go look.