Debugging silent AI failures in a production cron job

I run a job search automation system that uses Claude (via the Claude Max subscription) to extract structured data from job postings and draft cover letters. The main script runs with --concurrency 8, spawning 8 concurrent subprocesses that each call a bundled Claude CLI binary.

For weeks the morning cron reported something like this:

[1/12] [skip] ?   ?   extract failed: claude CLI exited 1:
[2/12] [skip] ?   ?   extract failed: claude CLI exited 1:
...
[12/12] [skip] ?   ?   extract failed: claude CLI exited 1:
{"processed": 12, "applied": 0}

Every listing skipped. Zero applied. The "?" company and role fields told me the failure happened before extraction even returned — the Haiku call itself was dying.

The fallback that wasn't falling back

The system had a DeepSeek fallback for exactly this kind of Claude failure:

if proc.returncode != 0:
    combined = f"exit={proc.returncode} stdout={text} stderr={err}"
    silent_exit = not text and not err
    empty_stdout = not text
    if _should_failover(combined) or silent_exit or empty_stdout:
        return await _deepseek_complete(...)
    raise ClaudeMaxError(f"claude CLI exited {proc.returncode}: {err[:500]}")

The logic covered:

• _should_failover() — pattern-matches known rate-limit / auth error strings in stdout or stderr
• silent_exit — both stdout and stderr empty (a session lock situation)
• empty_stdout — nothing usable in stdout regardless of stderr

And yet the error claude CLI exited 1: (with empty stderr, colon followed by nothing) was slipping past all three conditions and raising ClaudeMaxError instead of routing to DeepSeek.

What was actually happening

The Claude Max subscription has a concurrency model: it works like an authenticated session, not a stateless API key. When 8 subprocess calls hit the bundled CLI simultaneously, most of them can't get a session slot.

The ones that lose the race don't fail silently. They produce something on stdout — a canned "I can't help right now" or "I'm ready to assist" response from the CLI's session-conflict handling. Then they exit 1.

Because stdout was non-empty, empty_stdout was False. Because the output was a generic agent greeting rather than a rate-limit error, _should_failover() returned False. Because stderr was empty (the CLI didn't write an error there, it wrote to stdout), silent_exit was False.

All three conditions missed it. The error fell through to raise ClaudeMaxError.

The fix

The invariant I was missing: legitimate CLI errors always write to stderr. An exit code 1 with nothing in stderr isn't a content error — it's a session infrastructure failure. The CLI couldn't even start properly, so whatever it wrote to stdout is noise.

silent_exit = not text and not err
empty_stdout = not text
no_stderr = not err  # ← new
if _should_failover(combined) or silent_exit or empty_stdout or no_stderr:

One condition. The reasoning: if the process exited non-zero and produced no stderr, we don't know what failed, but we know the bundled session infrastructure is broken. DeepSeek can handle the completion instead.

Why this took a while to find

The error message claude CLI exited 1: looks like it should tell you something, but the empty string after the colon is the stderr content (err[:500]). There's no stderr because the CLI never got far enough to write an error — it produced its stub stdout and quit.

Running the CLI manually during a quiet period always worked. The failure mode was load-dependent: it only appeared when 8 instances ran simultaneously, which only happened during the cron run, which only produced log output after the fact.

The tell was in the log pattern: every listing in a batch failing with the same error, in parallel, at the same cron timing. Individual failures are bugs; all-or-nothing batch failures are almost always infrastructure — session limits, connection pool exhaustion, semaphore contention.

The broader lesson

When you build a fallback system, think about the failure modes where the primary can't communicate why it failed. Pattern-matching on error message content is the right approach for API errors with structured responses. But for subprocess failures, the absence of stderr is itself a signal — one worth acting on.

The fix is a single line. The debugging was the whole problem.