Two weeks of silent torrent failures from one chown

Nothing new had landed in my movie library since the 4th of the month. Radarr looked healthy, the queue was just stuck at 321 items. Most of them showed qBittorrent is reporting an error. That's a useless message on its own, so I went digging.

What was happening

Five qBittorrent containers, one each for movies, TV, music, books, and the one for the adult library. The TV container was importing fine — new episodes were showing up in Emby normally. The other four were dead. Same NFS export, same arr stack, same everything else.

Radarr's queue had 212 torrents in an error state on the movies container alone. qBittorrent's own log was much more direct:

file_open ... error: Permission denied

What I found

The torrent download directory is an NFS export from the NAS, mounted at /downloads inside each LXC. qBittorrent runs as a dedicated qbittorrent user (uid 999) in each container.

$ pct exec 111 -- stat -c '%U:%G %a' /downloads
root:root 755
$ pct exec 112 -- stat -c '%U:%G %a' /downloads
qbittorrent:qbittorrent 755

Four of the five containers had /downloads owned by root:root. The TV container was the only one with correct ownership. qBittorrent couldn't open any file for write, so every torrent went straight to error state and stayed there. The arr apps just reported "client error" and gave up.

How did the ownership change? Best guess is a Proxmox LXC idmap shift after an update, but I can't prove it. What I can do is recover.

The fix

The NFS export already had no_root_squash, so a chown from inside the container would carry through to the NAS:

for ct in 111 113 114 115; do
  pct exec $ct -- chown qbittorrent:qbittorrent /downloads
done

Then resume and recheck all the errored torrents via the qBittorrent API:

for QBT in 111 113 114 115; do
  HASHES=$(curl -s "http://10.x.x.$QBT:8080/api/v2/torrents/info?filter=errored" \
    | jq -r '.[].hash' | tr '\n' '|')
  curl -X POST "http://10.x.x.$QBT:8080/api/v2/torrents/resume" \
    -d "hashes=$HASHES"
  curl -X POST "http://10.x.x.$QBT:8080/api/v2/torrents/recheck" \
    -d "hashes=$HASHES"
done

Within two minutes the errored count was zero and the movies container was pulling at 25 MB/s.

What I'd do differently

This is the kind of failure where a 30-line "homelab vitals" cron job pays for itself many times over. The signal here was straightforward — every qBittorrent client exposes a count of errored torrents at /api/v2/torrents/info?filter=errored. If that number is greater than zero for more than one consecutive poll, send a push notification. I now have that check running every 10 minutes against all five clients. Anything weirder than zero shows up before the queue is two weeks deep.