The screen that watches you back

I trimmed a few pages off the dashboard that runs on my living room TV, deployed it, and then went to do the thing I always make my agent do before it says "done": look at the screen and confirm the change is actually there.

The kiosk is an Android box running a browser in fullscreen. ADB is enabled, so a screen check is one command:

adb -s <kiosk>:5555 exec-out screencap -p > check.png

That gave me a perfect capture of... my agent's live activity feed. Every tool call it had just made, rendered in a nice big font on the TV.

Which is correct behaviour. The dashboard has a takeover mode: while a job is running, the page swaps out the rotating panels and shows what the agent is doing. It's the best feature on the thing. It also means the dashboard is never showing the dashboard at the exact moment something is in a position to take a picture of it.

A little observer effect, in my house.

The wrong fixes

The first instinct is to find a back door into the page — remote-debug the WebView, drive it from a headless browser somewhere else, query the DOM over an API. I did check the DOM over the network, and it proved the served bundle was right. It does not prove the kiosk reloaded, and it does not catch a layout that renders fine at desktop width and clips at 1080p on a TV eight feet away. I wanted the actual pixels on the actual panel.

The second instinct is to poll: take a shot, look, take another. Same problem. The polling process is the thing keeping the takeover on screen.

Move the shutter

The observer and the camera don't have to be the same machine. The device can hold the camera:

adb -s <kiosk>:5555 shell \
  "nohup sh -c 'sleep 350; for i in 1 2 3 4 5 6 7 8 9; do
      screencap -p /sdcard/shot\$i.png; sleep 26; done' >/dev/null 2>&1 &"

Then disconnect and go away. The captures fire on the device during the window when nothing is connected to it, which is exactly the state I could never observe from the outside. Come back a few minutes later and pull the files.

Two details make it work:

The lead time is a constant you can read, not a guess. The takeover lingers for a fixed interval after a job ends — in my case LINGER_SECONDS = 300, one grep away in the feed service. Sleep past it. Sleeping "a while" would have burned another cycle.

The interval undersamples the rotation on purpose. Panels rotate every 30 seconds; the captures fire every 26. Nine shots at 26s covers a full seven-panel cycle with drift, so every panel lands in at least one frame. Shooting at exactly 30 would have risked photographing the same panel nine times.

Reading nine screenshots without reading nine screenshots

Nine 1080p frames is a lot to look through when the only question is "which panel is this, and is it the right list of panels". Every frame has a page-name strip near the top, so crop just that band out of each and stack them:

for i in $(seq 1 9); do
  convert shot$i.png -crop 1920x120+0+60 +repage -resize 700x strip$i.png
done
montage strip[1-9].png -tile 1x9 -geometry +0+2 strips.png

One image, nine rows: CLIMATE / HVAC / NETWORK / PROXMOX / BACKUP / PERSONAL / HOME / CLIMATE. The removed panels are gone, the page-dot count dropped from nine to seven, and I know without opening anything else that frame 7 is the one I actually need to inspect.

That last part turned out to matter more than the trick that got the shots. Looking at frame 7 full size showed the change was in — and that removing two cards from that page had left the remaining ones stretched, with the forecast's rain percentage clipped off the bottom edge. Every automated check I had said green. The grid was doing exactly what I told it to. It just looked wrong, and nothing but the actual pixels was ever going to tell me that.

Three rounds of the same loop — adjust the row heights, deploy, let the device photograph itself while I wasn't looking — and the panel renders clean.

The general shape, for anything with a UI that reacts to being connected to: if watching changes what you'd see, arrange for the watching to happen when you're not there.