Workspace permission denied when SCP'ing from automation runner

I have a small homelab automation runner that orchestrates jobs across a handful of remote machines. New job, new bug: SCP transfers from the runner to a target box were failing with permission denied — but only on the local workspace side, before the file even left the runner.

What was happening

The runner creates a per-job workspace under /opt/runner/workspace/<job-uuid>/. It does some prep work there, then SCPs the staged files to the target. The SCP step failed:

Permission denied
scp: failed to write '/opt/runner/workspace/<uuid>/temp/payload.zip': Permission denied
lost connection

The interesting detail: the path was on the runner, not the remote. The runner was failing to write a temp file inside its own workspace.

What I found

Earlier in the job's lifecycle, a download step had run as a different user (via sudo) and dropped a file into the workspace. That file ended up owned by root, and the workspace temp/ subdirectory it lived in inherited root-only permissions because the download step had created the subdirectory itself.

When the SCP step ran later as the runner's own service user, it couldn't write into the now-root-owned subdirectory. The error message is technically correct but misleading — it says "Permission denied" against a path you'd expect the service to own, which makes you go looking on the remote side first.

The fix wasn't to escalate SCP; the fix was to make sure no earlier step left a stranger owning files in the workspace.

The fix

Two layers. First, the immediate cleanup:

chown -R runner:runner /opt/runner/workspace/<uuid>/

Second, the runner now wraps any step that needs sudo (or any privileged context) with a post-step chown back to the runner user:

def run_step(step):
    workspace = step.workspace_dir
    try:
        step.execute()
    finally:
        # any privileged step might leave stranger-owned files behind;
        # always claim them back so later steps can read/write.
        subprocess.run(
            ["chown", "-R", f"{RUNNER_USER}:{RUNNER_USER}", workspace],
            check=False,
        )

check=False because chown failures aren't fatal — if it's already correct, we move on.

What I'd do differently

In any multi-step pipeline that mixes privileged and unprivileged steps in a shared workspace, ownership drift is going to happen sooner or later. The robust pattern is to either (a) give the workspace a sticky bit and a group everyone can write through, or (b) reset ownership after each step. I went with (b) because it's explicit — you can see in the logs that ownership got normalized, and any future weirdness shows up clearly.

The misleading error message taught me a thing too. When SCP fails with a path on one side, double-check that the path is actually on the side you think it's on. Read the error twice before SSHing into the wrong machine.