A tiny S3 proxy in front of a pool of Drive accounts

I had ~50 TB of media I wanted to back up off-site. Synology HyperBackup can speak S3. Google Drive Workspace gives you generous per-user storage, but the per-user upload cap is 750 GB/day. Multiple Workspace users in the same tenant, however, can all upload in parallel. The interesting question is how to bridge the two.

What was happening

The first attempt was an off-the-shelf restic clone running on a single Drive account: throttled at the per-user cap, ~14 days to seed even before the disk-full incidents kicked in. Single-account encryption meant the entire backup format was custom and only restorable by my code, which I didn't love.

I deleted it. The second design is much simpler: a stateless proxy that speaks the subset of S3 that HyperBackup uses, fans writes out across multiple Drive accounts (round-robin weighted by daily remaining quota), and stores the bucket+key → (account_id, drive_file_id) mapping in Postgres.

What I found

I only needed a small slice of the S3 surface: PUT (single + multipart), GET, HEAD, LIST, plus the bucket operations HyperBackup probes on first connect — CreateBucket, HeadBucket, GetBucketLocation, ListBuckets, ListObjectsV2.

Subset of the S3 spec, but it had to be byte-exact. ETag had to be the MD5 of the object on PUT (HyperBackup verifies). Multipart ETag had to be MD5-of-MD5s concatenated with -N. Empty PUT had to return Content-Length 0 and the right ETag for an empty string. Virtual-host-style addressing had to work too — HyperBackup uses bucket-name-as-subdomain unless you tell it otherwise, which means a wildcard DNS entry and a wildcard cert.

Three weird gotchas worth flagging:

1. Chunked transfer encoding on small writes. HyperBackup sends config files with no Content-Length, using chunked encoding. The original handler errored on missing Content-Length. Fix: detect missing/zero Content-Length, buffer to memory, reserve a 1-byte quota, then adjust once the actual size is known.

2. Virtual-host vs path-style addressing. Setting up the wildcard DNS entry + cert (DNS-01 via certbot) unlocks both styles. nginx forwards to the proxy with the bucket carried in either the Host header or the path; a small middleware rewrites Host-based requests into path format before route matching, so the rest of the codebase only sees one shape.

3. HyperBackup gets stuck on orphan chunks. If the client disconnects after the server has written the file but before the 200 response is delivered (a "499" in nginx terms), the file is durable but HyperBackup thinks the upload failed. Next "Back Up Now" detects the present-but-uncommitted chunk, flags the repo as inconsistent, and refuses to upload anything further. Fix: cleanup pass that lists files written without a corresponding commit row in the DB, deletes them, then tells HyperBackup to retry.

The fix

A few hundred lines of FastAPI. Round-robin account selection weighted by remaining-quota-today plus an OAuth multi-account flow so I can add capacity by clicking through Google's consent screen one more time. Adding a fifth account doubles the seed ceiling from ~750 GB/day per account to ~3.75 TB/day pool-wide. Beyond ten accounts the gigabit upstream becomes the bottleneck.

def pick_account(accounts):
    """Round-robin weighted by min(quota_free, daily_remaining)."""
    candidates = [
        (a, min(a.quota_free, a.daily_upload_remaining))
        for a in accounts
        if a.daily_upload_remaining > 0
    ]
    if not candidates:
        raise NoCapacity()
    # bias toward accounts with more headroom but still rotate
    candidates.sort(key=lambda c: c[1], reverse=True)
    return candidates[0][0]

After cutover, the seed throughput is bottlenecked by Google's per-account quota, not the network. Five accounts hit ~3.75 TB/day ceiling; the disk-full incident on the staging dir (separate post) was the only other constraint.

What I'd do differently

The first design tried to be its own backup format. Replacing it with a thin proxy lets HyperBackup own chunking, deduplication, encryption, retention, restore UI — every hard problem in backup, solved by code I didn't have to write. The proxy is the boring glue between "thing that wants S3" and "thing that wants Drive."

That's a pattern I want to lean on more: when a project starts to feel like I'm reinventing a domain (backup, search, queueing, storage), the right move is often to stop reinventing and become a small adapter between two mature systems instead.