Self-hosting operations

It's almost always a UID/GID mismatch combined with NFS user-squashing. Reads succeed under squash because the files are world-readable; writes need the squashed-to user to own the directory, and it doesn't.

Synology (and most NAS boxes) default to squashing every incoming request to a fixed UID — usually admin (UID 1024) or nobody (65534). Your container connects as UID 1000, the NAS rewrites the request to 1024, and since the share is owner-only-writable by a DSM user, the write fails even though DSM shows the share as read-write. ls and cat work; the first touch foo returns Permission denied.

Two clean fixes:

• Match UIDs. Make the container (or VM user) run as the squash target: PUID/PGID on linuxserver.io images, or usermod -u 1024.
• Set the export's anonuid/anongid to the UID your client actually uses, so files land owned by your user.

Don't chmod -R 777 — it "works" but opens anyone on the network to write your shares and confuses DSM's own services. Verify with touch then ls -la: the owner should be your UID, not 65534.

The broader NFS-for-Docker rules that keep it boring: use NFSv4 with sec=sys and explicit anonuid; keep actimeo below the 60s default so containers don't read stale stat() data; know that inotify watchers don't fire over NFS (Sonarr/Syncthing degrade to polling); and put sqlite databases and app config on local disk, media on NFS — the *arr suite and NZBGet corrupt their sqlite DBs over network filesystems no matter how careful you are.

Run it in a VM (or, in narrow cases, an LXC) — not on the host. Installing Docker on the Proxmox host is one command and tempting, but it trades small near-term wins for large compounding costs.

What you give up by running Docker on the host:

• Networking conflicts. Docker creates docker0 and manages iptables aggressively, which clashes with Proxmox's bridge networking in subtle ways that surface months later.
• Storage layered weirdly. Docker's storage driver sits on top of Proxmox's LVM-thin/ZFS; snapshots and quotas don't compose.
• Backup blind spot. Proxmox Backup Server backs up VMs and LXCs, not host-level Docker state — your containers and volumes are silently excluded from your normal backups.
• Upgrade fragility & painful recovery. Major Proxmox upgrades bump the kernel or networking and can break Docker; a corrupted host means rebuilding everything by hand.

A VM makes the container host one self-contained unit: atomic backups, one-click restore, clone it for staging, migrate it to another host. Reference spec for a small homelab: 4 vCPU / 8 GB RAM / 100 GB SSD, VirtIO on vmbr0, QEMU guest agent enabled, Docker CE from download.docker.com (not the distro's docker.io), Compose v2 plugin.

Use LXCs directly for things that ship as LXC templates (Pi-hole, AdGuard, Home Assistant). Docker-in-LXC is a footgun for many disparate containers or rootless needs. Migration off the host is a Saturday afternoon: re-pull images, tar /var/lib/docker/volumes, redeploy your compose files, cut over DNS/proxy, then reclaim the host resources.

There are four breakage modes, in order of frequency, and a short diagnostic tells you which one you're hitting:

1. Cert expiry. Let's Encrypt certs last 90 days; a renewal hook failed silently last week and the cert died today.
2. DNS confusion. Your home IP changed, DDNS broke, or an A record got edited — the cert is fine, the connection isn't reaching the right server.
3. Mixed content. An HTTPS page loads JS/CSS/images over HTTP and the browser blocks them.
4. HTTPS-from-LAN. You reach the site via an internal hostname/IP the cert doesn't match — errors that vanish over cellular.

Run the 20-minute sequence in order; the first check that fails is your issue:

• Cert dates: echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null | openssl x509 -noout -dates — is notAfter in the past?
• DNS: dig +short example.com @8.8.8.8 vs the server's own curl -4 ifconfig.me.
• Listener: curl -vI https://example.com from outside — Connection refused/timed out means the service, firewall, or port-forward is down.
• Mixed content: DevTools console for Mixed Content warnings.
• LAN vs cellular: works on cellular but errors at home = hairpin-NAT/hostname mismatch.

Fixes: force a renewal (the usual root cause is a port-80 redirect blocking the ACME HTTP challenge); fix the A record and add DDNS via the Cloudflare API; set the app's BASE_URL/overwrite.cli.url, pass X-Forwarded-Proto https, and add HSTS; use split-DNS so your internal resolver points the public hostname at the LAN IP. Prevent recurrence with a daily cron that checks expiry and emails you 14 days out, plus an external uptime ping.

Every container has its own filesystem, so the same host file appears at different paths depending on which container is looking. qBittorrent saved the download to /data/torrents/Show.S01E01.mkv; Sonarr has no /data/torrents mount, so when qBittorrent reports "complete at that path," Sonarr looks inside its own container, finds nothing, and auto-import silently fails — even though you can SSH in and ls the file.

The fix is unglamorous: mount the same host directory at the same path in every container (the "one mount, one path" model). Give qBittorrent, Sonarr, and Radarr all /mnt/storage:/data, and lay out /mnt/storage/torrents and /mnt/storage/media on disk. Now /data/torrents/tv/... resolves identically in every container, and no path translation is needed. Bonus: hardlinks only work within a single filesystem — with the unified /data root, Sonarr correctly sees torrents and media share a root, so imports become atomic and disk usage doesn't double. (The TRaSH Guides are the canonical reference here.)

The exact same path-visibility gap bites PaaS file uploads on Dokploy, Coolify, and CapRover: you click "Upload," the dashboard reports success, and the app can't see the file — because the dashboard wrote to its volume while the app reads from a different one. "Upload successful" only ever meant "saved to the dashboard's volume." Diagnostic: find the file, docker volume inspect to get its mountpoint, then docker inspect the app container's Mounts array — nine times out of ten the volume simply isn't mounted where the app expects it.

When an error fires only some of the time, the problem is rarely the thing the error names — it's a routing or fan-out issue upstream of the named subsystem. A CORS error makes you debug CORS for hours in the wrong direction.

In the case that took three hours to diagnose, Access-Control-Allow-Origin: * was set and the CORS middleware's unit tests passed, but ~20% of preflight OPTIONS requests came back with no CORS headers. The cause was Docker networking: a stale app_net_old network from a previous deploy still had an api-named container attached. Docker's embedded DNS round-robined between the live container and the orphan — which was running months-old code without the CORS middleware. The 80% of requests that hit the live backend worked; the 20% that hit the orphan failed.

The diagnostic path that broke the deadlock, worth copying for any intermittent bug:

1. Reproduce from the server itself — curl -v -X OPTIONS https://api.example.com/users in a loop of 50 to rule out the browser. ~20% were missing the headers.
2. Diff a passing vs failing response — the failing one carried Server: gunicorn/19.x, a version not deployed in months. Different version = different container.
3. List everything answering on that network — docker network inspect app_net and app_net_old both showed an api container.

Fix was three commands: stop and rm the orphan, then docker network rm app_net_old. CORS errors gone within a minute, no restart needed. The rule: for intermittent errors, check for stale containers, dual networks, and leftover replicas before you spend 30 minutes on the named problem.

Jellyfin is where most migrators land: free, open-source, no paid tier, no account requirement, and no company sitting between you and your own files. Point it at your existing media folders and it builds a polished library with artwork and metadata, then streams to apps on your TV, phone, and browser. The core feature set — libraries, transcoding, multi-user, remote access — is the Plex experience without the subscription or the cloud relay.

The trade-offs, stated plainly: you run the remote-access infrastructure yourself (a reverse proxy or a tunnel, not a checkbox someone else operates); the smart-TV client apps are good but not always as polished as Plex's on certain platforms; and updates, backups, and the occasional transcoding tweak are now yours. For people already self-hosting, that's the same bargain they've accepted everywhere else.

The migration breaks into four concrete steps: stand up an always-on server (a homelab box, or a VPS for reliable remote streaming); mount your media (your existing folder structure and good naming carry over, so metadata matches cleanly); set up remote access safely with a reverse proxy + SSL or a secure tunnel; recreate household accounts. The single most common trip-up is path consistency — the server, the container, and your storage all have to agree on where the media lives (the same mount-mapping problem behind *arr import failures). On where to run it: homelab is cheapest if you already have hardware on 24/7; a VPS wins when you want access from anywhere or when home upload bandwidth would choke remote streaming. Many people run a hybrid — media at home, a small VPS handling secure remote access.

The awesome-self-hosted lists answer "what's possible," but possible was never your constraint — maintenance was. Most apps people deploy in month one are dead by month three: not because they broke, but because nobody used them and they quietly became maintenance debt. The skill in self-hosting isn't deploying things; it's choosing the few that earn their keep.

The filter that matters: if it disappeared tomorrow, would you actually feel it? Apps that pass tend to share three traits — you'd use it weekly or more, it replaces something you were paying for or trusting to a third party, and it's low-maintenance (no babysitting). A starter stack that consistently survives that test:

• A password manager (Vaultwarden) — high daily use, replaces a paid subscription.
• A file sync / cloud drive (Nextcloud, or Syncthing for pure sync) — replaces Dropbox/Google Drive.
• A media server (Jellyfin) — among the highest-satisfaction self-hosts if you have a library.
• A read-it-later / bookmarks tool (Linkding, Wallabag) — tiny footprint, near-zero maintenance.
• A monitoring / uptime tool (Uptime Kuma) — the boring pick that pays for itself the first time it warns you something's down.

That's five, not three hundred. Every app you run is an ongoing tax — updates, backups, the 2am "why is this down" — and that cost scales with the number of apps, not their usefulness. So make maintenance a first-class decision: run a monthly pass over updates, backups, and disk space, and back up the data that matters (Vaultwarden and Nextcloud especially — the app you can redeploy in five minutes isn't the risk; the data inside it is).

It's essential, not arbitrary. Infrastructure setup is a chain where each step's output (a domain, an IP, a key, a record) becomes the next step's input. Do it out of order and you either block on yourself, redo work, or — worst — create silent failures you discover much later, far from the change that caused them.

A four-stage model covers ~90% of setups:

1. Identity — domain, DNS records (A/AAAA/CNAME/MX/TXT), PTR, TLS certs. Longest lead time (propagation, CA checks) and almost everything depends on it, so do it first.
2. Substrate — VPS, block/object storage, networks, firewall. Do it in parallel once you have an IP.
3. Services — web server, database, mail, reverse proxy, auth. These need both identity and substrate; setting one up before DNS means reconfiguring it later and forgetting a setting.
4. Integrations — Stripe, transactional email, analytics, monitoring, CI/CD, OAuth apps. Last, because their setup wizards want to verify a callback URL or from-domain that are Stage 3 outputs.

The classic mistakes are all order violations: standing up a mail server before DNS (no SPF/DKIM/DMARC/PTR burns IP reputation for weeks); pointing DNS at production before the firewall is up (scanners find you within hours); setting OAuth callbacks to localhost you then never update. The lesson isn't to memorize an order from a tutorial — it's to learn the dependency model so you can reason about any novel setup. A checklist saves you once; a model saves you every time.

Three related disciplines close the gap between "we attempted a deploy" and "the right thing is live."

1. Preflight before you ship. A small, idempotent script that verifies the environment can support the change — required env vars set and non-empty, DB at the migration the new code expects, upstream services reachable, disk free, TLS not about to expire — and exits non-zero to block the deploy if anything's missing. It's not a test suite: tests check code in a fake environment; monitoring tells you after it broke. It runs in under 30 seconds: ./preflight.sh && ./deploy.sh. The same snapshot→change→compare pattern catches silent proxy breakage too — capture the status code, cert, and health-body hash for every host before an nginx -s reload, diff after, and the route nobody had open shows up as a diff instead of a user report three days later.

2. Codify, don't re-document. Turn each hard-won gotcha into a preflight check and wrap the whole flow as a runnable skill (a shell script, a Makefile target, or an MCP module) the next operator invokes by name. Docs rot silently, don't carry parameters, and can't check anything; a skill can. After three or four deploys, the pain stops compounding.

3. Verify after. A green checkmark usually means the attempt succeeded (image built, container started, command exited zero) — not that the live app is the new version. Fingerprint each release (a /version endpoint or a response header carrying the git SHA), probe the live surface through the same proxy and domain real users hit, and treat that match as the closing signal. A mismatch names the cause: old container still running, stale proxy upstream, a cache serving old responses, or an unmigrated volume.