Homelab monitoring & backups

Scroll any r/selfhosted "how do you monitor your cron jobs?" thread and the upvoted answers are never "install Prometheus." They're heartbeats, in three shapes:

1. ntfy or gotify push at the end of the job. Two lines added to the script. Success is a small notification that fades; failure is loud. It works because it's push — you don't have to remember to check anything, the system reaches out to you.
2. Healthchecks-style start/finish pings. Each job hits a unique URL at start and end; a central service that knows the schedule alerts you if a ping doesn't arrive by the deadline. The key advantage: this catches non-runs (the job never fired, or ran past its deadline), not just failures. A clean run stays silent — the right default.
3. A tiny custom collector. For users who already run ntfy/gotify, a small script that listens for posts, stores them in SQLite, and serves a one-page dashboard.

The insight underneath: "how do I monitor crons" reads as a request for tooling, but the real need is a signal — did the backup run today, did the cleanup finish, did the sync push its files, and when did it fail. Heavyweight stacks answer a different set of questions (requests/sec, p99 latency, which endpoints return more 500s) that side-project operators mostly aren't asking.

Upgrade — to self-hosted Healthchecks, then Cronicle, then Airflow/Temporal/Argo — only when you feel the pain: dozens of interdependent jobs, historical analysis across long windows, customer SLAs, or concurrency/overlap issues. Upgrade in response to felt pain, not in anticipation of it. Premature upgrade is exactly how people end up stuck installing Grafana when they should be writing a two-line wrapper.

The gap here isn't capability — SigNoz and OTel can absolutely answer your questions. The gap is scope: you wanted answers, and the tooling asks you to build infrastructure first. You open the docs, see the architecture diagram, and quietly close the tab. The momentum dies right there.

Start by naming the top five questions self-hosters actually ask:

1. Is the app up right now? (binary health)
2. What's the latency on the slow endpoints? (p50/p95 over a recent window)
3. Which endpoints are throwing errors? (error rates by route)
4. What changed — was there a deploy that started this? (events overlaid on metrics)
5. What's going on in this request? (a log or trace pulled up by request ID)

All five are answerable with basic infrastructure. The lightweight layer is three pieces: structured JSON logs in one searchable place (each line has timestamp, level, request ID, route, status, latency); a tiny metrics rollup — a script that reads those logs and, every minute, stores total/error counts and p50/p95 by route in a 30-day rolling table; and events overlaid on the timeline — deploys, cron runs, and proxy reloads written as one-line timestamped markers so a latency spike lines up visibly with the deploy that caused it. A couple hundred lines of code and a SQLite or DuckDB file.

The failure modes of starting at OTel + SigNoz are predictable: weeks lost to collector/exporter/dashboard config while the actual product stops getting attention; drift, where the stack becomes its own thing to maintain; underuse, because the deep capability sits idle when your real questions are the simple five; and distrust of answers you're not sure you configured right. The lightweight layer scales down, which the full stack doesn't — and its structured logs and rollups are exactly the inputs a heavier stack would consume later, so the upgrade path stays open. If you're not sure which bucket you're in, build the smaller thing first.

Homelabs degrade quietly. Catastrophic failures get attention; slow degradation doesn't — until Plex won't start because the disk hit 100% from log files you forgot existed, and your last good backup was the day you set up restic and never looked again. A 30-minute monthly pass catches roughly 80% of the slow stuff. The checks:

• Disk space, every drive (df -h --total). Over 80% gets investigated this month, over 90% gets fixed today. Usual culprits: Docker images/volumes (docker system df), /var/log, old kernels (apt autoremove).
• SMART on every spinning disk (smartctl -H). Pre-fail/failed = replace this month; reallocated sectors > 0 = watch closely.
• Backups actually completed last cycle. Check the last run timestamp and exit code, not just that it ran (restic snapshots --last 5). Older than your interval means the cron/timer broke — usually an expired token or a full remote.
• A restore actually works — restore one file to a temp dir and diff it.
• SSL certs not expiring within 14 days, and if one is, find out why auto-renewal didn't fire.
• Container health: docker ps --filter health=unhealthy and --filter status=restarting should both be empty.
• Memory/swap pressure (free -h, swapon --show) — unusual heavy swap means something's leaking.
• Security updates (apt list --upgradable | grep -i security) applied this session.
• Failed logins (lastb, Failed password in auth.log) — watch for a repeated IP or valid usernames being tried.
• Cron/timer history (systemctl list-timers --all, journalctl for failures).
• Open ports (ss -tlnp) compared against expected.
• Docker network sanity — stale _old networks cause the intermittent routing bugs that eat hours later.

Deliberately out of scope: major version upgrades (schedule separately — doing them as routine maintenance produces routine outages), app-level config tweaks, and anything optional you'll skip by month three and feel bad about. After a real incident, add the one check that would have caught it. The goal is a 30-minute habit you actually run, not a 4-hour audit nobody does.

The dangerous failure mode of a partial restore is that the app comes back up and starts taking traffic in a corrupt state. By the time you notice, you've also taken new writes against the half-restored data — so rolling back now loses both the new writes and whatever the original failure lost.

Treat it as a partial restore even if the restore tool exited 0 (tools lie about success more than you'd think) when you see: 500s on routes that touch specific tables/files, background jobs failing with "row not found"/foreign-key violations, uploads working but referenced media missing, or a log that mentions skipped files or failed checksums.

Then, in order:

1. Freeze writes immediately. This is the single biggest mistake to avoid — every new write is one you might lose. Take the app offline (maintenance page or a 503 from the reverse proxy), stop background workers and queues (the sneaky write source), set the DB read-only or revoke the app user's write permission, and revoke object-storage write credentials. Sixty seconds with prep.
2. Inventory the real state. What actually restored (integrity checks, row counts vs. the snapshot's manifest), what's missing or corrupt, and what the app has written since the restore started (logs, audit trail, queue history — this is what a full rollback would throw away).
3. Work the decision tree. Roll back further when the current restore is too damaged, you have a verified clean older backup, and you can afford the data loss. Repair the partial — fill the gaps from a secondary backup, a replica, or another environment — when the damage is localized and you have a source; this is usually the highest-EV move when you have the inputs. Roll forward when the damage is small and self-healing and you can't afford the rollback's loss. Most leads default to rolling back out of caution; that's wrong about as often as it's right.
4. Stage the fix against a sandbox copy first, verify it merges cleanly (no FK violations, no checksum mismatches), then apply to prod.
5. Bring the app back gradually: read-only for 30 minutes, then workers at reduced concurrency, then full writes — watching error rates at each stage.

The recurring 2am story: someone restores from a corrupted restic repo, or finds the last good snapshot is a week old because a token expired and nobody was watching. A backup you've never restored isn't a backup — it's a hope. Turning hope into confidence is a short discipline:

• Verify completion, not existence. Every cycle, check the backup tool's last-run timestamp and exit code — not merely that a job ran. If the last snapshot is older than your declared interval, the cron/timer broke; it's almost always an expired token or a remote that ran out of space.
• Run actual restore drills. Pick one file, restore it to a temp/sandbox directory, and diff it against the original. Quarterly is acceptable, monthly is better. The first time you genuinely need a backup is the worst possible time to discover the restore is broken.
• Store verification alongside the data. Checksums, row-count manifests, and schema dumps kept next to the data dumps catch corruption before you're depending on the backup.
• Keep multiple snapshots. Daily for a week, weekly for a month. That retention is exactly what gives you options when the latest snapshot turns out to be the bad one — the precise situation that forces a "roll back further" call during a botched restore.
• Know your restore-time SLO. Measure how long a full restore actually takes before you need to know. "How long will this take?" is a terrible question to first ask at hour two of an outage.

The payoff is turning the next incident from a three-hour panic into a 45-minute, mostly-automated, mostly-stress-free procedure. Most teams have never tested a restore until they needed one — don't be most teams.

The naive script — gpg -d > /tmp/backup.tar.gz, untar, edit, re-tar, re-encrypt, rm — works right up until a step fails: disk full, corrupt archive, Ctrl-C, an OOM SIGKILL. Then the decrypted /tmp/backup.tar.gz and /tmp/extract/ sit on disk indefinitely with your database dumps, env files, encryption keys, and customer data in the clear. You find them three weeks later running du. Backup files are the worst kind of exposure: high-value, low-visibility (nobody audits /tmp), and persistent.

The cleanup-guaranteed pattern costs about three extra lines:

• set -euo pipefail at the top.
• TMPDIR=$(mktemp -d); chmod 700 "$TMPDIR" — a unique dir only your user can read.
• A cleanup() that shred -u -n 1s the files, then rm -rfs the dir.
• trap cleanup EXIT INT TERM so cleanup fires no matter how the script exits — success, Ctrl-C, or SIGTERM from systemd.
• Re-encrypt to a .new filename, then mv it into place for an atomic replacement with no window where the archive is missing.

Why euo earns its keep: without -e, a failed gpg -d continues and re-encrypts stale or empty data; without -u, an unset $TMPDIR turns rm -rf "$TMPDIR/" into rm -rf "/" (the famous SteamOS bug); without pipefail, gpg -d backup.gpg | tar -x returns 0 even when gpg failed. On macOS there's no shred — fall back to rm -P.

Best of all, if you only need to read from the archive, never write plaintext to disk at all: gpg -d backup.tar.gz.gpg | tar -xzO somefile.txt. The -O flag sends tar to stdout, so the file never touches disk. The defining property to aim for: it should be impossible to leave plaintext behind even if you Ctrl-C, even if the disk fills, even if you forget — the trap handler is what makes "impossible" actually true.