Appearance
NAS backup alerts and the weekly restore test
Issue #297, DH-14 (decided by the Product Owner on 2026-11-06), plan docs/plans/deploy-hardening.md ("Backups and restore", slice 2). Nothing here is installed yet: the scripts and the task list are in the repo, and an administrator installs them in change window W2 (DH-10).
What alerts, and how
| Event | Alert |
|---|---|
| Nightly or pre-deploy backup fails, including a failed off-site copy | ntfy push now, from infra/nas/backup/getmy-books-backup |
| Weekly restore test fails | ntfy push now, from infra/nas/backup/getmy-books-restore-test |
| Health check fails | ntfy push now, from scripts/health-check.sh through scripts/alert.sh |
| A job stops running, or its host is down | The hosted dead-man's-switch monitor alerts the owner's phone when a ping is late |
The ntfy topic is ALERT_NTFY_TOPIC, the same one docs/plans/alerting.md describes. A successful run pings its own check. Pings are sent by:
| Job | Setting | Monitor check |
|---|---|---|
getmy-books-backup (nightly run only) | HEARTBEAT_BACKUP_URL | period 24 h, grace 2 h |
getmy-books-restore-test | HEARTBEAT_RESTORE_TEST_URL | period 7 d, grace 1 d |
scripts/health-check.sh | HEARTBEAT_HEALTH_URL | period 15 min, grace 30 min |
The monitor must be hosted off the NAS and off the Mac mini. If a ping URL is absent, the job logs a warning and carries on; a missing or failing heartbeat never fails a backup or turns a health check red. A pre-deploy backup alerts on failure but sends no heartbeat.
The three ping URLs are secrets. They are never in the repo:
- NAS scripts read them, as data, from the root-owned
/usr/local/etc/getmy-books/alert.env(root:root 0600), besideALERT_NTFY_TOPIC:HEARTBEAT_BACKUP_URL=...andHEARTBEAT_RESTORE_TEST_URL=.... - The health job reads
HEARTBEAT_HEALTH_URLfrom the environment or from Dopplergetmy-books/prdthroughscripts/health-check.sh's existing lookup.
Install (administrator, in window W2)
Follow plan items 0d, 11a, 19 and 20. For this issue:
- Create the three checks in the hosted monitor with the periods above, each alerting the Product Owner's phone. Owner decision outstanding: which monitor (see the pull request).
- Add
HEARTBEAT_BACKUP_URLandHEARTBEAT_RESTORE_TEST_URLtoalert.envon the NAS, andHEARTBEAT_HEALTH_URLto Dopplerprd. - Install
getmy-books-backupandgetmy-books-restore-testroot-owned, mode 0755, at the destinations ininfra/nas/manifest. - In DSM, Control Panel, Task Scheduler, Create, Scheduled Task, User-defined script, create the two root tasks in
infra/nas/backup/dsm-tasks: "getmy-books backup" daily 02:30, and "getmy-books restore test" weekly on Sunday at 04:00. Remove the/etc/crontabline (infra/nas/backup/crontab-line) once the backup task has run once. - Run each task once with Run, check Action, View result, and check that the monitor shows a ping for each.
- Prove an alert: stop a job's ping or run a task with a bad argument, and confirm the phone is alerted. Undo by deleting the checks and tasks.
Rollback: delete the two tasks, restore the crontab line, copy the previous scripts back.