Skip to content

NAS backup alerts and the weekly restore test ​

Issue #297, DH-14 (decided by the Product Owner on 2026-11-06), plan docs/plans/deploy-hardening.md ("Backups and restore", slice 2). Nothing here is installed yet: the scripts and the task list are in the repo, and an administrator installs them in change window W2 (DH-10).

What alerts, and how ​

EventAlert
Nightly or pre-deploy backup fails, including a failed off-site copyntfy push now, from infra/nas/backup/getmy-books-backup
Weekly restore test failsntfy push now, from infra/nas/backup/getmy-books-restore-test
Health check failsntfy push now, from scripts/health-check.sh through scripts/alert.sh
A job stops running, or its host is downThe hosted dead-man's-switch monitor alerts the owner's phone when a ping is late

The ntfy topic is ALERT_NTFY_TOPIC, the same one docs/plans/alerting.md describes. A successful run pings its own check. Pings are sent by:

JobSettingMonitor check
getmy-books-backup (nightly run only)HEARTBEAT_BACKUP_URLperiod 24 h, grace 2 h
getmy-books-restore-testHEARTBEAT_RESTORE_TEST_URLperiod 7 d, grace 1 d
scripts/health-check.shHEARTBEAT_HEALTH_URLperiod 15 min, grace 30 min

The monitor must be hosted off the NAS and off the Mac mini. If a ping URL is absent, the job logs a warning and carries on; a missing or failing heartbeat never fails a backup or turns a health check red. A pre-deploy backup alerts on failure but sends no heartbeat.

The three ping URLs are secrets. They are never in the repo:

  • NAS scripts read them, as data, from the root-owned /usr/local/etc/getmy-books/alert.env (root:root 0600), beside ALERT_NTFY_TOPIC: HEARTBEAT_BACKUP_URL=... and HEARTBEAT_RESTORE_TEST_URL=....
  • The health job reads HEARTBEAT_HEALTH_URL from the environment or from Doppler getmy-books/prd through scripts/health-check.sh's existing lookup.

Install (administrator, in window W2) ​

Follow plan items 0d, 11a, 19 and 20. For this issue:

  1. Create the three checks in the hosted monitor with the periods above, each alerting the Product Owner's phone. Owner decision outstanding: which monitor (see the pull request).
  2. Add HEARTBEAT_BACKUP_URL and HEARTBEAT_RESTORE_TEST_URL to alert.env on the NAS, and HEARTBEAT_HEALTH_URL to Doppler prd.
  3. Install getmy-books-backup and getmy-books-restore-test root-owned, mode 0755, at the destinations in infra/nas/manifest.
  4. In DSM, Control Panel, Task Scheduler, Create, Scheduled Task, User-defined script, create the two root tasks in infra/nas/backup/dsm-tasks: "getmy-books backup" daily 02:30, and "getmy-books restore test" weekly on Sunday at 04:00. Remove the /etc/crontab line (infra/nas/backup/crontab-line) once the backup task has run once.
  5. Run each task once with Run, check Action, View result, and check that the monitor shows a ping for each.
  6. Prove an alert: stop a job's ping or run a task with a bad argument, and confirm the phone is alerted. Undo by deleting the checks and tasks.

Rollback: delete the two tasks, restore the crontab line, copy the previous scripts back.