Skip to content

Plan: alerting ​

Status: built (AUDIT.md operations queue). Off until the Product Owner does the setup at the end; nothing fails while it is missing.

Note 2026-11-06: the pipeline moved to GitHub Actions (ADR 0046). The workflows are in .github/workflows/, the Health job runs on GitHub's runners (so it keeps watching while the Mac mini is off), and both read DOPPLER_TOKEN_PRD as a secret of the production environment.

Why ​

Three things can go wrong in production without anyone noticing: a deploy fails, the app stops answering, or a company's Xero connection dies. The Product Owner must hear about each one on their phone, without opening GitHub or a conversation.

Channel: ntfy push ​

  • ntfy (ntfy.sh): a free push service. The sender posts a message to a topic with one HTTP request; the phone app subscribed to that topic shows it as a notification. No account, no new server, no new setting in GitHub, Cloudflare or Entra.
  • Why not email: nothing sends email yet (the Microsoft Graph mailbox is only planned, see email-reminders.md). When it exists, sendAlert and scripts/alert.sh are the two places to add it.
  • Why not GitHub's own notifications: they would not cover the app's own Xero alert. They can be turned on as well, at no cost, as a second channel.
  • The topic is a password. Anyone who knows a topic name can read it, so the name is long and random, lives only in Doppler prd, and every message carries no company, person, id or figure. At most it says what failed, a short commit id and a link to the GitHub run.

What alerts, and how ​

EventWhere it is caughtWhat sends it
Main fails: checks, migration, deploy or the post-deploy smoke testalert-main-failed job in .github/workflows/ci.yml, runs only when a job before it failed on a push to mainscripts/alert.sh, topic from Doppler prd via DOPPLER_TOKEN_PRD
/health down.github/workflows/health.yml, every 15 minutes, three tries 20 seconds apartscripts/health-check.sh, then scripts/alert.sh; the run also goes red
Xero sign-in failingsrc/xero/session.ts: 3 failed token refreshes in a row for one entitylogger.error with alert: 'xero_auth_failures' and the entity id, then sendAlert (src/platform/alerts.ts) with no identifiers
Xero reconnect neededsrc/xero/session.ts: a refresh Xero refuses because the connection must be reconnected, once per entity until a refresh succeeds (SPEC-008 S18, #297)logger.error with alert: 'xero_reconnect_needed' and the entity id, then sendAlert with no identifiers
Database unreachable at sign-insrc/web/app.ts: the /mcp principal lookup throws (the person gets a 503, not "no access")logger.error with err on every failure, then sendAlert with no identifiers, once per outage and at most once every 15 minutes

Notes:

  • Reaching /health. The app publishes no ports; its only way in is the Cloudflare tunnel. Access bypasses mcp.getmy.group/health only (ADR 0038), so the check is a plain request. /health returns just the status and the commit. Every other path, /mcp included, still needs sign-in.
  • Service token only for the smoke test (2026-11-03, #81). /health is bypassed (ADR 0038), so neither the scheduled check nor the smoke test needs an Access service token to reach it. The only use for a token is the post-deploy smoke test's MCP check (scripts/smoke-test.sh, the smoke-nas job in ci.yml): with SMOKE_ACCESS_CLIENT_ID and SMOKE_ACCESS_CLIENT_SECRET in Doppler prd it GETs the smoke probe /smoke/mcp through Access; without them it checks /health reports the deployed commit and skips the probe with a notice. Amended 2026-11-06 (Product Owner decision on #81, SPEC-017 S26): /mcp still refuses a service token (no email, 403). The probe has its own Access application with a Service Auth policy, takes only the service token whose client id is SMOKE_ACCESS_CLIENT_ID, and answers only {"status":"ok","tools":N}. Set it up in this order, or main goes red: install the updated env-allowlist and compose file on the NAS, create the token and the application, add CF_ACCESS_SMOKE_AUD and SMOKE_ACCESS_CLIENT_ID to Doppler prd and let a deploy carry them, then add SMOKE_ACCESS_CLIENT_SECRET.
  • Xero counts are per entity, in the app's memory: a restart starts again. A success ends the run of failures; one run alerts once. Only the refresh is counted, which is where a dead connection shows (refresh token expired or revoked); a 401 on an API call with a still-valid access token shows up at the next refresh, within 30 minutes.
  • Database outage is tracked in the app's memory too. Every /mcp request fails the lookup while the database is down, so only the first failure of an outage alerts; a lookup that succeeds ends the outage, and the next failure alerts again if 15 minutes have passed since the last alert. Tested in tests/web/app-outage.test.ts.
  • Tests: tests/xero/auth-failures.test.ts, tests/xero/session.test.ts and tests/platform/alerts.test.ts, all with made-up ids and topics. The two scripts were run against a local fake server (up, Access redirect, no answer, no topic).
  • Not verified yet: the alert-main-failed job only runs when main fails, so it is first proven by a real failure. Test the channel itself with the command below.

What the Product Owner must do ​

  1. Phone: install the ntfy app (iOS or Android) and subscribe to a new topic with a long random name, e.g. getmy-books- followed by 24 random letters and digits. Keep "Instant delivery" on.
  2. Doppler getmy-books/prd: add ALERT_NTFY_TOPIC with that name. This turns on the failed-main alert at once (the runner already has DOPPLER_TOKEN_PRD) and, from the next deploy, the Xero alert.
  3. NAS compose file: reinstall the root-owned copy of infra/nas/docker-compose.yml at /usr/local/etc/getmy-books/docker-compose.yml (ADR 0023), since it now passes ALERT_NTFY_TOPIC to the app. Until then the Xero alert is only logged, not pushed.
  4. Test: doppler run -p getmy-books -c prd -- scripts/alert.sh "Test" "Alerts work" should reach the phone.
  5. For the /health check (done 2026-10-20):
    • Cloudflare Zero Trust: the Access application "GetMy Books health (public)" covers mcp.getmy.group/health only, with a Bypass policy (ADR 0038: /health stays bypassed). It returns only the status and the commit. Every other path still needs sign-in.
    • Doppler prd: HEALTH_CHECK_URL = https://mcp.getmy.group/health. A service token (HEALTH_CHECK_ACCESS_CLIENT_ID and _SECRET) is optional, and is sent only if both are set, for when /health goes back behind Access.
    • Nothing to switch on in GitHub: the Health workflow runs every 15 minutes. If the URL or the runner's prd token is missing, it goes red and alerts that monitoring is not configured.
  6. Optional: turn on email for failed Actions runs in your GitHub notification settings as a second channel.

Later ​

  • Email through the Graph mailbox once it exists, beside ntfy.
  • A self-hosted ntfy (or another push service) if the public server is unwelcome: set ALERT_NTFY_SERVER in Doppler prd (and pass it through the compose file).