Appearance
Plan: alerting
Status: built (AUDIT.md operations queue). Off until the Product Owner does the setup at the end; nothing fails while it is missing.
Note 2026-11-06: the pipeline moved to GitHub Actions (ADR 0046). The workflows are in .github/workflows/, the Health job runs on GitHub's runners (so it keeps watching while the Mac mini is off), and both read DOPPLER_TOKEN_PRD as a secret of the production environment.
Why
Three things can go wrong in production without anyone noticing: a deploy fails, the app stops answering, or a company's Xero connection dies. The Product Owner must hear about each one on their phone, without opening GitHub or a conversation.
Channel: ntfy push
- ntfy (
ntfy.sh): a free push service. The sender posts a message to a topic with one HTTP request; the phone app subscribed to that topic shows it as a notification. No account, no new server, no new setting in GitHub, Cloudflare or Entra. - Why not email: nothing sends email yet (the Microsoft Graph mailbox is only planned, see
email-reminders.md). When it exists,sendAlertandscripts/alert.share the two places to add it. - Why not GitHub's own notifications: they would not cover the app's own Xero alert. They can be turned on as well, at no cost, as a second channel.
- The topic is a password. Anyone who knows a topic name can read it, so the name is long and random, lives only in Doppler
prd, and every message carries no company, person, id or figure. At most it says what failed, a short commit id and a link to the GitHub run.
What alerts, and how
| Event | Where it is caught | What sends it |
|---|---|---|
| Main fails: checks, migration, deploy or the post-deploy smoke test | alert-main-failed job in .github/workflows/ci.yml, runs only when a job before it failed on a push to main | scripts/alert.sh, topic from Doppler prd via DOPPLER_TOKEN_PRD |
/health down | .github/workflows/health.yml, every 15 minutes, three tries 20 seconds apart | scripts/health-check.sh, then scripts/alert.sh; the run also goes red |
| Xero sign-in failing | src/xero/session.ts: 3 failed token refreshes in a row for one entity | logger.error with alert: 'xero_auth_failures' and the entity id, then sendAlert (src/platform/alerts.ts) with no identifiers |
| Xero reconnect needed | src/xero/session.ts: a refresh Xero refuses because the connection must be reconnected, once per entity until a refresh succeeds (SPEC-008 S18, #297) | logger.error with alert: 'xero_reconnect_needed' and the entity id, then sendAlert with no identifiers |
| Database unreachable at sign-in | src/web/app.ts: the /mcp principal lookup throws (the person gets a 503, not "no access") | logger.error with err on every failure, then sendAlert with no identifiers, once per outage and at most once every 15 minutes |
Notes:
- Reaching
/health. The app publishes no ports; its only way in is the Cloudflare tunnel. Access bypassesmcp.getmy.group/healthonly (ADR 0038), so the check is a plain request./healthreturns just the status and the commit. Every other path,/mcpincluded, still needs sign-in. - Service token only for the smoke test (2026-11-03, #81).
/healthis bypassed (ADR 0038), so neither the scheduled check nor the smoke test needs an Access service token to reach it. The only use for a token is the post-deploy smoke test's MCP check (scripts/smoke-test.sh, thesmoke-nasjob inci.yml): withSMOKE_ACCESS_CLIENT_IDandSMOKE_ACCESS_CLIENT_SECRETin Dopplerprdit GETs the smoke probe/smoke/mcpthrough Access; without them it checks/healthreports the deployed commit and skips the probe with a notice. Amended 2026-11-06 (Product Owner decision on #81, SPEC-017 S26):/mcpstill refuses a service token (no email, 403). The probe has its own Access application with a Service Auth policy, takes only the service token whose client id isSMOKE_ACCESS_CLIENT_ID, and answers only{"status":"ok","tools":N}. Set it up in this order, or main goes red: install the updatedenv-allowlistand compose file on the NAS, create the token and the application, addCF_ACCESS_SMOKE_AUDandSMOKE_ACCESS_CLIENT_IDto Dopplerprdand let a deploy carry them, then addSMOKE_ACCESS_CLIENT_SECRET. - Xero counts are per entity, in the app's memory: a restart starts again. A success ends the run of failures; one run alerts once. Only the refresh is counted, which is where a dead connection shows (refresh token expired or revoked); a 401 on an API call with a still-valid access token shows up at the next refresh, within 30 minutes.
- Database outage is tracked in the app's memory too. Every
/mcprequest fails the lookup while the database is down, so only the first failure of an outage alerts; a lookup that succeeds ends the outage, and the next failure alerts again if 15 minutes have passed since the last alert. Tested intests/web/app-outage.test.ts. - Tests:
tests/xero/auth-failures.test.ts,tests/xero/session.test.tsandtests/platform/alerts.test.ts, all with made-up ids and topics. The two scripts were run against a local fake server (up, Access redirect, no answer, no topic). - Not verified yet: the
alert-main-failedjob only runs when main fails, so it is first proven by a real failure. Test the channel itself with the command below.
What the Product Owner must do
- Phone: install the ntfy app (iOS or Android) and subscribe to a new topic with a long random name, e.g.
getmy-books-followed by 24 random letters and digits. Keep "Instant delivery" on. - Doppler
getmy-books/prd: addALERT_NTFY_TOPICwith that name. This turns on the failed-main alert at once (the runner already hasDOPPLER_TOKEN_PRD) and, from the next deploy, the Xero alert. - NAS compose file: reinstall the root-owned copy of
infra/nas/docker-compose.ymlat/usr/local/etc/getmy-books/docker-compose.yml(ADR 0023), since it now passesALERT_NTFY_TOPICto the app. Until then the Xero alert is only logged, not pushed. - Test:
doppler run -p getmy-books -c prd -- scripts/alert.sh "Test" "Alerts work"should reach the phone. - For the
/healthcheck (done 2026-10-20):- Cloudflare Zero Trust: the Access application "GetMy Books health (public)" covers
mcp.getmy.group/healthonly, with a Bypass policy (ADR 0038:/healthstays bypassed). It returns only the status and the commit. Every other path still needs sign-in. - Doppler
prd:HEALTH_CHECK_URL=https://mcp.getmy.group/health. A service token (HEALTH_CHECK_ACCESS_CLIENT_IDand_SECRET) is optional, and is sent only if both are set, for when/healthgoes back behind Access. - Nothing to switch on in GitHub: the Health workflow runs every 15 minutes. If the URL or the runner's prd token is missing, it goes red and alerts that monitoring is not configured.
- Cloudflare Zero Trust: the Access application "GetMy Books health (public)" covers
- Optional: turn on email for failed Actions runs in your GitHub notification settings as a second channel.
Later
- Email through the Graph mailbox once it exists, beside ntfy.
- A self-hosted ntfy (or another push service) if the public server is unwelcome: set
ALERT_NTFY_SERVERin Dopplerprd(and pass it through the compose file).