Appearance
Plan: deploy hardening (issues #75, #74, #73)
Status: plan, approved by the Product Owner on 2026-11-04 (DH-1; DH-6 to DH-8 decided the same day). DH-2 to DH-5 and DH-9 to DH-15 decided on 2026-11-06, with ADRs 0028, 0029 and 0030 accepted; DH-16 decided 2026-11-09 (Doppler prd; see Decisions). No infrastructure changed yet. Everything below was checked read-only against main at 57ad59a on 2026-09-30. Nobody has logged in to the NAS, and nothing has been changed in Doppler, the runners, the registry or Cloudflare. ADR drafts: 0028 (#75), 0029 (#74) and 0030 (#73), all proposed.
2026-11-09: slices 1 and 2 are not being built. #75 and #74 were closed as not planned on 2026-11-08, and the Product Owner accepted that on 2026-11-09 (status notes on ADRs 0028 and 0029). Three things follow:
- #76 (automatic rollback) has no image by digest to go back to. The previous release folder on the NAS stays the fallback. Putting it back means rebuilding that release, about 14 minutes. #76 needs a new, smaller design before it is built.
- Slice 3 (#73) still assumes
migrate-nasstops using port 5433 when it closes (the "#73 cannot go first" note below). With #74 dropped, step 3c (closing the port) needs another way to run migrations before its window, for example running them in the deploy job over SSH. Steps 3a and 3b, and DH-6 to DH-8, do not depend on this. - P1 / DH-13 is proven: a branch job asking for the
productionenvironment is refused (issue #71, run 42599859048, 2026-11-09).
2026-11-06: the code and the pipeline moved to GitHub (ADR 0046). DH-13 is answered with a GitHub environment (see P1 and the Decisions table), which supersedes #71 stage 2 as P1's route. Where this plan describes the previous git host, its registry or getmy-books-deploy holding DOPPLER_TOKEN_PRD, read ADR 0046 first; slice 1 is re-read against it before it is built.
Revised on 2026-09-30 after a second review. The main changes: slice 1a now waits for the #71 stage 2 hole to close (prerequisite P1); the automatic rollback uses a root-owned last-good record, the current env and only local images; the NAS enforces its own env allowlist; the backup runs after postgres starts; restores are proven before W2; there is a manual deploy path that does not depend on re-running a pipeline; and there is a disaster-recovery runbook.
Why
The deploy works (ADRs 0019 and 0023), but the adversarial review of 2026-09-29 found three weaknesses that stack on top of each other:
- #75: the NAS builds the image. Each deploy runs
pnpm installand a compile on the NAS. On 2026-09-29 that took 14m28s against a 15-minute limit. The build context comes from thedeployaccount. Nothing scans the image that runs, nothing signs it, and there is no ready image to fall back to. - #74: migrations run before the build, over the LAN, with no backup.
migrate-naschanges the schema, and then the old app serves on it for as long as the build takes. Nothing requires migrations to be backward compatible, nothing takes a backup first, and nothing can roll back. - #73: there is one database, and it sits on the LAN. The production app, the owner's local Claude Code (Doppler
dev) andmigrate-nasall use the NAS Postgres. The last two reach it through port 5433, which is published on every interface with password login. Local testing therefore changes production data and the real FEAW Xero, and every laptop holdingdevhas the owner (superuser) password.
Found on the way, and fixed here because it is cheap:
scripts/deploy-nas.shline 24 writes the whole Dopplerprdconfig into the release.envon the NAS. That puts the deploy SSH private key, and the ownerDATABASE_URLon NAS disk, in the current and previous release folders.- The local route and the NAS app refresh the same
xero_connectionrow with no compare-and-set. Xero rotates refresh tokens, so a race can end ininvalid_grantand cut production off from Xero. - Nothing stops
doppler run -c dev -- pnpm testfrom pointing the destructive test fixtures at the NAS database. Only the docs forbid it.
Target end state
- CI builds the image once per commit on
main. It scans it and blocks on fixed HIGH or CRITICAL findings, makes an SBOM, pushes it to the registry and signs it with cosign. - The NAS never builds.
getmy-books-upchecks the signature, pulls that exact digest and starts it. The NAS side of a deploy takes a couple of minutes. - Before any migration, the deploy takes a checked backup. It then migrates in a one-shot container on the compose network, from the same signed image, before the new app starts. A failed migration changes nothing, and the old app keeps serving.
- Every migration is expand-only. Contract changes wait at least one release. CI refuses a destructive migration without a marker, and refuses one that the deployed code cannot live with.
- Postgres publishes no port.
nc -z 192.168.0.80 5433fails from the LAN. - Doppler
devpoints at a local throwaway database, a separate Xero app on the Demo Company, and its own encryption key. Development and agents cannot reach production data. - CI holds no database credential except the throwaway one in
verify. No runner that a branch or pull-request workflow can reach holds aprdtoken, the signing key, the registry push token or a Docker socket (P1). - The NAS release
.envholds only allowlisted secrets,COMMIT_SHAandAPP_IMAGE_DIGEST. The NAS enforces that list itself, from a root-owned file. Security settings that are not secret (the Access pair,OWNER_EMAIL,MCP_ALLOWED_ORIGINS,XERO_WRITE_PRINCIPALS) live in Dopplerprd(DH-16, decided 2026-11-09), not in a root-owned file on the NAS. - A failed deploy puts the last release that passed
/healthback, from a root-owned record, with the current secrets and without the network.
What does not change: the deploy user, its forced command getmy-books-receive, the sudoers rule (getmy-books-up still takes no arguments), the nightly backup, the Cloudflare tunnel and Access set-up, and the MCP tools. No MCP tool touches the deploy.
Order and slices
The order is slice 0 (repo only), then #75, then #74, then #73:
- #73 cannot go first. Closing 5433 breaks
migrate-nas, which turns main red, anddeploy-nasneedsmigrate-nas. It also cuts the owner's local Claude Code off from real FEAW data before anything replaces it. - #74 needs #75. The in-deploy migrate step needs a ready image that contains the SQL files. Doing it on today's build-on-the-NAS path would add a second build to a job that already takes 14.5 minutes, and that work would be thrown away.
- #75 needs slice 0. The image has to carry the migration files, and the env allowlist should be in place before new secrets (the registry token and the cosign key) are added to
prd.
Prerequisites
P1 (blocks slice 1a): no branch or pull-request job can reach the signer, the push token or the build host. #71's 2026-09-30 comment leaves this open: "A PR can still name getmy-books-deploy in its own copy of the workflow." Anyone who can push a branch (people and agents) can therefore run a job on the deploy runner today, and that job can read DOPPLER_TOKEN_PRD. If 1a put COSIGN_PRIVATE_KEY, COSIGN_PASSWORD and REGISTRY_PUSH_TOKEN in prd and mounted /var/run/docker.sock on that runner (root on the Colima VM: build cache, loaded images, other containers), any branch pusher could sign an image the NAS would accept. Signing would then add nothing. So nothing in 1a that holds a secret or the socket starts until one of these is in place, decided as DH-13:
- A runner that only takes
mainpush jobs. Read-only first: does the CI host or its runner offer any way to restrict a runner or label to jobs from a protected branch or apushtorefs/heads/main? If it does, the build-and-sign runner is a separate registration with that restriction, and the check is proven by a branch that names the label and is never picked up. - #71 stage 2, extended to the build. If the CI host cannot restrict runners: a pull-based agent on the NAS takes
DOPPLER_TOKEN_PRDout of CI for the deploy, and a pull-based builder outside the CI host (a launchd service on the Mac mini with its own Colima profile, watchingmainthrough the git host's API) builds, scans, pushes and signs. No workflow can name it. CI keepsverifyandsecurityonly. - Restrict who can push branches to the Product Owner and Rob, with agents working from forks. This changes the SGE workflow and is listed only for completeness.
Note 2026-11-06 (Product Owner, DH-13 reopened and decided): a fourth route, taken instead of (2). The code moved to GitHub (ADR 0046), and every production secret is a secret of the GitHub environment production, whose deployment branches are main only. Only main-only jobs name it, so no branch or pull-request job can read the secrets, whichever runner it names; tests/infra/ci-workflow.test.ts keeps the workflows to that. P1 closes when a branch run is shown to be refused the environment. The signer and push token (DH-3) go in the same environment. Options (2) and (3) below stand as history.
Recommended (2026-09-30): check (1) read-only; if it is not available, (2), built as #71 stage 2 before slice 1a. Until P1 is closed, 1a may run only a secret-free build-and-scan dry run (no push, no signature, no socket on the deploy runner) to prove the cross-build and the scan baseline. The risk is not accepted in the meantime; it is scheduled.
P2 (blocks W2): evidence that backups exist and that a restore works. See "Backups and restore". W2 starts only from a recorded, successful run of the new restore test.
P3 (blocks 1d): a manual deploy and rollback path that does not depend on re-running a pipeline. The previous host's API could not re-run jobs and its UI Re-run returned a 500 before. See "Manual deploy path".
Change windows
A window must not overlap a deploy. Two controls, both from slice 0 and 1b:
- NAS side: if
/usr/local/etc/getmy-books/maintenanceexists,getmy-books-uprefuses with "maintenance window open" before touching anything. The window starts by creating it and ends by removing it. - CI side:
build-image,migrate-nasanddeploy-nasskip when the repository variableDEPLOY_PAUSEDistrue. Setting it is part of each window's approval. Merges may continue; they deploy when the window ends.
Every slice is split into repo work and infrastructure work:
- Repo work is ordinary SGE PRs. They merge with no infrastructure change and are inert on the NAS until the matching window, because the NAS only reads its root-owned copies of the scripts and the compose file.
- Infrastructure work is a short NAS window, run between deploys with no deploy job in flight (see "Change windows"). Each one first copies the current root files to
/usr/local/etc/getmy-books/rollback/<date>/, so undoing it is one copy back. There are three windows, one per issue (W1 #75, W2 #74, W3 #73), which keeps each one small enough to check and to undo.
Slice 0: repo-only safety work (no approval needed)
Each item is its own small PR through /sge:sge-implement.
0.1 Env allowlist in scripts/deploy-nas.sh.
- Replace
doppler secrets downloadwith an explicit list of the namesinfra/nas/docker-compose.ymlinterpolates:POSTGRES_PASSWORD,APP_DB_PASSWORD,OWNER_EMAIL,XERO_CLIENT_ID,XERO_CLIENT_SECRET,XERO_TOKEN_ENCRYPTION_KEY,COMPANIES_HOUSE_API_KEY,CF_ACCESS_TEAM_DOMAIN,CF_ACCESS_AUD,MCP_ALLOWED_ORIGINS,ALERT_NTFY_TOPICandGETMY_BOOKS_TUNNEL_TOKEN, plusCOMMIT_SHA(andAPP_IMAGE_DIGESTfrom slice 1b). - A test asserts that every
${VAR}in the compose file is either on the list or has a default, that every name on the list appears in the compose file, and that the.envnever holdsDATABASE_URL,NAS_DEPLOY_*,DOPPLER_*,COSIGN_*orREGISTRY_*. getmy-books-receivekeeps onlycurrentandprevious, so two deploys after this merges, no release on the NAS holds those secrets. (Copies may still sit in off-site backups; see "Backups and restore".)- This list is enforced on the sending side only. The NAS side follows in 1b:
getmy-books-upchecks the env file against its own root-owned allowlist, and the non-secret security settings move off the deploy path. - Rollback: revert the PR.
- The owner notices: nothing.
0.1b Deploy pause switch. build-image (when it exists), migrate-nas and deploy-nas gain if: vars.DEPLOY_PAUSED != 'true'. ci.yml gains workflow_dispatch, and every deploy job also requires github.ref == 'refs/heads/main', so a fresh run of main's head never depends on the Re-run button. Rollback: revert the PR.
0.2 A production guard on destructive tooling.
- Copy the guard in
perf/load/server.tsintodb:setup,db:push,db:create-role,db:bootstrap,scripts/db-seed-owner.ts,drizzle.config.tsand the database test set-up. - It refuses any host other than
localhost,127.0.0.1,::1orpostgres, and it refusesNODE_ENV=production. The tests have no override. db:migrateis left alone for now, becausemigrate-nasstill needs it until slice 2c.- Rollback: revert the PR.
- The owner notices: nothing, unless someone points a test at the NAS, which now fails straight away.
0.3 Xero refresh compare-and-set.
src/xero/entity-token-store.tssaves a refreshed token onlywhere token_version = <the version read>.- If no row is updated, the caller re-reads the row and uses the token that won, instead of refreshing again.
- This is needed while the laptop and the NAS share the row (until slice 3), and it does no harm afterwards.
- Rollback: revert the PR.
- The owner notices: fewer "reconnect Xero" surprises.
0.4 An image that can migrate, and bootstrap an empty cluster.
pnpm buildcopiessrc/platform/migrations/*.sqlintodist/platform/migrations/(the pathreadMigrationsreads throughnew URL('./migrations/', import.meta.url)), andpolicies.sql,bootstrap.sqlandcreate-role.sqlintodist/platform/.- A compiled
dist/platform/migrate-cli.js, in order:- creates the
approle if it is missing (thecreate-role.sqllogic; today onlydb:setupdoes this, sopolicies.sql'sto appfails on a fresh or restored cluster); - sets the
apppassword fromAPP_DB_PASSWORD, as a SCRAM-SHA-256 verifier computed in the client, never a plain-text literal, so a failure logged by Postgres (a lock timeout, say) never holds the password; - calls
runMigrations.
- creates the
- Steps 1 and 2 live only in
migrate-cli, never inrunMigrations.migrate-nasrunsrunMigrationswith Dopplerdev'sAPP_DB_PASSWORDduring the 2b overlap, anddev,dev_personalandprdall define it: in the shared function each deploy would flip production's app password twice. runMigrationssetslock_timeout = '5s'andstatement_timeout = '120s'inside its transaction, so a lock fight with the running app fails fast instead of hanging the deploy.tsconfig.build.jsonkeepsrootDir: "src", sodist/main.js, theCMD, theHEALTHCHECKand themigrate-clipath do not move. A test assertsdist/main.jsexists after the build.- Tests: the built
distholds the SQL files at those paths; the CLI migrates an empty CI Postgres with noapprole, after whichappcan log in withAPP_DB_PASSWORDand a query asappis scoped by RLS; running it twice changes nothing. - Rollback: revert the PR.
- The owner notices: nothing.
0.5 Dockerfile ready for a cross-build.
- The build stage becomes
FROM --platform=$BUILDPLATFORM node:22-bookworm-slim@sha256:<pin> AS build. The runtime stage uses the same base, pinned by digest. - The pin is the multi-arch index digest, never a single-platform manifest digest (for example one read from the NAS). With an amd64 manifest pinned, the build stage on the arm64 runner would be amd64 and fail with
exec format error, since no QEMU is installed. A test reads the pin, andbuild-imagechecks withdocker buildx imagetools inspect --rawthat it is an index listing bothlinux/amd64andlinux/arm64. - The runtime stage keeps no
RUNsteps. Because of that, and because every production dependency is pure JavaScript, an arm64 host can produce a linux/amd64 image with no emulation. - Add a
packageManagerpin topackage.json, so corepack's pnpm cannot drift. Add the labelorg.opencontainers.image.revision. - Keep
ARG COMMIT_SHAand theHEALTHCHECK, which cloudflared'sdepends_onneeds. .dockerignoregainsinfra,scripts,**/.env*andstatements.- Fix
base_images()inscripts/security/trivy.shso it skips a--platform=flag and readsname@sha256:images. - Rollback: revert the PR. The NAS build still works, because
$BUILDPLATFORMthere is amd64. - The owner notices: nothing.
0.6 Migration rules and lint.
- Write the rules below (see "Migration rules") into ADR 0029 and
docs/sge/change-protocol.md. scripts/check-migrations.tsruns inverifyand fails if:- a migration file that exists on
mainwas edited or deleted; - a file contains a destructive statement without a
-- contract: #<issue>header; - a statement is neither guarded (
if not exists,if exists,on conflict do nothing) nor one of the idempotent forms that have no guard syntax:grant,revoke(contract, so it needs the marker),alter table … enable row level security,create or replace function, and ado $$ … $$block (which must test its own condition, and is flagged for review); - a new migration file is added while
.migration-freezeexists in the repo (added by the 2b PR, removed by 2c), so the freeze is enforced, not a convention.
- a migration file that exists on
- Files 0000 to 0006 are grandfathered by name. 0004's
add constraint … foreign keycounts as a tighter constraint under the rules, but it is already applied everywhere. - The lint also covers
policies.sql,bootstrap.sqlandcreate-role.sql, which are re-applied on every deploy and are not append-only: a removed or narrowed policy, arevoke, or adropin them needs the contract marker in the PR description and the N-1 test. policies.sqlstops dropping and re-creating every policy on every deploy, which takes an ACCESS EXCLUSIVE lock on every RLS table against the 5-secondlock_timeout. It creates a policy only when it is missing; a changed policy definition is a numbered migration.- The lint has its own failing-first test with a deliberately breaking migration, and one per rule above.
- Rollback: revert the PR.
- The owner notices: nothing.
Slice 1: #75, build in CI, sign, deploy by digest
Before 1a (read-only, no approval beyond the checks themselves):
- P1 is closed (see Prerequisites). Where this section says
getmy-books-deploy, read "the build-and-sign host chosen in DH-13". - Scan baseline. Run the blocking scan (
trivy image --platform linux/amd64 --severity HIGH,CRITICAL --ignore-unfixed) once against today's pinned bases (node:22-bookworm-slim,postgres:17-alpine,cloudflare/cloudflaredat the chosen version) and record the result in the PR. ADR 0027 already records fixed HIGH and CRITICAL findings, and the only planned exception covers npm and corepack, not Debian packages. If the baseline is not clean after a digest bump, 1a does not start. - Exception process (
security/image-exceptions.json, the same form asaudit-exceptions.json): each entry names the CVE, the package, the image, why it does not apply or cannot be fixed yet, who approved it, and an expiry of at most 30 days. The Product Owner approves each one in its PR. Break-glass for an urgent fix blocked by a newly published CVE or advisory: add a dated exception in the same PR as the fix, approved by the Product Owner. There is no bypass flag. The same process coverspnpm audit, which also becomes a deploy gate throughneeds: security. - Bump owner. A scheduled monthly issue, assigned to the Product Owner, bumps all three base digests.
postgresandcloudflaredare scanned (blocking) by thesecurityjob whenever their pin changes, and in a report-only weekly run, so their known HIGHs are not frozen in unseen. - Arch and host checks (item 5): Colima's
docker version,docker info --format '{{.DriverStatus}}'(classic or containerd image store),docker buildx version, and the VM's CPU, memory and free disk fromcolima list. Trivy runs with--platform linux/amd64on the arm64 host. - Registry auth checks (items 6a to 6c): token login at
/v2/tokenfor a web-login-prohibited user; push by thepackages-citeam to an org-owned container package; and a root pull on the NAS through Cloudflare (the hairpin path).
1a dry run, before P1 (built 2026-11-03, #75). build-image exists as a secret-free dry run on the general runners: steps 1 to 4 below, via scripts/image/build-image.sh, with no push, no signature and no secret. It runs only on workflow_dispatch with the build_image box ticked, so a plain dispatch of main (the 0.1b deploy route) never runs it, until one green manual run shows the general runners give jobs a Docker daemon; then the owner switches it to pushes to main. Its Trivy database still comes from ghcr.io at job time (the mirror below is not built). The first run blocks on fixed Debian findings (perl-base, libpcre2-8-0) in the pinned node:22-bookworm-slim, as the scan baseline item above expects; a digest bump clears them.
1a. Shadow build (repo, plus approvals 1 to 6c). A new job, build-image:
- Runs on pushes to
mainonly,needs: [verify, security](so the security job finally gates deploys; see the exception process above), on the host chosen in DH-13, timeout 15 minutes. Pull requests never build, push or sign. - Trivy's vulnerability database is not downloaded from ghcr.io in the job: that download is a known flaky rate limit, and a red
mainstops feature work (rule 7). A scheduled job mirrors the database into the registry daily;build-imagepulls the mirror and uses--skip-db-update, and fails only if the mirror is more than 72 hours old. - The job container is
node:22-bookwormplusdocker-ce-clianddocker-buildx-plugin, at pinned versions from download.docker.com. It uses the Colima daemon through the mounted/var/run/docker.sock. cosign v2.x comes from a pinned release, with its checksum checked. - Steps:
docker buildx build --platform linux/amd64 --provenance=false --sbom=false --build-arg COMMIT_SHA=<sha> --load -t ghcr.io/feaw-serv/getmy-books:<sha> .- Guards, each of which fails the job:
- the image config's architecture is not
amd64; - there is a
*.nodefile or alinux-arm64prebuild innode_modules; - any layer is 90 MB or more (Cloudflare's request-body limit).
- the image config's architecture is not
trivy.sh built, a new mode, scans the loaded image. It blocks on fixed HIGH or CRITICAL findings. The npm and corepack copies bundled in the Node base (the app never runs them, sinceCMDisnodedirectly) are covered by a dated exception insecurity/, in the same form asaudit-exceptions.json, with an expiry. This is not a permanent--skip-dirs. A runtime without npm (distroless) is a follow-up (DH-4).- A CycloneDX SBOM of the image, uploaded as an artifact.
docker push. Read back the manifest digest, and check that its config digest equals the config digest of the image that Trivy scanned. That way the image signed is the image scanned, with no rebuild in between. With the classic image store,docker image inspect .Idis the config digest; with the containerd store it is the manifest digest, so the step reads the config digest in the way the store recorded in item 5 requires, and its test covers both.cosign sign --key env://COSIGN_PRIVATE_KEY --tlog-upload=false -a commit=<sha> -a ref=refs/heads/main <repo>@sha256:<digest>cosign verifyof the same reference, with the committedinfra/nas/deploy/cosign.puband the same annotations. This proves that the registry stores and serves the.sigtag.- Output the digest.
deploy-nasdoes not use the digest yet: the NAS still builds.- Run it for at least three
maindeploys, and check each time that the image label's commit matches/health. alert-main-failedgainsbuild-image(and every later job onmain) in itsneeds, so a failed build that leavesdeploy-nasskipped still alerts. A test asserts that everymainjob is in that list.- Time budget.
ci.yml's header target is "under 6 minutes". Main today runs in about 2.5 to 6.4 minutes end to end, deploy included, because the NAS build is usually warm. Withbuild-imagein series (the buildx build, the scan, a push through Cloudflare, sign and verify) and then pull, backup and migrate, that target cannot hold. The shadow runs measure each step, and the ADR records the amended target: pull requests under 6 minutes;mainto deployed under 12 minutes, withbuild-imageunder 6 of those. If the measured figure is above that, 1b does not start. All of this shares one capacity-1 runner with the 15-minute health job, which is counted in the budget. The N-1 job (2a) runs on pull requests that touch the schema, not onmain. - Rollback: delete the job. It has no effect on production.
- The owner notices: nothing yet.
maintakes longer until W1 removes the NAS build.
If signatures cannot be stored in or read from the registry, W1 does not go ahead. The options come back to the Product Owner.
1b. Digest deploy, repo side (merged, inert until W1).
deploy-nasgainsneeds: build-imageand writesAPP_IMAGE_DIGEST=sha256:<64hex>into the release.env, next toCOMMIT_SHA. Its timeout stays at 15 minutes until 1d. A cold NAS build took 14m28s, and until 1d a NAS build of the previous commit is still the fallback, so the timeout must not be cut while that fallback exists.- The tar still carries the whole tracked tree until 1d, because the NAS's current compose file builds with context
../.., and because the W1 fallback builds from it. - Root-owned state in
/usr/local/etc/getmy-books/state/(root 0700), whichgetmy-books-receivecannot write:last-good: the commit and digest of the last release that passed/health, and the date it was verified. Written byuponly after/healthpasses.failed/<commit>: written when a release fails/health.env-allowlist: the only names the release.envmay set.config.env(root 0644): the security settings that are not secret:CF_ACCESS_TEAM_DOMAIN,CF_ACCESS_AUD,OWNER_EMAIL,MCP_ALLOWED_ORIGINS, and laterXERO_WRITE_PRINCIPALS. The compose file reads it withenv_file:forapp. Changing any of them is a NAS window, not a Doppler edit plus a deploy. Superseded 2026-11-09 by DH-16: these settings stay in Dopplerprd; there is noconfig.env.
infra/nas/deploy/getmy-books-upv2, still with no arguments from thedeployaccount:- Refuses if the maintenance flag exists (see "Change windows").
- Resolves
currentas today. Refuses ifstate/failed/<commit>exists: an admin re-runningupby hand (after a DSM update, say) must not redeploy a release that already failed. The admin usesgetmy-books-up-last-goodinstead (below). - Checks the release
.envagainststate/env-allowlist: every line must beNAME=valuewith a name on the list, with no duplicates. Anything else (includingCOMPOSE_*,DOCKER_*, or a moved setting such asOWNER_EMAIL) refuses the deploy.COMPOSE_PROJECT_NAMEwould otherwise beat the top-levelname:and start a second project on an empty<name>_postgres-datavolume, sharing the tunnel token, while the backup still readsgetmy-books-postgres-1. - Runs every compose command with
-p getmy-books, an explicit--env-file, and a scrubbed environment (env -iwith onlyPATHandHOME), so nothing inherited can set interpolation or compose control variables. - Reads
APP_IMAGE_DIGEST, checks it against^sha256:[0-9a-f]{64}$, and exports it. - Runs
compose --profile migrate config(all profiles, so a latermigrateservice is included) throughcompose-check, withEXPECTED_APP_IMAGE=ghcr.io/feaw-serv/getmy-books@<digest>. - Runs
cosign verify --key /usr/local/etc/getmy-books/cosign.pub --insecure-ignore-tlog=true -a commit=<release commit> -a ref=refs/heads/main <expected image>and refuses if it fails. That refuses an unsigned image, an unknown digest, and a validly signed old image replayed under a new commit. - Pulls with
docker pull <expected image>.DOCKER_CONFIG= /usr/local/etc/getmy-books/dockeris set only on thecosignanddocker pullcommands. Setting it globally would also move where the docker CLI looks for thecomposeplugin ($DOCKER_CONFIG/cli-plugins). Compose never pulls the app; it runs with--pull neverfor it. - Uses
docker compose(the plugin) only. Ifdocker compose versionfails, it refuses; it never falls back to standalonedocker-compose, which may differ in--wait,profilesandrun --no-deps. up -d --no-build --pull never --wait, while the old app kept serving until this point.- The existing
/healthcommit check. On success, writesstate/last-good(commit, digest, "verified"). - Automatic rollback if
/healthfails:- writes
state/failed/<commit>; - takes the digest and commit from
state/last-good, not fromprevious; - builds the rollback env from the current release
.env, replacing onlyAPP_IMAGE_DIGESTandCOMMIT_SHA. The current env holds today's secrets: after anAPP_DB_PASSWORDrotation (which the migrate step applies), a new Xero client secret or any other rotation, the previous release's env would start the old image with stale credentials, and the rollback itself would fail; - needs no network: the last-good digest was verified when it was deployed,
upchecks that it is still present locally (docker image inspect), and starts it with--pull never. If the registry or Cloudflare is the cause of the failure, the rollback still works; - checks
/healthfor the last-good commit, sends ntfy ("deploy of <sha7> failed; rolled back to <sha7>") from the rootalert.env, and still exits 1, so CI goes red; - if there is no
last-goodyet (the first deploy after W1), it says so and exits 1. See W1 for that case.getmy-books-receivestill rotatescurrentandprevious, but rollback no longer depends on them, so a second consecutive failure cannot "roll back" to a known-bad release.
- writes
infra/nas/deploy/getmy-books-up-last-good(root only, not in the sudoers rule): startsstate/last-good's digest with the current env, offline, then checks/health. This is the admin's manual rollback and the "reinstall after a DSM update" command.- Image clean-up on the NAS never removes the
last-gooddigest. getmy-books-compose-checkv2 keeps every existing rule, and also refuses (since 2026-11-03 the check is an allow-list, #80, ADR 0023 §4a, and it already refuses an image not pinned by@sha256, #77; these become additions to that list):build:andpull_policy: build;- any service
image:not pinned by@sha256:<64hex>; - an
app(and, later,migrate) image that differs fromEXPECTED_APP_IMAGE; - an
env_file:other than the release.envand the rootconfig.env. It reads the output ofconfigwith every profile active. A test proves that a bad service behind a profile (a bind mount,privileged, an unpinned image) is refused.
infra/nas/docker-compose.ymlv2:appusesimage: ghcr.io/feaw-serv/getmy-books@${APP_IMAGE_DIGEST:?}, with nobuild:, andenv_file: /usr/local/etc/getmy-books/config.env;postgresispostgres:17-alpine@sha256:<the digest running today>, on the same 17 major, so the data volume stays compatible;cloudflarediscloudflare/cloudflared:<version>@sha256:<digest>;- the postgres port stays until W3.
scripts/deploy-nas.shdrops the names that moved toconfig.envonce W1 is done (a follow-up PR, so the old NAS files still get them until then).infra/nas/deploy/env-allowlistis committed, and a test keeps it equal to the sending-side list.cosign.pubandcosign.sha256(the checksum of the NAS cosign binary) are committed underinfra/nas/deploy/.- Tests are rewritten first:
tests/infra/compose-check.test.ts(fixture, profiles and ordering assertion),tests/infra/receive.test.ts, and a newtests/infra/up.test.tsthat runsupagainst a stubdocker: env allowlist refusals,-p getmy-books, the failed-release refusal, the last-good rollback with a rotated password, and a rollback with the registry unreachable. - Rollback: revert the PR. The old NAS
upignoresAPP_IMAGE_DIGESTand still builds. - The owner notices: nothing yet.
1c. W1, the NAS switch-over (approvals W-1 to W-3 and 7 to 12a).
- Open the window (maintenance flag,
DEPLOY_PAUSED). Save the current root files to the rollback folder. - Install cosign,
cosign.pub, the pull credential, the state folder,env-allowlist,config.envandalert.env. - As root, dry-run, each of which must pass:
docker compose version(the plugin, root's normal environment);cosign verifyanddocker pullof the latest shadow digest, withDOCKER_CONFIGset on those two commands only;docker compose -p getmy-books --profile migrate configof the new compose file through the newcompose-check;- that the installed compose supports
up --pull never --waitandrun --no-depswith profiles.
- Install
up,up-last-good,compose-checkand the compose file from the merged commit. - As root on the NAS, run
sudo /usr/local/sbin/getmy-books-upagainst the release already incurrent(the latestmaindeploy, whose.envcarries a digest since 1b merged). Watch it. Declared effect: the newimage:strings forpostgresandcloudflaredmake Compose recreate both, so the database restarts (seconds; the app reconnects) and the tunnel drops briefly. Tell the owner before the window. - Close the window.
The first digest deploy has no automatic rollback, because there is no last-good yet. If it fails: copy the rollback files back and run the old sudo /usr/local/sbin/getmy-books-up by hand on the NAS. The old script builds from current, which still holds the whole tree, and it has no CI timeout, so a cold 15-minute build is fine. Do not rely on re-running a CI job for this.
- Rollback: copy the four files back and run the old
upby hand. - The owner notices: deploys finish in a few minutes, not 15, and a failed deploy puts the last good version back on its own, with an alert.
1d. Tidy-up after two clean digest deploys (repo, plus approvals 13 to 17b). Preconditions: state/last-good is set; the automatic rollback has been exercised once on purpose (acceptance test 11); P3 is met, including one rehearsal of getmy-books-up-last-good.
- This is the point of no return for NAS builds. After 1d, the tar shrinks, the build cache is pruned and the old images are removed, so a NAS build is no longer a fallback. The only rollback is to a signed digest that is still on the NAS (
up-last-good) or in the registry. - The tar shrinks to
infra/nas/docker-compose.ymlplusinfra/nas/.env, which is what receive requires. deploy-nas's timeout drops to 10 minutes.- Check that no release
.envon the NAS contains aNAS_DEPLOY_name (a count only, no values). - Prune the NAS build cache, rotate the deploy SSH key, delete the stale
prdnames and add a package clean-up rule. - Update ADRs 0019, 0023 and 0027.
Manual deploy path (P3)
W1, W2 and W3 all need a deploy after the files change. None of them depends on re-running a pipeline, because the previous host's API could not re-run jobs and its UI Re-run could return a 500:
- Redeploy what is in
current: an admin on the NAS runssudo /usr/local/sbin/getmy-books-up. It refuses a release marked failed. - Put the last good release back: an admin runs
sudo /usr/local/sbin/getmy-books-up-last-good. Offline. - Deploy
main's head afresh:workflow_dispatchonci.ymlfrom the Actions page (0.1b). It deploysmain's head only, never a chosen commit.
Slice 2: #74, backup first, migrate inside the deploy
Before 2a merges (P2, read-only, run by an admin on the NAS and pasted into the PR; counts, names, ages and sizes only, no contents):
- whether the 02:30 backup line is in
/etc/crontabtoday, and whether a DSM Task Scheduler task exists for it; - the newest nightly dump's name, age and size, and the number of dumps;
- the last line of
/var/log/getmy-books-backup.log, and of the restore-test log if one exists; - Hyper Backup's job scope (which shared folders and homes it copies), destination, schedule and retention (DH-12). If
homes/deployis in scope, today's full-prdrelease.envfiles (the deploy SSH key, the ownerDATABASE_URL) are already in off-site retention, which makes the 1d key rotation and the W3 password rotations mandatory, not optional, and the retention of those versions a question for the Product Owner.
2a. Repo (merged, inert until W2).
- A
migrateservice in the compose file:- the same
image:digest asapp; command: ["node", "dist/platform/migrate-cli.js"];DATABASE_URL: postgres://${POSTGRES_USER:-getmy_books}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-getmy_books}, plusAPP_DB_PASSWORD;depends_on: postgres: service_healthy,restart: "no",profiles: ["migrate"];- no ports and no volumes, so it passes
compose-check, which reads the--profile migrateoutput (1b) and so does see it.
- the same
migrate-clicreates theapprole if missing and sets its password as a client-computed SCRAM verifier (0.4). Rotating the app password then becomes a Doppler change plus a deploy, and the automatic rollback keeps working after it because it reuses the current env (1b).getmy-books-backupgains apredeploy <sha7>mode:- it writes
/volume1/backups/getmy-books/predeploy/getmy_books-<UTC>-<sha7>.dump; - it applies the same
PGDMPandpg_restore -lchecks; - it keeps 5;
- the nightly glob does not see that folder, so nightly retention is untouched.
- it writes
getmy-books-upv3, after the pull and beforeup:up -d --no-build --pull never --wait postgres. If postgres cannot start, refuse.- Then the pre-deploy backup, printing its path and size. That line appears in the CI log. If the backup fails, refuse the deploy. The backup is skipped only when the
getmy-books_postgres-datavolume was created by this very step (it did not exist before step 1) and the database holds no tables. If the volume existed but the container did not (aftercompose down, a DSM update or a manual removal), the backup still runs. Any other doubt refuses. --profile migrate run --rm --no-deps migrate. If it fails, refuse with "migration failed; app not restarted". The old app keeps serving the old schema.up -d --no-build --pull never --wait, then/health, then the v2 last-good rollback on failure.
getmy-books-restore-testv2 runs the manual restore runbook's exact commands, so the runbook is rehearsed every week:- takes a dump path (default: the newest nightly), so pre-deploy dumps can be tested too;
- starts a throwaway compose project (
-p getmy-books-restoretest) on an--internalnetwork, with the same pinned postgres digest as production and the currentlast-goodapp image; - creates the
approle, thenpg_restore --clean --if-existsas the runbook does, without--no-owner --no-privileges, so owners, grants, default privileges and RLS policies are restored and proven; - runs
migrate-cliagainst it (the deploy path on restored data); - counts tables and
pg_policiesagainst the expected numbers; - queries as
appwith a principal that belongs to no entity and expects zero rows from an RLS table, then with the FEAW principal and expects rows (row counts only in the log); - starts the app container against it and checks
/health; - tears the project down, and sends ntfy on failure.
- The backup and the restore test run from DSM Task Scheduler, not
/etc/crontab, which DSM regenerates from Task Scheduler and on updates (the deploy-shell fix already uses a scheduled task for this reason). The existing 02:30 line moves to Task Scheduler too. - Dead-man check, not on the NAS. The backup and the restore test each ping an external heartbeat monitor on success (DH-14). It alerts when the nightly ping is more than 26 hours old or the weekly one more than 8 days old. A job that never runs cannot send its own failure alert, so ntfy-on-failure is not enough on its own.
- The backup script sends ntfy on failure too. The topic comes from the root-owned
alert.envinstalled in W1. - An N-1 compatibility test in
verify. On a PR that touchessrc/platform/migrations/,policies.sql,bootstrap.sql,create-role.sqlorsrc/platform/migrate*.ts:- it builds the throwaway schema the way production does, from
create-roleplusmigrate-cliwith the PR's files, not fromdb:setup, which usesdrizzle-kit pushand would testschema.tsinstead; - it then runs
origin/main's database tests (tests/platform,tests/banks,tests/xerostore tests) against it. That is "a deliberately breaking migration fails CI" for changes the lint cannot see by keyword. It runs on pull requests only.
- it builds the throwaway schema the way production does, from
- Rollback: revert the PR.
2b. W2 (approvals W-1 to W-3, 17c and 18 to 21). Starts only when P2 holds: one run of restore-test v2, by hand, has passed against the newest nightly dump.
- Open the window. Save the current root files.
- Create the pre-deploy folder.
- Install
getmy-books-backup,getmy-books-restore-test,getmy-books-upv3 and the compose file v3. - Create the Task Scheduler tasks (nightly backup, weekly restore test) and remove the
/etc/crontabbackup line once the task is proven. - As root,
sudo /usr/local/sbin/getmy-books-up(the manual path), and watch it. Close the window. - Run restore-test v2 against the new pre-deploy dump.
migrate-nas keeps running for now. Double-running is safe: the migrations are idempotent, the advisory lock serialises the two runs, and migrate-nas never touches the app password (0.4). New migrations are frozen until 2c: the 2b PR adds .migration-freeze, and the lint enforces it.
- Rollback: copy the v2 files back and run the old
upby hand. - The owner notices: every deploy log shows "backup ok … predeploy/…" before "migrate ok".
2c. After two clean in-deploy migrations (repo, plus approvals 22 and 23).
- Delete the
migrate-nasjob.deploy-nasthen needsbuild-imageonly. scripts/db-migrate.tsgets the 0.2 guard too, since nothing needs it to reach the NAS any more.- Update ADR 0014.
- Remove the dev
DOPPLER_TOKENfrom the deploy runner. - Remove
.migration-freeze; the freeze ends.
Slice 3: #73, separate dev database, close port 5433
This slice waits for the local Claude Code decision (DH-6 to DH-8), for the DH-6 spike, and for anything that decision needs to be built first.
3a. Repo.
.mcp.json: the local stdio server keeps its namegetmy-books-mcpuntil DH-6 settles naming. Renaming it breaks CLAUDE.md line 40, README lines 79-80, tool names in agents' allowlists and memory, and, worse, the conversation-start deadline heads-up (SPEC-004): oncedevis the local Demo database, a local server would show Demo's deadlines, not FEAW's. FEAW's Jun-Aug VAT return is due 7 October, so losing the real heads-up is a product regression. 3a does not merge until the owner's default Claude Code session still shows FEAW's heads-up through the chosen real-work route, and the dev server's start-of-conversation instruction is dropped or marked as Demo.- The connect scripts (
scripts/xero-connect.ts,starling-connect.ts,statement-account-add.ts) go into the image without changingtsconfig.build.json'srootDir: "src": widening it would move the output todist/src/main.jsand breakCMD,HEALTHCHECKand themigrate-clipath. They get their own build config (tsconfig.tools.json,outDir: dist-tools), copied into the runtime stage as/app/dist-tools/. A test assertsdist/main.jsanddist-tools/scripts/xero-connect.jsboth exist. docs/dev/local-postgres-tests.mdgains "seed a dev database":db:setup, thendb-seed-ownerwithENTITY_NAME=Demo(never "FEAW", because migration 0005 grants FEAW by name), thenxero-connectto the Demo Company.- The compose file drops the postgres
ports:and its header comment is rewritten.compose-checkv3 refusesports:on every service. - Update README lines 79-84, CLAUDE.md if the server name changes, and ADR 0030.
- Anything DH-7 and DH-8 decide (the web write gate, and statement CSV content) ships as its own spec change first.
3b. Doppler split (approvals 24 to 27). Create the dev Xero app, then repoint dev and dev_personal. This is the moment local tools stop seeing production. Only do it once the owner's real-work route (DH-6) works for them.
3c. W3 (approvals W-1 to W-3 and 28 to 31).
- Open the window. Install the compose file with no ports and
compose-checkv3, then runupby hand. Postgres is recreated, which takes seconds. - Rotate the owner password, the app password and the production Xero client secret, because all three sat on laptops through
dev. The automatic rollback reuses the current env (1b), so a failed deploy after a rotation rolls back with the new secrets. - From a LAN machine, check that
nc -z -w3 192.168.0.80 5433fails. - Close the window.
- Rollback: copy the compose and
compose-checkfiles back and runupby hand. The port is back. - The owner notices: local Claude Code shows demo data. Real FEAW work goes through the route chosen in DH-6.
Local Claude Code: the decision
Today the owner's local Claude Code is the only route with Xero writes (the web route sets xeroWrites: false, src/web/app.ts:75), with Amex statement files (read from the local disk), and with the connect scripts. All of it works against production through port 5433. Closing the port ends that, so something must replace it.
| Option | What it is | For | Against |
|---|---|---|---|
| (a) Deployed endpoint | claude mcp add --transport http getmy-books https://mcp.getmy.group/mcp, signed in through the same Cloudflare Access login as the claude.ai connector | No database credentials on laptops; one Xero token refresher, which ends the refresh race; one ledger_write duplicate guard; identity per person from Access instead of OWNER_EMAIL; fits "MCP first" | The web route is read-only by product decision (README), so Xero writes need a write gate (DH-7). Amex files cannot be read from a laptop, so they need a tool that takes CSV content (DH-8). Connect scripts need a break-glass runbook |
| (b) SSH tunnel | Postgres bound to 127.0.0.1:5433 on the NAS; a dbtunnel user whose key may only forward to that port; a prd_local Doppler config holding the app role URL only | Everything local keeps working unchanged: writes, Amex, the connect scripts | Production app credentials and real data stay on laptops. It needs a new DSM user, which DSM resets to /sbin/nologin (as happened to deploy, #69), so the shell-restore task must cover it too. sshd may need a Match block for forwarding, and DSM updates can reset that |
| (c) Local dev database | Local Postgres (docker-compose.dev.yml or localhost:55432), a separate Xero app on the UK Demo Company, and its own encryption key | Safe for development and agents, as CLAUDE.md rule 6 already requires | Not real FEAW data, so it cannot replace real work on its own |
Recommendation: (c) for all development and every agent, and (a) for the owner's real FEAW work. The port closes only once (a) covers what the owner does today:
- DH-6 spike first (before DH-6 is decided, not at item 27). The app only reads
cf-access-jwt-assertion, so route (a) works only if Claude Code completes the Cloudflare Access sign-in tohttps://mcp.getmy.group/mcpas an HTTP MCP server. A time-boxed spike (/sge:spike), read-only against production, checks: that the sign-in completes for each owner; what happens when ADR 0020's short session expires mid-conversation (a clean re-prompt, or silent tool failures); and whether the conversation-startdeadlines_upcomingheads-up shows FEAW's deadlines through that route. If any of these fail, DH-6 is re-decided before DH-7 and DH-8 are built. - DH-7: Xero writes on the web route. Owners write as today; from Claude Code they approve on the web through the preview's link (SPEC-022, built 2026-11-09). Named non-owners, from
XERO_WRITE_PRINCIPALSin Dopplerprd(DH-16, decided 2026-11-09), come later, after the non-owner read access spec. Preview then confirm by ID is unchanged, and no tool files or submits anything (rule 5). - DH-8: A statement tool that takes the CSV content from the client and parses it in pure code, with the entity from the session. It must fit within the approved HTTP request size limit (C4; 1 MiB,
DEFAULT_MAX_BODY_BYTESinsrc/web/app.ts) for a full Amex statement. The spec measures the largest real statement first; if it does not fit, the tool takes the statement in pages, rather than the limit being raised quietly. - Reconnecting Xero or Starling for production (rare) becomes a break-glass runbook. From an administrator's DSM SSH session, run
docker compose -p getmy-books … run --rm --no-deps -p 127.0.0.1:8734:8734 app node dist-tools/scripts/xero-connect.js, with the laptop forwarding 8734 over SSH for the OAuth callback. Written up indocs/runbooks/xero-reconnect-break-glass.md.
If the Product Owner does not want the web route to write, (b) is the fallback. It works, but it keeps production credentials on laptops.
Whatever is chosen:
- Never let a dev database talk to the real FEAW Xero. There must be exactly one duplicate guard.
- Never copy
xero_connectionorbank_account.credentials_encrypted, or a production dump, into dev.
Migration rules (expand, then contract)
Every migration must work with both the current and the previous app. A release may only expand:
- new tables;
- nullable columns, or columns with a constant default;
- new indexes;
- new policies;
- additive data.
Contract changes wait at least one release, until no deployed code (current or previous) reads the old shape. They include:
- drop a table or column;
- rename;
- change a type;
SET NOT NULL;- tighten a constraint;
- delete data, including removing a principal or grant (0005 can only add).
Each carries a
-- contract: #<issue>header and always runs behind the pre-deploy backup.A rename is spread over several releases: add the new one, dual-write, backfill, switch reads, and only then drop the old one.
Every statement is guarded or idempotent, because the runner re-applies every file on every deploy. The idempotent forms without guard syntax are listed in 0.6.
Migration files are append-only. Never edit one that has reached
main.policies.sql,bootstrap.sqlandcreate-role.sqlare not append-only but are re-applied on every deploy, so the contract rules, the lint and the N-1 test cover them too. A changed policy is a numbered migration;policies.sqlre-creates only a policy that is missing or whose definition differs, so a deploy does not take an ACCESS EXCLUSIVE lock on every RLS table.Role passwords are set only by
migrate-cli, as client-computed SCRAM verifiers, never byrunMigrationsand never as a plain literal.A tracking table (
schema_migration(name primary key, applied_at), written in the same transaction) arrives with the first contract migration, which cannot be re-run safely (DH-11).create index concurrentlycannot run in the single transaction. It waits for a tracked, non-transactional migration, when first needed.
Enforced by: the lint (0.6, including the freeze), the N-1 test (2a, which builds the schema the way production does), and review.
Backups and restore
| What | When | Where | Kept | Checked by |
|---|---|---|---|---|
| Nightly dump | 02:30 NAS time, moved from /etc/crontab to DSM Task Scheduler | /volume1/backups/getmy-books/ | 14 | PGDMP plus pg_restore -l; ntfy on failure (new); external heartbeat, alert after 26 hours (new) |
| Pre-deploy dump (new) | Every deploy, after postgres starts and before migrating | …/predeploy/ | 5 | The same checks; a failure refuses the deploy |
| Restore test v2 (now scheduled) | Weekly, Sunday 04:00, DSM Task Scheduler | Throwaway project on an internal network, pinned postgres, last-good app image | Nothing kept | The runbook's own commands; table and pg_policies counts; an RLS query as app; /health; ntfy on failure; heartbeat, alert after 8 days |
Nothing in this plan yet proves that the nightly backups exist today or that a restore has ever worked with grants and policies intact: the current restore test uses --no-owner --no-privileges on an unpinned postgres:17-alpine. P2 records that evidence before W2.
- A failed migration needs no restore. It is one transaction, so nothing changed.
- A failed
/healthafter a good migration needs no restore. The previous image runs on the expanded schema. - A restore from a dump is manual and never automatic. It is a runbook step the Product Owner approves at the time, because it discards writes made after the dump (
ledger_writeaudit rows,vat_treatment). The steps:- stop
app; - create the
approle if the cluster is new (dumps do not hold roles); pg_restore --clean --if-existsinto the running postgres;- start
appand check/health.
- stop
- Dumps are personal and financial data under ADR 0026: the retention above stays inside its proposed 35 days. The off-site copy (Hyper Backup) is still not recorded in the repo, and should be (DH-12).
- The dumps and the key that decrypts them sit on the same NAS. Dumps hold Xero and Starling tokens encrypted with
XERO_TOKEN_ENCRYPTION_KEY. That key is on the 0.1 allowlist, so it is in every release.envunder~deploy/getmy-books/releases/, and indocker inspectof the app, on the same/volume1as the dumps. Hyper Backup may copy both off-site, and its scope is not recorded (DH-12; the P2 check records it). Residual risk, stated plainly: anyone who gets the NAS volume, root on the NAS, or the off-site backup target gets the tokens in usable form. Until the key is rotated with re-encryption (#63) and moved out of the release env (for example a root-owned secret file mounted only intoapp), the dumps are protected only as well as the NAS and the backup target are. If the P2 check showshomes/deployin Hyper Backup's scope, the full-prd.envfiles from before 0.1 are also off-site (see slice 2). - The restore runbook is rehearsed weekly by restore-test v2, which runs its exact commands in a throwaway project and checks
/healthand RLS asapp. Pre-deploy dumps are restore-tested after W2 and whenever the Product Owner asks.
Losing the NAS (disaster recovery)
The registry and the signatures, the app, Postgres and both dump folders all live on the NAS's /volume1. After 1d the NAS can no longer build, so nothing can deploy until the registry is back. The order:
- Get a host with DSM (or plain Linux with Docker) on the same LAN address, or update the runner's
from=and the tunnel. - Restore the registry from the off-site copy (DH-12). This brings back the container images, the signature tags and the
last-gooddigest. - Reinstall the root-owned files from the repo: the scripts, the compose file,
cosign.pub,env-allowlist,config.env,alert.env, the pull credential (a newread:packagetoken), the DSM tasks and thedeployuser (ADR 0023). - Restore Postgres:
up -d postgreson an empty volume, create theapprole,pg_restorethe newest off-site dump (the restore-test commands). - Write
state/last-goodfor the digest to run, verify it with cosign, and start it withgetmy-books-up-last-good. Check/health. - Reconnect the tunnel (its token is in
prd) and check the claude.ai connector.
If the registry cannot be restored, a new registry and a rebuild from a clone are needed, and signatures start again from a new key. This runbook is written into docs/runbooks/nas-disaster-recovery.md with DH-12's details, and rehearsed at least once on a spare host before 1d.
Follow-ups from pre-sale track B (2026-10-07)
Repo changes held back because the NAS runs its root-owned copy of infra/nas/docker-compose.yml. Each needs that copy reinstalled in a change window, together with the matching scripts/deploy-nas.sh and tests/infra/deploy-nas-env.test.ts change:
COMPANIES_HOUSE_API_KEYoptional.src/platform/settings.tstreats it as an optional integration key (the deadline tools say Companies House is unavailable without it), but the compose file requires it (:?) anddeploy-nas.shrefuses to deploy without it. Make it${...:-}and move it tooptional_env.OWNER_EMAILout of the app's environment. Only the local stdio MCP server reads it; the web app never does. Drop it from the compose file and the deploy allowlist once the reinstalled compose no longer requires it.- Pin
cloudflared. Done in the repo 2026-11-03 (#77):cloudflare/cloudflared:2026.10.7@sha256:…. Reaches the NAS when the root-owned compose file is reinstalled.
Acceptance tests
Slice 0
- Given the new
deploy-nas.sh, when a test builds the tar, then its.envholds exactly the allowlisted names plusCOMMIT_SHA, and noDATABASE_URL,NAS_DEPLOY_*orDOPPLER_*. - Given a variable added to the compose file but not to the allowlist, when the tests run, then they fail.
- Given
DATABASE_URLpointing at192.168.0.80, whenpnpm db:setup,db-seed-owneror the database tests start, then they refuse before connecting. - Given two processes refreshing the same expired Xero token, when both save, then exactly one refresh is stored and the other process uses it (
token_versiongoes up by one). - Given a migration containing
alter table … drop columnwith no-- contract:header, whenverifyruns, then it fails. Given an edit to0003-vat-treatment.sql, then it fails. Given a new migration file while.migration-freezeexists, then it fails. Given arevokeadded topolicies.sqlwith no contract marker, then it fails. 5a. Given an empty Postgres with noapprole, whenmigrate-cliruns twice, thenappexists, can log in withAPP_DB_PASSWORD, a query asappis scoped by RLS, the second run changes nothing, and no statement in the Postgres log contains the plain-text password.
Slice 1 (#75) 6. Given a push to main, when build-image runs, then it pushes one linux/amd64 image, and cosign verify with the committed public key and -a commit=<sha> passes in CI. 7. Given an image with a fixed HIGH vulnerability outside the dated exception, when build-image runs, then it fails before pushing. 8. Given the NAS after W1, when main deploys, then main reaches deployed within the budget recorded in 1a, /health shows the commit, docker inspect shows the app image by digest, and no docker build runs on the NAS (none in the deploy log, and no new build cache). 9. Given a release .env with a digest that is unsigned, unknown, or signed for a different commit, when getmy-books-up runs, then it refuses before touching the running app. 10. Given a compose file with build: or an unpinned image:, including on a service that is only in the migrate profile, when compose-check runs, then it refuses. 11. Given a new image that fails /health, when it deploys, then the last-good digest is running again, /health shows the last-good commit, ntfy fires, and the job is red. Three variants must pass: - rotate, then fail: APP_DB_PASSWORD (and, separately, XERO_CLIENT_SECRET) changed in this release; the rolled-back app starts with the new values and logs in to Postgres; - registry down: the registry is unreachable when /health fails; the rollback still completes, offline; - fail twice: a second failing release rolls back to the same last-good, not to the first failed release, and an admin running getmy-books-up by hand afterwards is refused ("release failed"). 11a. Given a release .env that sets COMPOSE_PROJECT_NAME, OWNER_EMAIL, CF_ACCESS_AUD or any name not on the root allowlist, when getmy-books-up runs, then it refuses before any compose command, and docker compose ls shows only getmy-books. 11b. Given the maintenance flag on the NAS, when a deploy arrives, then getmy-books-up refuses before touching anything.
Slice 2 (#74) 12. Given a deploy, when it migrates, then the log shows "backup ok" with a predeploy/ path before the migrate step. Given the postgres container removed but its volume kept, when a deploy runs, then the backup still runs before the migration. 13. Given a migration that fails on the NAS, when it deploys, then the schema is unchanged, the old app still answers /health with the old commit, and the job is red. 14. Given a PR whose migration drops a column still read by main's code, when verify runs, then the N-1 test fails. 15. Given migrate-nas deleted, when a migration merges, then it is applied by the deploy with no connection to port 5433 (the dev token is gone from the runner). 16. Given the weekly restore test, when it runs, then it restores with the runbook's commands into a throwaway project with the pinned postgres and the last-good app image, reports the table and policy counts, finds a query as an unrelated app principal returns no rows, gets /health ok, and sends ntfy if anything is wrong. 16a. Given the nightly backup did not run, when 26 hours pass, then the external heartbeat alerts, with no help from the NAS.
Slice 3 (#73) 17. Given Doppler dev, when doppler run -c dev -- pnpm db:migrate runs, then it migrates localhost and not the NAS. 18. Given a LAN machine, when it runs nc -z -w3 192.168.0.80 5433, then the connection fails. 19. Given W3 is done, when main deploys, then it migrates and /health shows the commit. 20. Given the owner's chosen real-work route, when they run bank_reconciliation_check for FEAW Starling, and (if DH-7 is approved) bank_entries_preview then confirm, then it works on production data, and the local dev server shows only the Demo company. 21. Given the old owner or app password, when anyone tries to log in to the production Postgres, then it fails.
Risks
- Cosign and the registry are unproven together. The registry may not store or serve
.sigtags. The shadow period proves the round trip before the NAS depends on it. If it fails, W1 does not go ahead. - Cloudflare's roughly 100 MB request-body limit could block a large layer. The layer guard fails early. The fallback, a LAN push over plain HTTP to
192.168.0.80:3080, needs its own approval. - The cross-build is correct only while production dependencies are pure JavaScript. The
*.nodeand architecture guards catch a change, and/healthis the backstop. - Open today, not accepted: any branch pusher can run a job on the deploy runner (#71, "A PR can still name
getmy-books-deployin its own copy of the workflow"). Such a job can readDOPPLER_TOKEN_PRD, and with 1a as first drafted it could also have read the signing key and the push token and used a Docker socket that is root on the Colima VM. That would let it sign an image the NAS accepts. P1 closes this before 1a adds any of those; until then the plan adds nothing to that runner. - The signing key, the push token and the deploy key share one trust domain (
prd) unless DH-13 chooses otherwise. Even with P1 closed, signing guards against registry tampering and a stolen push token, but not against a compromised build host. A separate Doppler project for CI secrets is a follow-up (DH-3). - The NAS used to trust every value in the deploy env. That env set the internet-facing app's Access settings and owner identity, and could set Compose control variables. 1b makes the NAS enforce its own allowlist and moves those settings to a root-owned file; until W1 the old behaviour stands.
- A signature proves what was built, not what is newest. The deploy key can still ask for an older, validly signed commit. This is accepted for now.
- The first deploy after W1 has no automatic rollback. It is run by hand, with someone watching, and the fallback is the old
uprun by hand on the NAS. After an automatic rollback,currentstill names the failed release until the next good deploy; the rootfailed/andlast-goodrecords stop that from mattering. A host reboot restarts the containers that were running, notup. - After 1d the NAS cannot build. Rollback is then only to a signed digest held locally or in the registry, and recovering from a lost NAS needs the registry restored first (see "Losing the NAS").
- Deploys now depend on the registry and Cloudflare at pull time. If either is down, the deploy is refused before the running app is touched. The automatic rollback does not depend on them.
- The pipeline gets slower and more fragile.
build-imageadds a buildx build, a scan, a push through Cloudflare and signing to everymainrun, on one capacity-1 runner shared with the health job. The budget is measured in 1a; the Trivy database comes from a mirror so a ghcr rate limit cannot turnmainred. - A new CVE can block every deploy, including an urgent fix. The break-glass route is a dated exception in the fix's PR, approved by the Product Owner; there is no bypass flag.
- Alerting runs on the Mac mini.
alert-main-failedand the health check both run there, so if Colima or the Mac mini is down, nothing alerts. The external heartbeat (DH-14) covers that: the health job pings it on success, so a silent runner shows up as a missed ping. - The package clean-up rule could delete signature tags or the rollback digest. It is tested against the rollback digest before it is enabled.
- Digest pins freeze security fixes. Base images need a monthly bump, owned by the Product Owner through a scheduled issue, and every exception has an expiry that forces the question.
postgresandcloudflaredare scanned when their pin changes and weekly in report mode. - A backup that fails, or a full
/volume1, blocks every deploy. This is intended fail-closed behaviour. Retention of 5 plus disk monitoring limit it. - Lock contention.
lock_timeoutmakes a busy moment fail the deploy safely rather than hang, but it can repeat. - Expand/contract relies on the lint, the N-1 test and review. Old-code paths that no test covers can still slip through, and then the automatic rollback runs old code on a schema it cannot use. That case needs a manual restore.
- Migrations still run as the Postgres superuser, confined to a one-shot container. A dedicated schema-owner role is a follow-up.
- Slice 3 takes away the owner's current local route to real data. It waits until DH-6 to DH-8 are built. Until then, 0.3 protects the shared Xero token row.
- Password rotation opens a short window of refused new connections between
ALTER ROLEand the redeploy. It is done in one sitting, with the redeploy ready. The automatic rollback reuses the current env, so a failed deploy straight after a rotation does not take production down. XERO_TOKEN_ENCRYPTION_KEYwas on laptops, and is on the NAS next to the dumps. It sits in every release.envand indocker inspect, on the same volume as the dumps, and may be in off-site backups. Rotating it needs the re-encrypt tool that #63 describes. Until then, the old key stays valid, and the dumps are only as safe as the NAS and the backup target.- DSM rewrites
/etc/crontab. Scheduled jobs move to Task Scheduler, and the external heartbeat catches a job that silently stops running. - Any DSM update can reset
/usr/local/*, sudoers or the login shell (ADR 0023). The first deploy after an update fails loudly, and the files are reinstalled from the repo.
Changes for the Product Owner to approve
Each item is an exact change outside the repo. Nothing here is done until the Product Owner approves the list for that slice. Checks marked "read-only" change nothing, but are listed so that they are agreed. Numbers are stable: items added in the 2026-09-30 revision carry a letter.
Every NAS window (W1, W2, W3)
- W-1. Repository variable
DEPLOY_PAUSED=truefor the window, back tofalse(or deleted) at the end. Undo: delete the variable. - W-2. NAS:
touch /usr/local/etc/getmy-books/maintenance(root 0600) at the start,rmat the end. (Honoured byupfrom W1's install onwards; for W1 itself, W-1 and watching the Actions page cover the gap.) Undo: remove the file. - W-3. Tell the owner the window's declared effects before it starts.
Before any slice (read-only and owed) 0a. CI runner, read-only: can a runner or label be restricted to push jobs on refs/heads/main or a protected branch (P1, DH-13)? Record the answer, with the docs or config key. 0b. Doppler dev: rotate the git host token, still owed from #71 (any job on the old general runner could read it). The plan keeps that token in dev, so it must be a fresh one. Undo: not applicable (revoke the new one and issue another if needed). 0c. GitHub and Doppler prd, enable the health check (health.yml is off today: the repository variables are empty and prd has no HEALTH_CHECK_URL or HEALTH_CHECK_ACCESS_* names): - create a Cloudflare Access service token for /health only; - add HEALTH_CHECK_URL, HEALTH_CHECK_ACCESS_CLIENT_ID and HEALTH_CHECK_ACCESS_CLIENT_SECRET to prd; - set the repository variable HEALTH_CHECK_ENABLED=true. Undo: set the variable to false, delete the names, revoke the token. 0d. External heartbeat monitor (DH-14): create checks for the nightly backup (period 24 h, grace 2 h), the weekly restore test (period 7 d, grace 1 d) and the health job (period 15 min, grace 30 min, which also catches a dead Mac mini or Colima), each alerting the Product Owner's phone. The ping URLs go into the root alert.env on the NAS and into prd. Undo: delete the checks.
Slice 0: none. Repo only.
Slice 1a (shadow build; only after P1 is closed)
- Registry:
- Create the bot user
getmy-books-ci: not an administrator, web login prohibited. - Create the team
packages-ciin orgGet-My, with the Packages unit set to Write and every other unit set to None, and add the bot to it. - Undo: delete the team and the user.
- Create the bot user
- Registry: create a token
registry-pushongetmy-books-ci, with scopewrite:packageonly. Undo: revoke the token. - Doppler (the config chosen in DH-3 and DH-13, not reachable by any branch job): add
REGISTRY_USER=getmy-books-ciandREGISTRY_PUSH_TOKEN=<item 2>. Undo: delete both names. - The same config:
- The Product Owner generates a cosign v2 key pair on their own machine (
cosign generate-key-pair). - Add
COSIGN_PRIVATE_KEYandCOSIGN_PASSWORD, then delete the local private key file. - The public key is committed to the repo.
- Undo: delete both names and remove
cosign.pubfrom the repo. Images already signed then fail verification, which is safe.
- The Product Owner generates a cosign v2 key pair on their own machine (
- The build-and-sign host chosen in DH-13:
- Read-only, recorded in the PR:
docker version,docker info --format '{{.DriverStatus}}'(classic or containerd image store),docker buildx version, and the Colima VM's CPU, memory and free disk (colima list), against the headroom buildx plus Trivy need. - Read-only: whether job containers there get
/var/run/docker.sock. - Only if they do not, and only on a host that P1 has made unreachable from branch jobs: add the socket mount to that runner's
config.yamland restart it. Never ongetmy-books-deploywhile #71's hole is open. - No QEMU, binfmt, Rosetta, insecure-registry or builder change.
- Undo: restore the previous
config.yamland restart.
- Read-only, recorded in the PR:
- Cloudflare, read-only: confirm that the registry's
/v2/path stays outside Access, and that a test push of a layer of about 90 MB succeeds. No change expected. If the push fails, a separate change list comes back. 6a. Registry, read-only: with the item 2 token,docker login ghcr.io -u getmy-books-cisucceeds (token auth at/v2/tokenfor a user with web login prohibited). 6b. Registry, read-only: a push of a throwaway tag to the org-owned packageget-my/getmy-bookssucceeds with the team's Packages Write, and the tag is then deleted. 6c. NAS, read-only, as root (item 7's token may be created early for this): adocker pullof the throwaway tag throughghcr.iosucceeds from the NAS (the Cloudflare hairpin path), withDOCKER_CONFIGpointing at a temporary root-only folder that is deleted afterwards. Record how the NAS resolvesghcr.io.
Slice 1c (W1, the NAS switch-over; a window) 7. Registry: create a token registry-pull on getmy-books-ci, with scope read:package only. Undo: revoke the token. 8. NAS: copy the current /usr/local/sbin/getmy-books-up, /usr/local/sbin/getmy-books-compose-check and /usr/local/etc/getmy-books/docker-compose.yml into /usr/local/etc/getmy-books/rollback/<date>/ (root, 0700). Undo: delete the folder once it is no longer needed. 9. NAS: install cosign v2.x, static linux-amd64, without running anything from /tmp (noexec on DSM): download to /usr/local/bin/cosign.new, check it against infra/nas/deploy/cosign.sha256, then chmod 0755, chown root:root and mv to /usr/local/bin/cosign. Delete it if the checksum fails. Undo: delete the file. 10. NAS: install infra/nas/deploy/cosign.pub as /usr/local/etc/getmy-books/cosign.pub (root:root 0644). Undo: delete the file. 11. NAS: create /usr/local/etc/getmy-books/docker/ (root:root 0700) containing config.json (root:root 0600), with an auth entry for ghcr.io using getmy-books-ci and the item 7 token. It is never shipped in the deploy tar. Undo: delete the folder and revoke item 7. 11a. NAS: create /usr/local/etc/getmy-books/state/ (root 0700) and install env-allowlist (root 0600) from infra/nas/deploy/, config.env (root:root 0644) holding CF_ACCESS_TEAM_DOMAIN, CF_ACCESS_AUD, OWNER_EMAIL and MCP_ALLOWED_ORIGINS with today's values from prd, and alert.env (root:root 0600) holding ALERT_NTFY_TOPIC and the item 0d ping URLs. Undo: delete the folder and files. 12. NAS: - Read-only dry run as root: docker compose version (root's normal environment); cosign verify and docker pull of the latest shadow digest with DOCKER_CONFIG set on those two commands only; docker compose -p getmy-books --profile migrate config through the new compose-check; and a check that the installed compose accepts up --pull never --wait. - Then install getmy-books-up, getmy-books-up-last-good and getmy-books-compose-check (root:root 0755) and docker-compose.yml (root:root 0644) from the merged 1b commit. - Then run sudo /usr/local/sbin/getmy-books-up on the NAS, and watch it. Declared effect: Compose recreates postgres and cloudflared because their image: strings change: a database restart of a few seconds and a short tunnel drop. - There is no change to getmy-books-receive, the sudoers file, getmy-books-deploy-shell or deploy's authorized_keys. - Undo: copy the three files back from item 8 and run the old getmy-books-up by hand on the NAS (it builds from current; no CI timeout applies). 12a. Rehearsal, after the first good digest deploy: run sudo /usr/local/sbin/getmy-books-up-last-good once and check /health (P3).
Slice 1d (after two clean digest deploys; the point of no return for NAS builds) 13. NAS, read-only: count the NAS_DEPLOY_ lines in ~deploy/getmy-books/releases/*/infra/nas/.env. Expect 0; counts only, no values. Then run docker builder prune -af and remove the old locally built getmy-books-app images (never the last-good digest). Build contexts included infra/nas/.env. Undo: not possible: after this, a NAS build is no longer a fallback. 14. Deploy SSH key rotation: - Read-only first: record both Mac mini LAN source addresses (CI reaches the NAS from the second interface) and the current from= list. - Make a new ed25519 key pair. - Add its line to ~deploy/.ssh/authorized_keys with restrict,from="<both Mac mini addresses>",command="/usr/local/bin/getmy-books-receive". - Set Doppler prd NAS_DEPLOY_SSH_KEY to the new private key. - After one good deploy, remove the old line. - Undo: put the old line back and restore the old Doppler value, which Doppler history keeps. 15. Doppler prd: delete the stale NAS_DEPLOY_DIR and NAS_DEPLOY_COMPOSE (ADR 0023 item 6), and the names that moved to config.env once the 1b follow-up PR stops sending them. Undo: restore them from Doppler history. 16. Registry: - Add a package clean-up rule for container get-my/getmy-books that keeps the newest 30 versions and never matches sha256-* tags. - Before enabling it, dry-run it against the current and last-good digests. - Rob to confirm that it leaves github-runner alone. - Undo: delete the rule. 17a. Cosign key leak runbook (approved now, used only if needed): 1. Open a window (W-1, W-2). 2. Generate a new key pair (item 4), replace COSIGN_PRIVATE_KEY and COSIGN_PASSWORD, commit the new cosign.pub. 3. Reinstall /usr/local/etc/getmy-books/cosign.pub on the NAS. From then on, every image signed with the old key is refused. 4. Deploy main's head through workflow_dispatch so it is signed with the new key. last-good stays usable for rollback because the rollback does not re-verify, but it is replaced by the next good deploy. 5. Review which digests were signed while the key was exposed. Undo: not applicable. 17b. Disaster-recovery rehearsal: run "Losing the NAS" once on a spare host before 1d, and record the time it took.
Slice 2b (W2; a window; only after P2) 17c. NAS, read-only (P2): the evidence listed at the start of slice 2, including Hyper Backup's scope, and one manual run of restore-test v2 against the newest nightly dump. 18. NAS: copy the current getmy-books-up, docker-compose.yml, getmy-books-backup and getmy-books-restore-test into a new dated rollback folder. Create /volume1/backups/getmy-books/predeploy (root, 0700). Undo: remove the folder after copying off any dumps. 19. NAS: reinstall /usr/local/sbin/getmy-books-backup (the new predeploy mode, ntfy on failure, heartbeat on success) and /usr/local/sbin/getmy-books-restore-test (v2: the runbook's commands, pinned postgres, last-good app image, RLS and /health checks, ntfy, heartbeat), both root:root 0755. alert.env is already there from W1. Undo: copy the old scripts back. 20. NAS, DSM Task Scheduler (not /etc/crontab, which DSM regenerates): create a root task "getmy-books backup" daily at 02:30 running /usr/local/sbin/getmy-books-backup, and a root task "getmy-books restore test" weekly on Sunday at 04:00 running /usr/local/sbin/getmy-books-restore-test, each logging to /var/log/getmy-books-*.log. After the first scheduled backup has run and pinged the heartbeat, remove the 02:30 line from /etc/crontab. Undo: delete the tasks and restore the crontab line. 21. NAS: reinstall getmy-books-up v3 and docker-compose.yml v3 (with the migrate service) from the merged 2a commit, then run sudo /usr/local/sbin/getmy-books-up on the NAS and watch it. Undo: copy the v2 files back from item 18 and run up by hand.
Slice 2c (after two clean in-deploy migrations) 22. Runner macmini-getmy-books-deploy: remove DOPPLER_TOKEN (the dev token) from runner.envs and restart the runner. Then revoke that Doppler service token. DOPPLER_TOKEN_PRD stays, until #71 stage 2 removes it. Undo: issue a new read-only dev service token and add it back (only if migrate-nas is restored). 23. Doppler prd: delete DATABASE_URL and APP_DATABASE_URL. The NAS builds both from POSTGRES_PASSWORD and APP_DB_PASSWORD, and nothing else reads them from prd. Undo: restore them from Doppler history.
Slice 3b (the Doppler split; only once the DH-6 route works for the owner) 24. Xero developer portal: create the app GetMy Books (dev), with redirect http://localhost:8734/callback, used only with Xero's UK Demo Company. Undo: delete the app. 25. Doppler dev and dev_personal: - Set: - DATABASE_URL=postgres://getmy_books:dev_only_not_a_real_secret@localhost:55432/getmy_books_dev; - APP_DATABASE_URL=postgres://app:app_dev_password@localhost:55432/getmy_books_dev; - APP_DB_PASSWORD=app_dev_password; - a new XERO_TOKEN_ENCRYPTION_KEY (openssl rand -hex 32); - XERO_CLIENT_ID and XERO_CLIENT_SECRET from item 24. - Delete POSTGRES_PASSWORD and GETMY_BOOKS_TUNNEL_TOKEN. - Delete GETMYGROUP_ACCESS_CLIENT_ID, GETMYGROUP_ACCESS_CLIENT_SECRET and GETMYGROUP_TENANT_ID if nothing local reads them. - Keep OWNER_EMAIL and COMPANIES_HOUSE_API_KEY. - Undo: restore the previous values from Doppler history. The local route then points at production again. 26. Once the non-owner list is built (SPEC-022 S2, after the non-owner read access spec): - add XERO_WRITE_PRINCIPALS=<the email:entityId entries the Product Owner names> to Doppler prd (DH-16, decided 2026-11-09), and to the env allowlist if the release that reads it needs it there; - deploy, and watch /health. Undo: remove the entry from prd and deploy again. Web writes are off again. 27. Cloudflare, read-only: the DH-6 spike's result (it runs before DH-6 is decided, not here): Claude Code completes the Access sign-in to https://mcp.getmy.group/mcp for each owner, and ADR 0020's short session behaves acceptably. Re-checked here for each owner. No change expected. If it fails, a separate change list comes back. Only if DH-6 chooses (b) instead: - create a DSM user dbtunnel with /bin/sh, covered by the deploy-shell restore task; - give each person one authorized_keys line: restrict,port-forwarding,permitopen="127.0.0.1:5433",command="/bin/false" <key>; - add an sshd Match User dbtunnel block, with AllowTcpForwarding local and PermitTTY no, if DSM needs it; - create a Doppler config prd_local holding only the app role URL on 127.0.0.1:15433 and the Xero, Companies House and OWNER_EMAIL references.
*Undo:* delete the user, the block and the config.
Slice 3c (W3; a window) 28. NAS: copy the current compose file and compose-check into a dated rollback folder. Reinstall docker-compose.yml (postgres ports: removed, or 127.0.0.1:5433:5432 under option (b)) and getmy-books-compose-check v3 (refuses ports:, or allows only 127.0.0.1 under (b)). Run sudo /usr/local/sbin/getmy-books-up. Undo: copy both back and run up by hand. 29. NAS Postgres and Doppler prd, in one sitting: 1. docker exec -it getmy-books-postgres-1 psql -U getmy_books, then \password getmy_books at the prompt. The new password is typed at psql's prompt, so it never appears in root's shell history, the process list or the Postgres log (psql sends a SCRAM verifier). 2. Set prd POSTGRES_PASSWORD to the new value. 3. Set prd APP_DB_PASSWORD to a new random value. 4. Deploy main's head (workflow_dispatch), so a fresh release env carries the new values. The migrate step sets the app password as a verifier, and then the app starts with it. If /health fails, the rollback reuses this env, so it logs in.
*Undo:* `\password` back to the value in Doppler history, restore the
Doppler values, and redeploy.
- Xero, production app:
- First confirm in Xero's documentation that existing refresh tokens survive a new client secret.
- Then generate a new client secret, set
prdXERO_CLIENT_SECRET, and deploy in the same sitting. - Undo: not possible (the old secret is gone). Generate another one if needed.
- Read-only checks:
- From a LAN machine,
nc -z -w3 192.168.0.80 5433must fail. - Review who has access to the Doppler
prdconfig, now thatdevno longer holds production secrets.
- From a LAN machine,
Not in this plan, and left to #63: a new production encryption key with re-encryption, and a separate Companies House key.
Decisions
Answers are recorded in the Answer column, dated, with docs/ROADMAP.md's deploy hardening section. An unanswered row is Open.
| Ref | Decision | Recommended | Answer |
|---|---|---|---|
| DH-1 | Approve this plan and its order: slice 0, then #75, then #74, then #73, with one NAS window per issue | Yes | Approved 2026-11-04 (Product Owner): slice 0, then #75, #74, #73, one NAS window each |
| DH-2 | How the digest reaches the NAS | In the release .env, so getmy-books-receive and the sudoers rule stay unchanged. A new receive protocol (deploy <sha> <digest>, with no tar) is a later option | Decided 2026-11-06 (Product Owner): yes, as recommended: the digest goes in the release .env |
| DH-3 | Where the cosign key and registry push token live | Somewhere no branch or pull-request job can read, as DH-13 decides. Not in prd while DOPPLER_TOKEN_PRD sits on a runner that branch workflows can target. A separate Doppler project for build-only secrets, readable only by the DH-13 host | Decided 2026-11-06 (Product Owner): yes, as recommended: a separate Doppler project for build-only secrets, readable only by the DH-13 builder |
| DH-4 | The bundled npm and corepack in the runtime image | A dated Trivy exception now. A follow-up issue for a runtime without npm (distroless: exec-form HEALTHCHECK, /nodejs/bin/node in up) | Decided 2026-11-06 (Product Owner): yes, as recommended: a dated Trivy exception now, and a follow-up issue for a distroless runtime |
| DH-5 | #74 asks for CI tests against a restored copy of production | Deviate: keep production data out of CI. Use the lint plus the N-1 test on synthetic data, and a transactional migration that fails safely. Optionally, rehearse on the NAS against the fresh pre-deploy dump on a throwaway --internal network, if deploy time allows | Decided 2026-11-06 (Product Owner): yes, as recommended: no production data in CI; the N-1 test runs on synthetic data |
| DH-6 | The owner's real FEAW work after #73 | Decided only after the spike (Access sign-in from Claude Code, short sessions, FEAW heads-up). Then (a) the deployed endpoint, with (c) a local dev database for development. (b) the SSH tunnel only if DH-7 is refused or the spike fails. Local server naming is part of this decision | Decided 2026-11-04 (Product Owner): (a) the deployed endpoint for the owners' real work and (c) a local dev database for development and agents; the read-only Access sign-in spike runs first, and a failed spike reopens DH-6 |
| DH-7 | Xero writes on the web route | Yes, for named principals only (XERO_WRITE_PRINCIPALS), preview then confirm unchanged. A product change: the README and a spec are updated first | Decided 2026-11-04 (Product Owner): yes, for named principals via XERO_WRITE_PRINCIPALS; preview then approve unchanged; a product change, so the README and a spec are updated first. 2026-11-09: owners write as today and approve from Claude Code on the web page (SPEC-022, built); the non-owner list waits for the non-owner read access spec, and lives in Doppler prd (DH-16) |
| DH-8 | Amex statement files without a local disk | A statement tool that takes the CSV content from the client, within the approved 1 MiB request limit (C4), paging if a full statement does not fit. Slice 3 waits for it | Decided 2026-11-04 (Product Owner): a statement tool that takes CSV content |
| DH-9 | Separate Xero apps | A new dev app on the Demo Company. Keep the production app and generate a new secret for it, which avoids reconnecting FEAW. This changes #63 step 3 | Decided 2026-11-06 (Product Owner): yes, as recommended: keep the production Xero app and rotate its secret; a new dev app on the Demo Company; stored tokens are re-encrypted with the new key, so owners do not reconnect |
| DH-10 | Who runs the NAS windows | The Product Owner or Rob at the NAS, or the agent on their instruction as in ADR 0023, one window at a time | Decided 2026-11-06 (Product Owner): yes, as recommended: the Product Owner or Rob, or the agent on their instruction, one window at a time |
| DH-11 | Migration tracking table | Add it with the first contract migration, not now | Decided 2026-11-06 (Product Owner): yes, as recommended: with the first contract migration |
| DH-12 | The off-site backup copy | Record the Hyper Backup job (scope, destination, schedule, retention) in the repo, read-only first as part of P2. If homes/deploy is in scope, decide what to do about the full-prd .env versions already retained | Decided 2026-11-06 (Product Owner): yes, as recommended: record the Hyper Backup job in the repo, read-only first as part of P2 |
| DH-13 | How #71's hole is closed before 1a (P1) | Check read-only whether the CI host can restrict a runner to main pushes. If not, #71 stage 2: a pull-based NAS deploy agent and a pull-based builder outside the CI host that no workflow can target | First decided 2026-11-06 as option 2 (#71 stage 2), after a read-only check found the CI host could not restrict a runner by branch or event. Reopened and decided 2026-11-06 (Product Owner): the GitHub-environment route, superseding #71 stage 2. The repo moved to GitHub (ADR 0046); production secrets are secrets of the production environment, deployment branches main only, named only by main-only jobs. Closes P1 once a branch run is shown to be refused the environment |
| DH-14 | External heartbeat for the backup, the restore test and the health job | A hosted dead-man's-switch monitor, off the NAS and off the Mac mini, alerting the Product Owner's phone | Decided 2026-11-06 (Product Owner): yes, as recommended: a hosted dead-man's-switch monitor alerting the Product Owner's phone |
| DH-15 | CI time target | Amend ci.yml's header: pull requests under 6 minutes; main to deployed under 12 minutes, measured in the 1a shadow | Decided 2026-11-06 (Product Owner): yes, as recommended: pull requests under 6 minutes; main to deployed under 12 |
| DH-16 | Non-secret security settings (the Access pair, OWNER_EMAIL, MCP_ALLOWED_ORIGINS, XERO_WRITE_PRINCIPALS) | Move them to a root-owned config.env on the NAS. Changing them becomes a NAS window | Decided 2026-11-09 (Product Owner, issue #73): they live in Doppler prd, not a root-owned config.env. Where this plan still describes a config.env (item 1b and its NAS steps), DH-16 supersedes it |
Related decisions, Product Owner, 2026-11-04:
- #79 (least-privilege env file): narrowed to "allow-listed and enforced on the NAS", built in PR #350. The rest, including whether the env exists once, is folded into slice 1b, which keeps a per-release
.envfor the offline rollback. #79 closes. - #78 (Ansible for the NAS setup): the install manifest and the daily drift check (PR #350) are the answer for one NAS. Ansible is revisited if a second host appears (ADR 0023, note of 2026-11-04). #78 closes.