Appearance
0028: Build the image in CI, sign it, and deploy it by digest
Status note 2026-11-09: not being built. Issue #75 was closed as not planned on 2026-11-08, and the Product Owner accepted that on 2026-11-09. The image is still built on the NAS at each deploy. Nothing below is built. The secret-free build-image dry run in CI (trivy scan, image guards, SBOM) stays as a check. Automatic rollback (#76) no longer has a last good image by digest: see the plan's note of 2026-11-09.
Note 2026-11-06: ADR 0046 moves the git host and CI to GitHub. The registry this ADR names is now
ghcr.io/feaw-serv/getmy-books; the build-and-sign host is any job in the GitHubproductionenvironment (DH-13), and keyless signing with GitHub's OIDC issuer is now an option for slice 1. References below to the previous host are history; the plan's slice 1 is re-read against ADR 0046 before it is built.
Status
Accepted by the Product Owner on 2026-11-06. Decision 0 (DH-13) was reopened the same day after the move to GitHub (ADR 0046): the build-and-sign secrets live in the GitHub production environment, deployable from main only, instead of #71 stage 2.
Proposed, with docs/plans/deploy-hardening.md (slice 1). Nothing here is built, and no infrastructure has changed. Amends ADR 0023 (what the deploy account supplies), ADR 0019 (the deploy flow and its timeout) and ADR 0027 (the built image is scanned and blocks).
2026-11-03 (#75): the secret-free dry run that the plan allows before P1 is built. A build-image job in ci.yml runs scripts/image/build-image.sh on the general runners: the base index check, buildx --platform linux/amd64 --load, the decision 2 guards, trivy.sh built (blocking, with dated exceptions in security/image-exceptions.json) and an image SBOM. It pushes nothing, signs nothing and reads no secret. It runs only when started by hand with the build_image input ticked (a plain dispatch of main is a deploy and skips it) until the owner has seen one green run and switches it to pushes to main. Decisions 0 and 3 to 7 are unchanged and not built.
Context
Issue #75. Each deploy builds the image on the NAS: getmy-books-up runs docker compose up -d --build --wait, with the release folder that the deploy account sent as the build context. On 2026-09-29 that took 14m28s against a 15-minute job limit. There are three problems with this:
- nothing scans the image that actually runs;
- nothing signs it, and it is not reproducible;
- if a deploy fails, there is no ready-made previous image to go back to.
ADR 0023 already queued "build the image in CI and pull it by digest on the NAS".
These facts, checked on 2026-09-30, shape the decision:
- The container registry is ready. The git host's container registry was enabled, and an amd64 buildx image (
get-my/github-runner) was already stored there. The image path would beghcr.io/feaw-serv/getmy-books(ADR 0046), and the organisation is private. - The architectures differ. The deploy runner is an arm64 Mac mini (Colima). The NAS is x86_64.
- A cross-build needs no emulation. The Dockerfile's runtime stage has no
RUNsteps, and every production dependency is pure JavaScript. SoFROM --platform=$BUILDPLATFORMon the build stage produces a linux/amd64 image with no QEMU. - The NAS side takes no input today. The sudoers rule allows
getmy-books-upwith no arguments.getmy-books-receiveaccepts onlydeploy <40-hex commit>and requiresinfra/nas/docker-compose.ymlandinfra/nas/.envin the tar. getmy-books-compose-checkdoes not look atimage:orbuild:at all.- The whole
prdconfig reaches the NAS.scripts/deploy-nas.shwrites everyprdsecret into the release.envon the NAS, and the NAS trusts every value in it, including the Access settings, the owner's email and Compose control variables such asCOMPOSE_PROJECT_NAME. - Any branch pusher can run a job on the deploy runner. #71's 2026-09-30 comment leaves this open: "A PR can still name
getmy-books-deployin its own copy of the workflow." Such a job can readDOPPLER_TOKEN_PRD, and anything added toprdor to that runner. - The git host's API could not re-run jobs, and the UI Re-run returned a 500. A rollback that depends on re-running a pipeline cannot be relied on.
Decision
Prerequisite: close #71's hole first. No signing key, push token or Docker socket goes on a host that a branch or pull-request workflow can target. Either the CI host restricts a runner to
mainpushes (checked read-only first), or #71 stage 2 moves the deploy to a pull-based NAS agent and the build and signing to a pull-based builder outside the CI host. The plan's DH-13 records which; ADR 0046 closes it with a GitHub environment. Until then this ADR's build-and-sign steps do not start.Build once, on
mainonly.- A
build-imagejob (or the DH-13 builder) runs on a host no branch job can reach, and needsverifyandsecurity. - It runs
docker buildx build --platform linux/amd64 --provenance=false --load, from a base pinned by its multi-arch index digest, so the build stage runs natively on the arm64 host. - Pull requests never build, push or sign.
- A
Guard, scan, then push the same image. The job fails on any of these:
- an image architecture other than amd64;
- a
*.nodefile or an arm64 prebuild innode_modules; - a layer of 90 MB or more (Cloudflare's request-body limit);
- a fixed HIGH or CRITICAL finding from Trivy (
--platform linux/amd64, with its database from a daily mirror in the registry, not fetched from its source at job time). Allowances are dated exceptions of at most 30 days, each approved by the Product Owner, insecurity/image-exceptions.json; the first covers the npm and corepack bundled in the Node base, which the app never runs. An urgent fix blocked by a new CVE carries its own dated exception in the same PR; there is no bypass flag.
The job then:
- makes a CycloneDX SBOM of the image;
- pushes the image;
- checks that the pushed manifest's config digest equals the config digest of the image that was scanned, read in the way the host's image store (classic or containerd) requires.
Sign with a cosign key pair, offline.
- The job signs with the private key (
COSIGN_PRIVATE_KEY,COSIGN_PASSWORD) held where only the decision 0 host can read it, using--tlog-upload=false. - The signature carries the annotations
commit=<sha>andref=refs/heads/main. - The public key is committed at
infra/nas/deploy/cosign.puband installed root-owned on the NAS. - Keyless Sigstore is not used. The CI host was not an issuer that Fulcio trusts, and verifying would need the internet at deploy time.
- cosign is pinned to v2.x, which stores signatures as tags.
- The job signs with the private key (
The digest travels in the release env file.
deploy-nas.shwritesAPP_IMAGE_DIGEST=sha256:<64hex>next toCOMMIT_SHA.getmy-books-receive, the sudoers rule andauthorized_keysdo not change.- The env file becomes an allowlist of secrets plus
COMMIT_SHAandAPP_IMAGE_DIGEST. A test keeps the allowlist and the compose file in step. - The NAS enforces the allowlist itself.
getmy-books-uprefuses any name not on a root-ownedenv-allowlist, runs every compose command with-p getmy-books, an explicit--env-fileand a scrubbed environment. - The security settings that are not secret (
CF_ACCESS_TEAM_DOMAIN,CF_ACCESS_AUD,OWNER_EMAIL,MCP_ALLOWED_ORIGINS, laterXERO_WRITE_PRINCIPALS) move into a root-ownedconfig.envon the NAS. Thedeployaccount can no longer change who may sign in.
getmy-books-uppulls, verifies and never builds. It:- refuses during a maintenance window (a root-owned flag file), and refuses a release already recorded as failed;
- checks the env file against the root allowlist;
- validates the digest and exports it;
- runs the compose check, over the config with every profile active, with the expected image reference;
- runs
cosign verify --key cosign.pub --insecure-ignore-tlog=true -a commit=<release commit> -a ref=refs/heads/main; - pulls with
docker pulland a root-owned, read-only registry credential (/usr/local/etc/getmy-books/docker/config.json).DOCKER_CONFIGis set on the cosign and pull commands only, so thedocker composeplugin is still found;uprefuses rather than fall back to standalonedocker-compose; - runs
up -d --no-build --pull never --wait; - checks that
/healthreports the commit, and only then records the commit and digest in a root-ownedstate/last-goodthatgetmy-books-receivecannot write.
Rollback. If
/healthfails, it records the release as failed and starts thelast-gooddigest, with the current release's env and onlyAPP_IMAGE_DIGESTandCOMMIT_SHAswapped, so rotated passwords and secrets still work. It needs no network: the digest was verified when it was deployed and must still be present locally. It alerts through ntfy and still exits non-zero. A root-onlygetmy-books-up-last-good, outside the sudoers rule, does the same by hand.currentandpreviousfromgetmy-books-receiveare no longer used for rollback, so a second failure cannot roll back to a failed release.The compose check refuses, on the output of
configwith every profile active (so amigrateservice behind a profile is checked too):build:andpull_policy: build;- any image not pinned by
@sha256; - an app (or
migrate) image other than the verified one; - an
env_file:other than the release env and the rootconfig.env.
The compose file pins
postgres:17-alpineand a versionedcloudflaredby digest.Registry credentials belong to a bot user,
getmy-books-ci, whose team can only use Packages. It has two tokens:write:package, for CI;read:package, for the NAS root.
No person's token is used.
What changes, and where
| Where | Change |
|---|---|
| Repo | .github/workflows/ci.yml (the build-image job; deploy-nas needs it, keeping its 15-minute timeout until NAS builds stop being the fallback; workflow_dispatch; DEPLOY_PAUSED; alert-main-failed needs every main job); Dockerfile, .dockerignore, package.json (packageManager); scripts/deploy-nas.sh; scripts/security/trivy.sh, security/image-exceptions.json; infra/nas/deploy/getmy-books-up, getmy-books-up-last-good, getmy-books-compose-check, env-allowlist, cosign.pub, cosign.sha256; infra/nas/docker-compose.yml; tests/infra/ |
| Registry | Bot user and team; the push and pull tokens; a package clean-up rule that never deletes sha256-* signature tags or the last-good digest; the Trivy database mirror |
| Doppler | REGISTRY_USER, REGISTRY_PUSH_TOKEN, COSIGN_PRIVATE_KEY, COSIGN_PASSWORD in a config only the decision 0 host can read; delete the stale NAS_DEPLOY_DIR and NAS_DEPLOY_COMPOSE and the names moved to config.env; rotate NAS_DEPLOY_SSH_KEY |
| Runner | #71 stage 2, or a runner restricted to main pushes (decision 0); the Docker socket only on that host |
| NAS | /usr/local/bin/cosign; /usr/local/etc/getmy-books/cosign.pub; /usr/local/etc/getmy-books/docker/config.json; state/ (last-good, failed/, env-allowlist), config.env, alert.env; getmy-books-up, getmy-books-up-last-good, getmy-books-compose-check and the compose file reinstalled; rollback copies of the old files; the build cache pruned; the new deploy key in authorized_keys, with both Mac mini source addresses in from= |
| Cloudflare | Nothing: /v2/ is already outside Access. An Access service token for the health check |
The exact list, with how to undo each item, is in the plan under "Changes for the Product Owner to approve", slice 1.
Order
- Decision 0 is met.
- Shadow: CI builds, scans, pushes and signs for at least three
maindeploys while the NAS still builds. This proves that the registry stores and serves the signatures, that the layers fit through Cloudflare, that the registry auth works from the NAS, and what the pipeline now costs in time. - The repo changes for the NAS merge. They stay inert, because the NAS reads only its root-owned copies.
- One NAS window installs them between deploys, after a root dry run of
docker compose version, verify, pull and the compose check. An admin then runsgetmy-books-upon the NAS; no step depends on re-running a pipeline. - After two clean digest deploys and one rehearsed manual rollback, the NAS build path is removed (the tar shrinks, the build cache is pruned). That is the point after which a NAS build is no longer a fallback.
The first deploy after the window has no automatic rollback, because there is no last-good yet, so someone watches it. If it fails, the old files go back and the old getmy-books-up is run by hand on the NAS, which builds from the full tree still in current, with no CI timeout.
Consequences
- A deploy is a pull of the changed layers and a restart, and the NAS never runs
docker build. The pipeline as a whole gets longer:ci.yml's "under 6 minutes" target is amended to pull requests under 6 minutes andmainto deployed under 12, as measured in the shadow. - The
deployaccount now supplies only a commit, a digest and allowlisted secrets, and the NAS enforces that. An unsigned digest, an unknown digest, or a digest signed for another commit is refused before the running app is touched. - A signature says what was built, not what is newest. The deploy key could still ask for an older signed commit. This is accepted for now.
- Signing is worth something only once decision 0 holds. While any branch pusher can run a job on a host that holds the key, the push token or a Docker socket (root on the Colima VM), anyone who can push a branch can produce an image the NAS accepts. This is not accepted; it is why decision 0 blocks the rest. Even after it, signing protects against registry tampering and a stolen push token, not against a compromised build host.
- Deploys depend on the registry and Cloudflare at pull time. If either is down, deploys are refused, and the running app is unaffected. The automatic rollback does not depend on them.
- A new CVE can block every deploy until a dated exception is approved. That is intended; the exception is the break-glass route.
- After the NAS build path is removed, recovering from a lost NAS needs the registry and the signatures restored first. The plan's "Losing the NAS" runbook sets the order and is rehearsed once.
- A leaked signing key is handled by the plan's key rotation runbook: a new key pair, a new
cosign.pubinstalled on the NAS in a window, and a fresh deploy ofmain. - The cross-build stays correct only while production dependencies are pure JavaScript. The CI guards and
/healthcatch a change. - Digest pins freeze base-image fixes. They need a monthly bump, and the npm exception carries an expiry that forces the question. A runtime without npm (distroless) is a follow-up.
- ADR 0027's "Later" items are done: the built image is scanned and blocks, and the base images are pinned by digest.
Alternatives considered
- Build on the NAS, but cache better. This keeps the NAS building from a context the
deployaccount supplies, and still runs nothing through a scan or a signature. - Build on the secret-free x64 runners and hand the image to the deploy runner. This avoids the cross-build, but pull-request jobs run on those hosts, so the signer would vouch for an image built on a shared machine. It is kept as a possible later split.
- A new receive protocol,
deploy <sha> <digest>, with no tar. This is stronger (no tar parser), but it changes receive and needs the NAS to fetch its own env. It is deferred. - Digest pinning without signatures. This does not meet #75's "unsigned digest refused".