0075 — The autodeploy follows a deployment until the platform finished it
Status: Accepted
Edited 2026-11-15 to remove deployment details.
Date: 2026-11-15
Supersedes in part: ADR-0060 (what the script
records after a deploy, and where rollback goes: an image counts as having run once the container
platform finished deploying it, no longer once it accepted the request; and the polling of the platform that
ADR-0060 left out is now done. The rest of ADR-0060 stands)
Context and Problem Statement
Section titled “Context and Problem Statement”The timer once deployed a commit that the platform accepted, then failed at the image pull: containerd on the host could not commit a layer (“failed commit on ref … ingest”), a passing fault, as a pull by hand a minute later worked. The script had already recorded the commit as deployed and the head as dealt with, so no tick tried again. Production ran the image before it until an operator deployed by hand.
ADR-0060 knew this: “the timer still reports that a deploy was triggered, not that it is healthy … Confirming a deployment would mean polling the platform, which this decision leaves out”. The cost of leaving it out was a release that silently did not go out.
Decision Drivers
Section titled “Decision Drivers”- A deploy that failed for a passing reason goes out without anyone acting.
- One that keeps failing stops, and says so where an operator looks: the unit fails, and the watchdog, which compares the image production runs with the one the script triggered last, keeps alarming.
rollbackgoes to an image that ran, not to one that was only triggered.- The lock, the commands by hand and the timer’s mark of
masterstay as they are.
Considered Options
Section titled “Considered Options”- The next ticks read the deployment the platform named (chosen). The deploy call already answers
with its uuid;
GET /api/v1/deployments/<uuid>givesqueued,in_progress,finished,failedorcancelled-by-user. - Wait in the same run until it ends. Simpler state, but a tick would hold the lock for minutes, and a command by hand waits at most two of them.
- Compare the image the container runs with the one triggered (
docker ps). That is what the watchdog does. The script runs withDynamicUserand no access to Docker, and the platform’s own answer says why a deploy failed.
Decision Outcome
Section titled “Decision Outcome”After a deploy, by the timer or by hand, the script keeps the deploy it follows in its state
directory, in pending: the commit, the deployment the platform named, the attempt, and when the first
attempt was made. The platform names a deployment whenever it queued one, also one already queued for
the same image; an answer that names none means nothing was queued, and the script counts it as
not triggered. An attempt is written to pending before it is made, so the attempts stay bounded
when the state directory cannot be written.
Every tick reads that deployment first, and while one is followed it deploys nothing else:
finished: the image is recorded as running (running-sha), and the one that ran before it as the one before (previous-sha). The tick then goes on as usual and may deploy a newer head.failed: it is deployed again, with the same commit, at the next tick, up toDEPLOY_ATTEMPTS(3) attempts in all. An attempt the platform refused counts too. After the last one the script logs anERRORthat names what runs and the command to put the commit out, stops following it, and the tick exits 1. It does not deploy that commit again; the next commit onmasterdeploys as usual.cancelled-by-user: someone meant it. It is not deployed again.queued,in_progress, or no answer: the tick waits. One followed for longer thanDEPLOY_WAIT_SECS(45 minutes) is given up with anERROR, as above.
deployed-sha keeps its meaning: what the script triggered last, or was told runs. The watchdog
reads it and needs no change: after a give-up it still differs from the image that runs.
A command by hand first reads the deploy followed so far: one that finished is recorded, so that
what ran before it is known; one that failed or was cancelled is replaced; one still queued or in
progress is not replaced, because the platform would still run it after the command’s own, and the
command deploys nothing and says why. rollback deploys the image that ran before the one that
runs now, and refuses while the script does not know what runs. running <sha> records what
runs, ends any following, and on a state from an older script drops its previous-sha, which
names an image triggered, not one known to have run.
Consequences
Section titled “Consequences”- Good: a deploy that failed for a passing reason goes out at the next tick, without an operator.
- Good:
rollbacknever goes to an image that only failed. - Good: a deploy that keeps failing ends in a failed unit and an
ERRORline, and the watchdog’s alarm stays. - Bad: a newer head waits while a deploy is followed: up to three attempts a tick apart, or 45 minutes at most when the platform holds one queued.
- Bad: “finished” is the platform’s word. On this application the platform’s health check is off and the image has none, so it means the container started, not that it answers.
- Bad: a deploy or rollback by hand waits for a deploy under way to end, or for the operator to cancel it in the platform.
- Neutral: two more state files,
running-shaandpending. An older script’s state has norunning-sha;running <sha>once after the upgrade givesrollbackits starting point.