Chapter 25: Deploy, Verify, Roll Back¶
The Way Back Is Part of the Path¶
A release is a decision and the deploy is only its mechanism — and the way back belongs to the same path, which is what this chapter designs.
1. The Deploy Is Not the Release¶
The three structures ship through a deploy, and a deploy is the moment the system's truth is tested: the artefact runs, or it does not. Verification belongs here, not at the tag — the tag says what was intended, the deploy proves what happened.
The previous chapter decided whether to release. This one decides how the change reaches production, and how it comes back.
🎯 What You'll Learn¶
- Why progressive delivery is chosen by blast radius, not fashion
- The expand/contract window that lets two versions run
- Why the rollback decision must be written before the release
Progressive Delivery, by Blast Radius¶
| Mechanism | Use when | The cost |
|---|---|---|
| canary | traffic is observable and divisible | needs a signal that can judge it |
| blue/green | the state can be swapped atomically | needs double the capacity, briefly |
| feature flag | the behaviour can be separated from the code | needs a flag you will actually remove |
Chosen by blast radius, not fashion. A canary with no signal is a delay; a blue/green with unpriced capacity is a surprise; a flag nobody removes is a second codebase.
The Expand/Contract Window¶
A live deploy runs old and new code together. The migration pattern from the contracts chapter is what makes that window safe:
| Phase | The change | What is true |
|---|---|---|
| expand | add the new shape; write both | old code works, new code works |
| migrate | backfill | both agree |
| contract | remove the old shape | only the new code |
Each phase is a deploy with its own verification. The window is not an inconvenience to be shortened; it is the design that lets the release order stop mattering.
✅ What to Check¶
- Does your canary have a signal that could judge it? If not, it is a pause.
- Does every schema change fit a phase, or does one step both add and remove?
2. Verify in Production¶
A deploy is done when the running system proves it. Two properties make the proof real:
- It reads the running system — not the artefact, not the pipeline's opinion.
- It has a freshness rule — the signal it reads must be recent, or the check is a reading of history.
Verification, as a shape:
# post-deploy verification
checks:
- id: <check>
command: <what runs, against production>
expects: <the response or the signal>
freshness: <how old a reading may be>
on_failure: fail the deploy loudly — never "probably fine"
"Probably fine" is the failure this chapter exists to prevent. A deploy that reports success because the pipeline finished is a deploy nobody verified.
✅ What to Check¶
- Does your post-deploy check read production, or the build output?
- If the deployment succeeded but the feature is dead, would anything say so?
The data half of the way back is chapter 18: restoring a copy replaces every write since it was taken, which is why the rollback this chapter designs is built to need it as rarely as possible.
3. Rollback, Roll-Forward, and the Case With Neither¶
| Exit | When | The cost |
|---|---|---|
| roll back | the new version is the problem | a second deploy, and any state it wrote |
| roll forward | a fix is faster than a revert | a second change under pressure |
| neither | the data migration cannot be reversed | the decision was made before the release, or it is made in an incident |
The third case is the one that matters. A data migration that cannot be reversed means the rehearsal happens before the release — in a preview environment, against a copy — because after it, there is no rehearsal, only consequences.
The rollback decision is part of the evidence pack, written before the release. An agent that improvises a rollback mid-incident is the failure mode; a decision that already exists is the control.
And it is a record a gate reads — the three layers of chapter 24's rollback table, in the release's own file, checked by make release-gate before the tag exists. A rollback decision that lives only in a chapter is advice; one a gate refuses a tag without is a practice.
Preview Environments¶
The cheapest way to make a release boring: one environment per change, torn down on merge.
- it tests the real deploy, not a simulation;
- it gives the verification a place to run before production;
- it makes the rehearsal of the irreversible case possible.
✅ What to Check¶
- Has the rollback been run, at least once, outside an incident?
- Which change in your history could not have been rolled back? Was it rehearsed?
A dump taken before the change is a floor, not the way back: restoring it discards every write since, including the ones the incident itself recorded. That is why the way back is designed into the migration (chapter 15) instead of recovered after it — and why an irreversible change is a named class with three facts, not a backup taken in a hurry.
4. The AI Delta¶
An agent rolls forward faster than a human can assess, so the rollback decision must be written before the release; an agent improvising a rollback mid-incident is the failure this chapter exists to prevent.
And an agent's "verified" must be a machine check with a freshness rule, not a narration of what it believes it saw.
💰 Cost Check: a rollback nobody has run costs the incident, plus the discovery that a step was undocumented. A canary without a signal costs a delay that proves nothing. A preview environment costs per change and saves every release. The way back is cheaper when it is designed than when it is improvised.
flowchart LR
T["tag"] --> D["deploy"]
D --> V{"verify in production<br/>freshness checked"}
V -->|"passes"| L["live"]
V -->|"fails"| R{"rollback decision<br/>written before"}
R -->|"reversible"| B["roll back"]
R -->|"not reversible"| P["the rehearsal should have happened"]
What this shows: verification and the way back are part of the same path; the last box is the one that costs.
The Delegation Line¶
The agent executes the sequence and the verification. The human decides rollback versus roll-forward. The system refuses a deploy whose post-deploy verification did not run or whose signal is stale.
5. Build the Delivery¶
Deliverables¶
| Artefact | Shape | Where it lives |
|---|---|---|
| deploy sequence | tag → staging → production, each with verification | the pipeline |
| post-deploy verification | checks with a freshness rule | the pipeline |
| rollback runbook | ordered steps, including the data case | the Playbook |
| preview environment | per change, torn down on merge | the pipeline |
Decisions¶
| Choice | Default | What changes if you choose otherwise |
|---|---|---|
| delivery mechanism | canary where traffic is observable, blue/green where not | a delay that proves nothing |
| verification | a production read, with freshness | a pipeline-completion claim |
| the way back | roll back, with the impossible cases written down | an improvised decision |
| preview | per change | a rehearsal that only happens in production |
✅ Gates¶
- A deploy without a passed post-deploy verification is reported as failed.
- A rollback runbook older than its freshness window is a finding.
- A deploy from a commit that is not the tag is refused.
Runbook¶
- Write the deploy sequence. Checkpoint: each step has its verification.
- Add post-deploy verification with a freshness rule. Checkpoint: a stale reading fails it.
- Rehearse the rollback, including the data case. Checkpoint: the runbook is followable.
- Add preview environments. Checkpoint: one change, one environment, torn down on merge.
- Run one canary against a signal that can judge it. Checkpoint: its verdict is a number.
🎯 Proof¶
- A rollback executed in drill, and followed from the runbook.
- A deploy that fails its verification and says so.
- A preview environment that dies on merge.
6. 🔑 Chapter Summary¶
The deploy proves the change; the way back is designed before it is needed.
- The deploy is not the release — verification belongs to the deploy, because it reads the running system.
- Progressive delivery by blast radius, never by fashion — and a canary needs a signal.
- Expand/contract is the window that lets two versions run, one phase per deploy.
- Rollback, roll-forward, and neither — the third must be rehearsed before the release.
- The agent executes; the human decides the way back; the system refuses a stale verification.