Skip to content

Chapter 18: Recoverability

The Restore You Have Actually Run


1. What "Backup" Means Now

The three structures assume the system can come back. Recoverability is the difference between a copy and a recovery — and the distinction starts with a question most backup plans never ask: what can we rebuild, and what can we not?

Data Rebuildable? So
generated artefacts, caches, indexes yes rebuild them; do not back them up
configuration, rules, manifests yes, if in the repository the repository is the backup
user data, audit records no back it up, and prove the restore
keys and credentials no custody is the recovery

Backing up what you can rebuild is expensive theatre. Backing up what you cannot rebuild is the whole job.

💰 Cost Check: backing up what you can rebuild costs storage, restore time and a drill nobody finishes. A copy in the same account costs nothing and survives nothing. A key stored beside its data costs the whole backup. The cheap plan is the one that classifies first and stores second.

🎯 What You'll Learn

  • Which data is irreplaceable and which is derived
  • Versioning versus copy-out, and why both are needed
  • Key custody: the part that makes a backup decorative

Two Copies, Two Different Failures

Versioning survives mistakes: a delete or an overwrite can be undone from history. Copy-out survives account loss: the data exists outside the provider entirely.

A versioning bucket is not a backup. A copy in the same account is not a backup. Both fail the same day, in the same way, and the plan that relied on them looks complete right up until it is tested.

flowchart LR
  D["data"] --> V["versioning<br/>survives a mistake"]
  D --> C["copy-out<br/>survives account loss"]
  V --> R["restore drill"]
  C --> R
  R --> P["a proven restore<br/>— or a documented gap"]

What this shows: two mechanisms, two failures, and one drill that proves the pair.

✅ What to Check

  • Which of your data is irreplaceable? Is any of it only versioned?
  • Does any copy live only inside the account it protects?

2. Retention as Policy

Retention is a policy: daily for a week, weekly for a month, monthly for a year. It is enforced by the store's lifecycle rules — declared, versioned, reviewable — and never by a script that deletes.

A deletion script is a second implementation of a policy the platform already has, and it is the one that runs at 3am and removes the wrong prefix.

The Retention Policy, as a shape

# retention — declared, not scripted
daily:   { keep: 7 }
weekly:  { keep: 4 }
monthly: { keep: 12 }
enforced_by: <lifecycle rule id>
never_delete: <the artefacts that must survive every policy>

✅ What to Check

  • Is retention a rule in the store, or a script in the repository?
  • Does anything you must keep forever have an exception written down?

3. Key Custody

If the key that decrypts the backup lives only beside the backup, the backup is decorative. Custody is the recovery — and it is the step most plans omit, because the plan describes copies and not access.

The question The record
where is the key? outside the account, in a named store
who can use it? the roles, and the recovery path
what if that person is unavailable? the break-glass procedure, with its own audit

Break-glass is a control, not a loophole. It is documented, audited, and rehearsed — because the day it is needed is the day the normal path is gone.

✅ What to Check

  • Does the key exist outside the account it protects?
  • Has the break-glass path been used in a drill, and recorded?

The other half of the way back is the release's rollback (chapter 25). A restored copy and a restored deployment are different returns: the deployment costs nothing but a minute, and the copy costs every write since it was taken. Knowing which one an incident needs is the decision chapter 25 makes before the release, not during it.

4. Restore Drills as a Signal

RPO and RTO, sized for a small organisation. RPO is how much data you can lose; RTO is how long you can be down. Both are decisions, and both are cheap to state and expensive to discover.

The drill proves two things: the copy is restorable, and the runbook is followable. A restore that works because one person knows the undocumented steps has proven the person, not the plan.

The result is a signal. Last successful restore: 12 days ago is an observable invariant (Chapter 17's table), and it is the one that turns recoverability from a document into evidence.

The Drill Record

drill:
  at: <timestamp>
  from: <which copy — versioning | off-account>
  target: <where it was restored>
  rto_actual: <how long it took>
  broke: [<what the runbook got wrong>]
  next: <the fix, and the date of the next drill>

✅ What to Check

  • When was the last successful restore? Is it a signal, or a memory?
  • Did the last drill follow the runbook, or did it need someone who knew the steps?

5. What Not to Back Up

The inverse rule matters as much as the rule:

  • do not back up what you can rebuild — indexes, caches, generated artefacts;
  • do not back up what the repository already holds — rules, manifests, config;
  • do not keep a backup whose restore nobody has run — it is a rumour with a bill.

Every artefact you exclude makes the restore faster and the drill cheaper. Every artefact you include without a reason makes the drill longer and the signal weaker.

✅ What to Check

  • Name three things you back up that you could rebuild. Remove them from the plan.
  • Does the restore runbook fit on one page?

6. Build the Workbench

Artefacts

Artefact What it is for Where it lives
retention policy declared, lifecycle-enforced the store, reviewed
copy-out the off-account copy a separate account or store
key custody record where the key is, who can use it, break-glass docs/recovery/keys.md
restore runbook one page, followable the Playbook
drill record + signal the last restore, published as a reading the evidence layer

Decisions

Choice Default What changes if you choose otherwise
what to back up only the irreplaceable the rebuildable, which slows every restore
copies versioning and copy-out one mechanism that fails one way
retention lifecycle rules a script that deletes at 3am
the signal last successful restore, published a memory nobody checks

✅ Checks

  • A restore from the off-account copy succeeds, on a schedule.
  • The runbook is followable by someone who did not write it.
  • The last restore's age is a signal, with a threshold.

Sequence

  1. Classify the data — rebuildable or not. Checkpoint: the backup list contains only the second.
  2. Add copy-out and a lifecycle retention policy. Checkpoint: a copy exists outside the account.
  3. Record key custody and the break-glass path. Checkpoint: the key is outside, and reachable.
  4. Run the restore drill and publish the result as a signal. Checkpoint: the age is observable.

🎯 Done When

  • Every irreplaceable dataset has a copy outside the account.
  • The restore has been run by someone following the runbook.
  • The age of the last restore is a signal with a threshold.

7. 🔑 Chapter Summary

A backup is a copy; recovery is a drill.

  1. Back up only what you cannot rebuild; the rebuildable is theatre.
  2. Versioning and copy-out fail differently — a versioned bucket is not a backup.
  3. Retention is a lifecycle rule, never a deletion script.
  4. Key custody is the recovery; a key beside its data makes the copy decorative.
  5. The drill proves the copy and the runbook, and its age is a signal.