Chapter 30: Measurement and Economics¶
Six Numbers, or an Honest Gap¶
Everything so far has a cost and a claim; this chapter reduces both to six numbers with sources, or to one honest gap.
1. What to Measure¶
The three structures cost money and time; measurement is how you know whether they are working. It is also the chapter most easily ruined, because an agent inflates every volume metric it touches.
The six:
| Metric | The question | The source |
|---|---|---|
| lead time | change opened → change live | the log + the tag |
| rework | how often work is redone | the log's retries and reversals |
| gate escapes | defects found after the gate | incidents + the log |
| cost per service | what it costs to run | the fleet + the spend record |
| drill freshness | when the last drill ran | the drill records |
| onboarding time | a new person or agent to first change | the log |
What never to measure: lines, commits, tickets, "AI adoption %". Every one of them is a number a machine can inflate, and inflating it changes nothing except the number.
🎯 What You'll Learn¶
- The six metrics, each with its source
- Why review load matters more than review speed
- How to publish a gap instead of inventing a number
✅ What to Check¶
- Does each of your six have a named source? A metric without one is a claim.
- Which of your current metrics could be inflated by an agent? Those are the ones to drop.
2. The Bottleneck Is Review¶
The measured bottleneck in the AI era is review, not writing. A large organisation throttled its own AI change generation to protect its reviewers — and approval rates rose as scrutiny fell, which is the failure this chapter exists to catch.
So measure review load and assurance, never speed alone:
| Measure | What it tells you | What it hides |
|---|---|---|
| review load | how much is waiting for a human | nothing — it is the queue |
| review assurance | how much of a diff was actually read | speed, which can look like throughput |
| approval rate | how often a change passes first time | a rate that rises when scrutiny falls |
A faster review that reads less is the failure, and it improves every number on a naive dashboard.
✅ What to Check¶
- Do you measure review load? If not, the queue is invisible.
- Has any review metric improved while the review got shallower? That is the trap.
3. The Economics¶
Cost per service is the number that makes a portfolio decision possible: what this service costs to run, and what the headroom is. A model that ignores headroom is a model that plans a failure.
| Cost | Where it comes from |
|---|---|
| compute and storage | the provider's bill, per service |
| egress and storage moves | the bill, and the architecture |
| the agent's own spend | the log, per task |
| headroom | the capacity you keep for a bad day |
Chargeback and showback matter for a consulting practice: what a sovereign service costs to run is a price you can defend, and a number you cannot produce is a price you cannot quote.
The honest stop. This chapter must publish your numbers, or it is theory. If the numbers do not exist yet, the chapter says so, names the shortlist item that would produce them, and stops there.
The Honest Gap¶
This book has not yet run a measurement window. So the chapter ships with the definitions, the sources and the reader — and one declared gap:
| Metric | Status | What would close it |
|---|---|---|
| lead time | defined, not measured | a period of releases with the log and tags |
| rework | defined, not measured | retries and reversals counted from the log |
| gate escapes | defined, not measured | incidents traced to a gate that passed |
| cost per service | defined, not measured | the fleet report, per service, for a month |
| drill freshness | defined, not measured | the drill records, once drills run on a schedule |
| onboarding time | defined, not measured | a new agent's first change, timed |
| review load | defined, not measured | the review queue, sampled over a week |
A number invented here would be worse than the gap: it would end the book's credibility in the first table a reader checked.
✅ What to Check¶
- Does each metric name its source, or its intention?
- If you cannot measure it yet, is the gap written where a reader can see it?
4. The AI Delta¶
Agents inflate volume, and every volume metric flatters an agent: lines, commits, tickets, "adoption".
The numbers that survive are the ones a machine cannot manufacture — review load and assurance, gate escapes, rework, drill freshness. A number an agent produced without a named source is a claim wearing a decimal point.
💰 Cost Check: an unmeasured estate costs the decisions made on impressions — capacity bought late, rework invisible, review queues discovered by a missed release. The measurement window costs a period of collection and produces six numbers you can defend. The gap costs nothing and is honest; an invented number costs the book.
flowchart LR
L["the log"] --> S["six metrics"]
F["the fleet"] --> S
D["drill records"] --> S
S --> W{"a measurement window"}
W -->|"ran"| N["publish the numbers"]
W -->|"has not run"| G["declare the gap<br/>and the shortlist item"]
What this shows: both endings are acceptable; only an invented number is not.
The Delegation Line¶
The agent assembles the numbers from the log and names each source. The human publishes with the window and the caveat. The system refuses a metric without a source, and a published number without its window.
5. Build the Delivery¶
Deliverables¶
| Artefact | Shape | Where it lives |
|---|---|---|
| metric definitions | six metrics, each with a source | docs/metrics.yaml |
| review-load measure | load and assurance, from the log | the measurement script |
| cost model | per service, with headroom | docs/costs.yaml |
| measurement window | the period, and the caveat | the published report |
Decisions¶
| Choice | Default | What changes if you choose otherwise |
|---|---|---|
| which metrics | the six, no more | a dashboard nobody reads |
| measurement source | the log | a separate tool that can disagree |
| publish or declare | declare the gap when unmeasured | a number that ends the book's credibility |
| review measure | load and assurance | speed, which improves as review weakens |
✅ Gates¶
- A metric without a named source is refused.
- A published number without its window and caveat is refused.
- A volume metric is refused outright.
Runbook¶
- Define the six, each with a source. Checkpoint: no metric without a source.
- Build the reader from the log. Checkpoint: it produces the six from records, not from estimates.
- Run one measurement window. Checkpoint: the period is stated.
- Publish with the caveat — or publish the gap. Checkpoint: an unmeasured metric is labelled, not filled.
- Retire any metric nobody acted on. Checkpoint: the list stays at six.
🎯 Proof¶
- A window of real numbers — or a declared gap with its shortlist item.
- A review-load figure that changed a decision.
- A metric list that stayed at six.
6. 🔑 Chapter Summary¶
Six numbers with sources, or one honest gap — never a number you cannot defend.
- Six metrics: lead time · rework · gate escapes · cost per service · drill freshness · onboarding.
- Never volume: lines, commits, tickets and "adoption" flatter the machine that produces them.
- The bottleneck is review — measure load and assurance, never speed alone.
- The economics include headroom, and a price you cannot produce is a price you cannot quote.
- Publish the gap when you have not measured — this book does, and names what would close it.