Skip to content

Chapter 30: Measurement and Economics

Six Numbers, or an Honest Gap

Everything so far has a cost and a claim; this chapter reduces both to six numbers with sources, or to one honest gap.


1. What to Measure

The three structures cost money and time; measurement is how you know whether they are working. It is also the chapter most easily ruined, because an agent inflates every volume metric it touches.

The six:

Metric The question The source
lead time change opened → change live the log + the tag
rework how often work is redone the log's retries and reversals
gate escapes defects found after the gate incidents + the log
cost per service what it costs to run the fleet + the spend record
drill freshness when the last drill ran the drill records
onboarding time a new person or agent to first change the log

What never to measure: lines, commits, tickets, "AI adoption %". Every one of them is a number a machine can inflate, and inflating it changes nothing except the number.

🎯 What You'll Learn

  • The six metrics, each with its source
  • Why review load matters more than review speed
  • How to publish a gap instead of inventing a number

✅ What to Check

  • Does each of your six have a named source? A metric without one is a claim.
  • Which of your current metrics could be inflated by an agent? Those are the ones to drop.

2. The Bottleneck Is Review

The measured bottleneck in the AI era is review, not writing. A large organisation throttled its own AI change generation to protect its reviewers — and approval rates rose as scrutiny fell, which is the failure this chapter exists to catch.

So measure review load and assurance, never speed alone:

Measure What it tells you What it hides
review load how much is waiting for a human nothing — it is the queue
review assurance how much of a diff was actually read speed, which can look like throughput
approval rate how often a change passes first time a rate that rises when scrutiny falls

A faster review that reads less is the failure, and it improves every number on a naive dashboard.

✅ What to Check

  • Do you measure review load? If not, the queue is invisible.
  • Has any review metric improved while the review got shallower? That is the trap.

3. The Economics

Cost per service is the number that makes a portfolio decision possible: what this service costs to run, and what the headroom is. A model that ignores headroom is a model that plans a failure.

Cost Where it comes from
compute and storage the provider's bill, per service
egress and storage moves the bill, and the architecture
the agent's own spend the log, per task
headroom the capacity you keep for a bad day

Chargeback and showback matter for a consulting practice: what a sovereign service costs to run is a price you can defend, and a number you cannot produce is a price you cannot quote.

The honest stop. This chapter must publish your numbers, or it is theory. If the numbers do not exist yet, the chapter says so, names the shortlist item that would produce them, and stops there.

The Honest Gap

This book has not yet run a measurement window. So the chapter ships with the definitions, the sources and the reader — and one declared gap:

Metric Status What would close it
lead time defined, not measured a period of releases with the log and tags
rework defined, not measured retries and reversals counted from the log
gate escapes defined, not measured incidents traced to a gate that passed
cost per service defined, not measured the fleet report, per service, for a month
drill freshness defined, not measured the drill records, once drills run on a schedule
onboarding time defined, not measured a new agent's first change, timed
review load defined, not measured the review queue, sampled over a week

A number invented here would be worse than the gap: it would end the book's credibility in the first table a reader checked.

✅ What to Check

  • Does each metric name its source, or its intention?
  • If you cannot measure it yet, is the gap written where a reader can see it?

4. The AI Delta

Agents inflate volume, and every volume metric flatters an agent: lines, commits, tickets, "adoption".

The numbers that survive are the ones a machine cannot manufacture — review load and assurance, gate escapes, rework, drill freshness. A number an agent produced without a named source is a claim wearing a decimal point.

💰 Cost Check: an unmeasured estate costs the decisions made on impressions — capacity bought late, rework invisible, review queues discovered by a missed release. The measurement window costs a period of collection and produces six numbers you can defend. The gap costs nothing and is honest; an invented number costs the book.

flowchart LR
  L["the log"] --> S["six metrics"]
  F["the fleet"] --> S
  D["drill records"] --> S
  S --> W{"a measurement window"}
  W -->|"ran"| N["publish the numbers"]
  W -->|"has not run"| G["declare the gap<br/>and the shortlist item"]

What this shows: both endings are acceptable; only an invented number is not.

The Delegation Line

The agent assembles the numbers from the log and names each source. The human publishes with the window and the caveat. The system refuses a metric without a source, and a published number without its window.


5. Build the Delivery

Deliverables

Artefact Shape Where it lives
metric definitions six metrics, each with a source docs/metrics.yaml
review-load measure load and assurance, from the log the measurement script
cost model per service, with headroom docs/costs.yaml
measurement window the period, and the caveat the published report

Decisions

Choice Default What changes if you choose otherwise
which metrics the six, no more a dashboard nobody reads
measurement source the log a separate tool that can disagree
publish or declare declare the gap when unmeasured a number that ends the book's credibility
review measure load and assurance speed, which improves as review weakens

✅ Gates

  • A metric without a named source is refused.
  • A published number without its window and caveat is refused.
  • A volume metric is refused outright.

Runbook

  1. Define the six, each with a source. Checkpoint: no metric without a source.
  2. Build the reader from the log. Checkpoint: it produces the six from records, not from estimates.
  3. Run one measurement window. Checkpoint: the period is stated.
  4. Publish with the caveat — or publish the gap. Checkpoint: an unmeasured metric is labelled, not filled.
  5. Retire any metric nobody acted on. Checkpoint: the list stays at six.

🎯 Proof

  • A window of real numbers — or a declared gap with its shortlist item.
  • A review-load figure that changed a decision.
  • A metric list that stayed at six.

6. 🔑 Chapter Summary

Six numbers with sources, or one honest gap — never a number you cannot defend.

  1. Six metrics: lead time · rework · gate escapes · cost per service · drill freshness · onboarding.
  2. Never volume: lines, commits, tickets and "adoption" flatter the machine that produces them.
  3. The bottleneck is review — measure load and assurance, never speed alone.
  4. The economics include headroom, and a price you cannot produce is a price you cannot quote.
  5. Publish the gap when you have not measured — this book does, and names what would close it.