Skip to content

Chapter 17: Observability as the Evidence Layer

Signals Bound to Invariants


1. A Signal Is a Claim About an Invariant

The three structures fail quietly without evidence. A gate proves an invariant at a moment; a signal proves it continuously — and the value of the signal is exactly the invariant behind it.

So the design question is not what should we chart? It is which invariants must be observable, and what reads them?

💰 Cost Check: a dashboard nobody reads costs the build and the maintenance of a second implementation. A missing signal costs the incident discovered by a customer. An agent reading a stale green costs the decision made on an hour-old number. The signal table is the cheap artefact; the parallel dashboard is the expensive one.

🎯 What You'll Learn

  • How to bind every critical invariant to a live signal
  • Why the signal and the gate must share one check logic
  • Freshness as a first-class reading

The Two Failures

Failure What it looks like
a dashboard of everything nobody knows which panel matters
a signal with no invariant a green line that proves nothing

The fix is a table, not a tool: invariant → signal → what reads it → what freshness it needs.

flowchart LR
  I["invariant"] --> C["check logic"]
  C --> G["gate — at a moment"]
  C --> S["signal — continuously"]
  S --> H["human reads it"]
  S --> A["agent reads it"]
  S -.->|"stale"| X["must read STALE, never green"]

What this shows: one check logic feeds both the gate and the signal — the second half of "never a second source of truth".

✅ What to Check

  • Take your five most critical invariants. Is each one a signal you would notice?
  • Does any dashboard panel answer a question nobody asked?

2. One Check Logic, Two Consumers

A gate and a signal should read the same check. The gate runs it at merge; the signal runs it continuously. If they are implemented twice, they will disagree — and the disagreement is discovered at the worst moment, when someone is deciding whether to trust the console or the pipeline.

Consumer Runs Answers
the gate at merge, on every change may this land?
the signal continuously, against live state is it still true?

The rule: the check logic is written once, and both consume it. A dashboard that recomputes an invariant with its own query is a second implementation, and it is wrong the day the first one changes.

✅ What to Check

  • Pick a signal. Is its logic the same code as the gate's, or a copy?
  • If the copy drifted, would anything catch it?

3. SLOs, Budgets, and Two Kinds of Health

A signal without a threshold is a graph. An SLO gives it a line; an error budget gives it a consequence: spend this, stop feature work, fix the reliability.

System health and feature health are different. System health is whether the platform works: latency, errors, saturation. Feature health is whether what you built does what it promised: gate escapes, rework, adoption. A dashboard that mixes them produces a reading nobody can act on.

Health Signals The decision it drives
system availability · latency · saturation reliability work, capacity
feature gate escapes · rework · a signal's freshness design, review, the next iteration

✅ What to Check

  • Does each critical signal have a threshold and an owner?
  • Are system and feature health on the same panel? Separate them.

4. Machine-Consumable Evidence

The agent is a consumer of the evidence layer. An agent that can read the signal can verify its own work, notice a stale reading, and report it. An agent that cannot will infer — and its inference is a claim, not evidence.

That makes structured output a first-class requirement: a signal that exists only as pixels is available to humans alone. The same signal as a string, a file, or a query is available to both.

Freshness is first-class. A stale signal must read as stale, never as healthy. A reading from an hour ago presented as current is the most expensive lie a dashboard can tell, because it is believed.

✅ What to Check

  • Can an agent read your critical signals, or only a human?
  • Does a stale reading say so, in the output rather than in a colour?

5. Agent Observability

If the system is observable, so is the thing operating it. The agent's counterparts to latency and errors are:

Signal What it reveals
cost per task an oversized context or a tiering mistake
rework a gate or routing problem
gate escapes the quality of the gate estate itself
provenance who or what wrote a change

These belong in the same evidence layer because they answer the same question at a different scale: is the system still doing what it promised?

✅ What to Check

  • Can you name the agent's cost, rework and escapes for the last period?
  • Is any of them in the same place as system health, or scattered per tool?

6. Build the Workbench

Artefacts

Artefact What it is for Where it lives
signal table invariant · signal · reader · threshold · freshness docs/signals.tsv
SLO definitions the line, and the budget beside the table
machine output the signal as structured text the collector's output
drill-as-signal a drill's result published as a signal the evidence layer

The signal table, as a shape — one row per invariant, and the reader named:

# docs/signals.tsv — one invariant per row
invariant   signal      reader  threshold   freshness
no secret in repo   scan result gate+agent  0 findings  24h
api reachable       probe       agent           p95<300ms   5m
backup restorable   drill result    agent+human last<30d    30d

Decisions

Choice Default What changes if you choose otherwise
the unit invariant, not metric dashboards of everything
check logic shared with the gate a parallel truth that drifts
output structured, agent-readable a signal only humans can see
freshness printed with the value a stale reading believed as current

✅ Checks

  • Every critical invariant has a signal, a threshold and a freshness window.
  • The signal's logic is the gate's logic, not a copy.
  • A stale reading says STALE in the output.

Sequence

  1. List the invariants that must be observable. Checkpoint: each one has a signal that would fail it.
  2. Bind the check logic — one implementation, two consumers. Checkpoint: the gate and the signal agree.
  3. Set thresholds and budgets. Checkpoint: a breach has an owner and a consequence.
  4. Add freshness to the output. Checkpoint: an hour-old reading cannot be read as current.

🎯 Done When

  • Every critical invariant is a signal an agent can read.
  • The signal and the gate cannot disagree, because they share one logic.
  • A stale reading is impossible to mistake for a healthy one.

7. 🔑 Chapter Summary

Observability is the evidence layer: every critical invariant, one check logic, two consumers.

  1. A signal is a claim about an invariant — a chart without one proves nothing.
  2. The gate and the signal share one check logic; a dashboard that recomputes is a second truth.
  3. SLOs and budgets make a signal actionable; system and feature health are different.
  4. Agents read the signals too — structured output is a requirement, not a nicety.
  5. Freshness is first-class: stale must read as stale, never as healthy.