On Metrics & Comparison
ANNOTATED
Chapter I ended on a broken syllogism. This chapter is about the tool that breaks it: measurement. The printed text explains how to measure. The margins explain how measurement lies — and how to make it stop.
The Illusion of the Like-for-Like
Measurement is presented as the antidote to opinion. What gets measured gets managed; what cannot be measured cannot be improved. The manager who quantifies is held to be more rigorous than the manager who intuits, and rightly so — a number can be checked, argued with, and tracked over time in a way that a feeling cannot.
The danger arises when two numbers that appear comparable are not. Two regions report "conversion rate," but one counts a conversion at checkout and the other at first contact. Two quarters report "active users," but the definition of active changed between them. Two teams report "cost per acquisition," but one includes salaries and the other does not. In each case the arithmetic is flawless and the conclusion is worthless, because the ruler quietly changed between the two things being compared.
When social platforms compete on "monthly active users," the comparison is often theatre: one company counts anyone who opened the app, another counts only those who posted, a third counts a user across three apps as three. Investors have priced billion-dollar valuations off numbers that were never measuring the same thing. Before you envy a rival's metric, find out what they actually put inside it — the definition is where the story hides, not the digit.
Running campaigns for clients across platforms, I hit this weekly: every ad platform defines a "conversion" differently — one counts a click, another a form, another a view-through. Put them in one report unadjusted and you're comparing three different languages. Now the first slide of any client report is definitions, not numbers. — A.P.
Compare the numbers. Compare the definitions first. Then the numbers.
Goodhart's Revenge
A metric begins life as a proxy. It stands in for something we care about but cannot observe directly. "Customer satisfaction" is real but invisible, so we measure survey scores. "Code quality" is real but invisible, so we measure test coverage. The proxy is useful precisely because it correlates with the thing we actually want.
Once a proxy becomes a target, people optimise the proxy directly, and the correlation that made it useful begins to decay. Support agents pressured on call-handling time end calls faster without solving problems. Salespeople measured on activity log more activity without closing more deals. The number improves while the underlying reality it was meant to represent stagnates or declines. This is not dishonesty; it is the predictable response of rational people to what they are actually rewarded for.
The defence is to pair every target with a counter-metric — a second measure that degrades if the first is being gamed. Speed paired with quality. Volume paired with conversion. Growth paired with retention. A single metric optimised in isolation will eventually be satisfied at the expense of everything it was standing in for.
Wells Fargo set aggressive targets on new accounts per customer and rewarded staff against them, with no counter-metric watching whether customers actually wanted those accounts. Employees responded exactly as the incentive demanded: they opened millions of accounts customers never asked for. The number soared; the reality it was meant to represent — customer trust — collapsed into one of the costliest scandals in banking. The proxy became the target, and the target ate the company's reputation.
On install crews I learned to never post a speed target without its partner. The week we celebrated "fastest completions," rework quietly doubled. Now every pace metric travels with a first-pass-quality twin — the crew sees both on the same board, so gaming one shows up instantly in the other. — A.P.
Never ship a target without its counter-metric. A number that cannot be gamed is a number nobody has tried hard enough to game yet.
- Compare definitions before numbers. Flawless arithmetic on mismatched rulers is worthless.
- The comparison problem is the transitive trap. Same disease: the ruler changes mid-argument.
- A target corrupts its proxy. Pay people to touch the finger and they'll stop looking at the moon.
- Every target needs a counter-metric. If a number can only go up, it can only lie to you politely.
The Denominator You Forgot
A result reported without its base rate is a headline without a story. "The campaign generated two hundred leads" says nothing until you know it was shown to two hundred people or to two hundred thousand. "Nine out of ten users prefer the new design" says nothing until you know whether ten users were asked or ten thousand. The numerator is loud; the denominator is quiet; and the denominator is where the truth usually hides.
The related error is survivorship: measuring only the cases that remained visible. The customers who churned are not in your satisfaction survey. The projects that failed are not in your case studies. The candidates you rejected are not in your performance data. Any conclusion drawn only from what survived will systematically overstate how well the surviving strategy works.
In WWII, the military wanted to armour the areas of returning bombers most riddled with bullet holes. Statistician Abraham Wald pointed out the fatal flaw: they were only studying the planes that survived. The holes showed where a bomber could be hit and still fly home — so the armour belonged exactly where the returning planes had no holes, because planes hit there never came back. The lesson outlives the war: your data is made of survivors, and the ones who didn't return hold the information you most need.
The lost quotes taught me more than the won ones. When I started calling customers who didn't book us — the ones missing from every satisfaction survey — the pattern was never price, which is what the surviving data implied. It was response time. You can't learn that from the customers who stayed. — A.P.
Report no rate without its denominator, and trust no dataset that only contains survivors.
- Every rate needs its denominator. "Out of what?" is the fastest lie-detector in business.
- Survivors flatter you. Churned customers and failed projects hold the lesson and are missing from your data.
- A number is a claim, not a fact. Audit ruler, target, and denominator before you let it move you.