Deciding What Not to Measure
Every measure costs something to collect and something more to live with. The decision to leave things unmeasured is a real one.
Measurement is treated as free and additive: more data cannot hurt. It can, and deciding what to leave alone is as much a decision as deciding what to count.
The measurement warning in “Deciding What Not to Measure” matters whenever software records work patterns. Organisations researching stealth computer monitoring software can use view the platform for time and project context, while outcomes, quality checks and direct feedback remain necessary to explain what the metric cannot show.
What a measure costs
Collection effort, which is sometimes automatic and sometimes a person's time.
For an independent perspective related to “Deciding What Not to Measure”, consult the NIST Privacy Framework; it offers a useful external check on definitions, governance and the assumptions built into a proposed measure.
Attention: every measure reported is a thing somebody reads instead of something else.
Behaviour change, because measures that matter get managed.
And credibility, if the measure is poor and people can see that it is.
The test before adding one
What decision will this change?
What would we do differently if it moved?
Who will look at it, and how often?
If the honest answers are none, nothing and nobody, do not collect it.
What is usually worth leaving alone
Individual activity in professional work, which the knowledge section covers.
Anything where the measurement effort exceeds the value of the insight.
Things that vary for reasons outside the team's control, which produce explanations rather than actions.
And anything already visible without measuring, which is more than people expect.
The accumulation problem
Measures are added and almost never removed.
Within a few years a team reports a dozen figures, of which two are read.
The other ten still cost collection effort and still shape behaviour, which is the worst combination.
Its own note covers retiring a measure, and the practice is rarer than it should be.
The one-in-one-out rule
Adding a measure requires retiring one.
Crude, and it forces the conversation about which of the existing ones is actually used.
Organisations that adopt it find the first few rounds easy, which tells you something about what was being collected.
Measuring temporarily
Some questions need an answer once rather than a permanent series.
Measure for a quarter, answer the question, stop.
This is normal in operations research and rare in management reporting, where everything becomes permanent by default.
The things that genuinely need watching
Safety, quality, and whatever your organisation exists to deliver.
These earn permanent measurement.
Most other things are investigations rather than instruments, and treating them as such keeps the reporting small enough to be read.
What to check
How many measures does your team report, and how many are read?
When was one last retired?
Which of them would change a decision?
And is anything being collected that nobody looks at?