Wise metrics
Propositions on sensibly making use of metrics when designing things: (for mathematically minded people)
Axioms:
- Those metrics matter, which matter to the people the metrics are supposedly helping serve
- Those metrics matter, which represent the fitness of a "solution" to its context
- Those metrics which represent desired levels of stability & quality are not necessarily the same as the metrics that represent improvements in the solution
- A thing may not be measurable on a scale but still be usefully quantified in terms of presence
Consequences:
- The velocity of a metric is not necessarily representative of the success of the design/development processing shaping the solution
- A metric's utility is not the same across the product's lifecycle
- There is a non-linear relationship between good performance on metrics and good experience, and this relationship cannot be inferred from the metrics alone
- An overlap between the stakeholders' values and the metrics' values is a powerful incentive to human-centeredness; the converse is also true.
- In a complex problem space/environment, a metric constantly runs the risk of becoming irrelevant
- Consequently, metrics must succeed models of the problem, and metrics that precede them are in need of proof-of-relevance
Conclusions:
Though this may seem self-serving coming from a design researcher, there really are too many metrics and not enough models & narratives in the world. This leads to a massive lack of empathy and a predominance of well-engineered solutions to the wrong problems, and eventual stultification of innovation.
In education, for instance, many things that cannot be tested can still be measured (with current technology, even affordably). Unfortunately, the only things measured by policy are those that are testable. The consequences for the kind of people the countries of the world produce today are devastating, and very hard to spot, except by comparison with outliers like Scandinavia.
Metrics can be powerful. If used wisely.