Jerry Z. Muller
The Tyranny of Metrics
Princeton University Press
240 pages, 5 1/2 x 8 1/2 inches
ISBN 9781400889433
Jerry Z. Muller
On The Tyranny of Metrics
Interview of
April 16, 2018
In a nutshell
We increasingly live in a culture of metric fixation: the belief of so many organizations that scientific management means replacing judgment based upon experience and talent with standardized measures of performance, and then rewarding or punishing individuals and organizations based upon those measures. The buzzwords of metric fixation are all around us: “metrics,” “accountability,” “assessment,” and “transparency.”
The Tyranny of Metrics treats metric fixation as the organizational equivalent of The Emperor’s New Clothes. Though often characterized as “best practice,” metric fixation is in fact often counterproductive, with costs to individual satisfaction with work, organizational effectiveness, and economic growth. The book helps explain why metric fixation has become so popular, why it is so often counterproductive, and why some people have an interest in promoting it. It is a book that analyzes and critiques a dominant fashion in contemporary organizational culture, with an eye to making life in organizations more satisfying and productive. It’s a book about management broadly construed. But unlike most authors who write about management, it also tries to see organizations from the perspective of the managed.
The wide angle
I came to the subject as an outsider to the field of management: as an historian with a broad interest in the social sciences and in public policy. I was led to the subject by a confluence of my wider interests with events in my personal, professional experience. A few years ago, I found myself for the first time in the position of a manager, as chair of my department. At a certain point, I found my time increasingly devoted to answering queries for more and more statistical information about the activities of the department, which diverted my time from tasks such as research, teaching, and mentoring faculty. Over time, gathering and processing the information, in turn, required the university to hire ever more data specialists. Some of their reports were genuinely useful. But much of the information was of no real use, and indeed, was read by no one.
My experience was irritating, not shattering. But it stimulated me to inquire more deeply into the political and cultural forces leading to all of this emphasis on measurement.
The result is a work of synthesis, drawing on a wide range of studies and analyses from psychology, sociology, economics, political science, philosophy, organizational behavior, history, and other fields. But it doesn’t require prior knowledge of any of these fields. The book makes use of conservative, liberal, Marxist and anarchist authors – some of whom have surprising areas of analytic convergence.
There are chapters devoted to colleges and universities, K-12 education, medicine and health care, business and finance, non-profits and philanthropic organizations, policing, and the military. The goal is not to be definitive about any of these realms, but to explore instances in which metrics of measured performance have been functional or dysfunctional, and then to draw useful generalizations about the use and misuse of metrics. Drawing on a wide range of case studies from education to medicine to the military, the book shows how measured performance can be developed and used in positive ways.
Though focused in the first instance on the United States. There is also a lot of attention to Great Britain, which in many respects was at the leading edge of metric fixation in the government’s treatment of higher education (from the “Teaching Quality Assessment” through the “Research Excellence Framework”), health care (the NHS) and policing, under the rubric of “New Public Management.” From the US and Great Britain, metric fixation – often carried by consultants touting “best practices” – has spread to Continental Europe, the Anglosphere, Asia, and especially China (where the quest for measured performance and university rankings is having a particularly pernicious effect on science and higher education).
A close-up
Based on the real-life experiences of its creators, David Simon and Ed Burns, the HBO series “The Wire,” is regarded by some as among the greatest cultural documents of our age. And with good reason. Focused on a single American city, Baltimore, the series drills down into a few major institutions – the police, the school system, municipal politics, the press – and provides an X-ray-like image of their workings and dysfunctions. The series has attracted an international audience because its themes of organizational dysfunction resonate broadly across Western societies.
One of the recurrent themes of “The Wire” is the salience of metrics: of measured performance as the hallmark of “accountability.” Police commanders are obsessed with hitting the numbers, and they do so by a variety of means that sacrifice effectiveness to statistical targets. Politicians demand numbers that attest to police success in controlling crime. So the police units do their best to avoid having murders attributed to their district: when it turns out that a drug gang has been disposing of bodies in abandoned houses, the homicide sergeant discourages their discovery, since that would diminish the “clearance rate,” the metric of the percentage of crimes solved. Much of the plot revolves around dedicated detectives seeking to develop a complex criminal case against a major drug lord. But since building that case will take months if not years, they are discouraged from doing so by the higher-ups, who want the cops to rack up favorable metrics by arresting lots of low-level drug dealers, despite the fact that those arrested will be replaced almost instantly. The major’s office demands that the rate of major crimes will decline by five percent before the end of the year, a target that can only be reached by overlooking actual crimes or downgrading their seriousness. In each case, they are engaged in “juking the stats” – improving their metrics either by distorting actual results, or by diverting their time and effort from more productive to less productive uses.
Another plot line involves an ex-cop who teaches in a middle school in a neighborhood plagued by poverty, drug abuse, and family fragmentation. Students in the school perform poorly, and the school is in danger of being closed if the test scores of its students do not improve. So, in the six weeks before the standardized tests of English are to be administered, the teachers are instructed by their principal to focus all of class time on practicing for the tests, ignoring other subjects entirely (euphemistically denoted “curriculum alignment”). “Teaching to the test,” like juking the stats, is a way in which institutions are perverted, as effort is diverted from the institution’s true purpose (education) to meeting the metric targets on which its survival has come to depend.
The distortive effects of performance metrics are felt at least as much across the Atlantic, in Great Britain. There, another television series penned by a former real-life practitioner captures the same phenomenon. The series, “Bodies,” written by Jed Mercurio, a former hospital physician, takes place in the obstetrics and gynecology ward of a metropolitan hospital. In the first episode, a newly arrived senior surgeon performs an operation on a patient with complex co-morbidities, after which she dies. His rival then provides him with this advice: “The superior surgeon uses his superior judgment to steer clear of any situation that might test his superior ability.” That is, he avoids difficult cases as a way of maintaining his success rate. A classic strategy of “creaming,” that is, avoiding risky instances that might have a negative impact on one’s measured performance. The cost of this is that patients at greater risk for a failed surgery are left to a certain death without surgery.
“Bodies” is a medical drama, but the phenomena it depicts exist in the real world. Numerous studies have shown that when surgeons, for example, are rated or remunerated according to their success rates, some respond by refusing to operate on patients with more complex or critical conditions. Excluding the more difficult cases – those that involve the likelihood of poorer outcomes – improves the surgeons’ success rates, and hence their metrics, their reputation, and their remuneration. That of course comes at the expense of the excluded patients, who pay with their lives. But those deaths don’t show up in the metrics.
As readers will see, gaming the metrics occurs in every realm: in policing; in primary, secondary and higher education; in medicine; in non-profits; and, of course, in business. And gaming is only one class of problems that inevitably arise when using performance metrics as the basis of reward or sanction. There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring. What gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than their benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge – knowledge that seems solid but is actually deceptive.
Lastly
My purpose is in part to help articulate the vague sense of frustration that so many people have with the organizations in which they work. By pointing out the problems with metric fixation in a systematic way, I hope to make it possible for people to explain to themselves what has gone wrong, and to provide some useful guidelines about when to use measurement and how to do so. My point is not that measurement is bad. It is that measurement requires judgment: about what is worth measuring, about the costs of measuring, about the significance of what is measured.
Recently, I gave a talk about the book at a local Washington bookstore. During the Q&A, a fellow who had recently retired from a career in foreign aid rose and said that the dysfunctions I described were exactly what he had experienced in recent years. After the talk, he bought ten copies of the book, to distribute to his former bosses in foreign aid organizations. That’s the sort of reaction I’m hoping for.
