Another post and another Law, but this time no mathematics is involved.
Imagine you are running a team of salespeople and, as a highly motivated manager, you are working on strategies to improve the performance of your team. After a close study of your team’s client call reports you realise that the high performers in the team consistently meet with their clients more frequently than the poor performers. Eureka! You now have a plan: you set targets for the number of times your team should meet with clients each week. Bonuses will depend upon performance against these targets. Confident that your new client call metric is highly correlated with sales performance, is objective and easily measurable, you sit back and wait.
Six months later, it is time to review the results. Initially you are pleased to discover that a number of your poor performers have achieved very good scores relative to your new targets. Most of the high performers have done well also, although you are a little disappointed that your best salesperson came nowhere near the “stretch target” you set. You then begin to review the sales results and find them very puzzling: despite the high number of client meetings, the results for most of your poor performers are worse than ever. Not only that, your top salesperson has had a record quarter. After you have worked out whether you can wriggle out of the commitment you made to link bonuses to your new metric, you would do well to reflect on the fact that you have fallen victim to Goodhart’s Law.
According to Goodhart’s Law, the very act of targeting a proxy (client meetings) to drive a desired outcome (sales performance) undermines the relationship between the proxy and the target. In the client meeting example, the relationship clearly broke down because your team immediately realised it was straightforward to “game” the metric, recording many meetings without actually doing a better job of selling. Your highest performer was probably too busy doing a good job to waste their clients’ time with unnecessary meetings.
The Law was first described in 1975 by Charles Goodhart in a paper delivered to the Reserve Bank of Australia. It had been observed that there was a close relationship between money supply and interest rates and, on this basis, the Bank of England began to target money supply levels by setting short-term interest rates. Almost immediately, the relationship between interest rates and money supply broke down. While the reason for the breakdown was loosening of controls on bank lending rather than salespeople gaming targets, the label “Goodhart’s Law” caught on.
Along with its close relatives Campbell’s Law and the Lucas Critique, Goodhart’s Law has been used to explain a broad range of phenomena, far removed from its origins in monetary policy. In 18th century Britain, a crude form of poll tax was levied based on the number of windows on every house. The idea was that the number of windows would be correlated with the number of people living in the house. It did not take long for householders to begin bricking up their windows. A more apocryphal example is the tale of the Soviet-era nail factory. Once central planners set targets for the weight of nail output, artful factory managers met their target by making just one nail, but an enormous and very heavy nail.
Much like the Law of Unintended Consequences, of which it is a special case, Goodhart’s Law is one of those phenomena that, once you learn about it, you cannot help seeing it at work everywhere.
Possibly Related Posts (automatically generated):
- Online Data and Charts with Swivel (10 August 2008)
- Rudd, Carbon and the Price of Petrol (19 December 2008)
- Banks, banks, banks (5 November 2010)
- Where does the money go? (31 December 2010)
I have that problem with the fallacy of composition.
I need a clearer example of the fallacy of aggregation though than what I have discovered so far in fallacies.
Or Laws??
When discussing Goodhart’s Law and related phenomena like Moral Hazard we tend to focus on the gaming aspect but equally culpable is our inability to set goals that reflect what we really want to achieve. There is the another Soviet example of a glass factory which initially has its target set in terms of volume and got lots of very thick heavy, but useless, glass so changed the target to area and got lots of very thin, equally useless, glass. The proxies, volume and area in this case, seem to be for a maximum number of panes of acceptable quality glass. If there is a downside to doing this then from an evolutionary psychology point of view it should have been selected against by now.
My own theory is that one of our superior human abilities is to simplify complex problems so as to solve them quickly and effectively ie. find a good enough, if not perfect, solution and move on. So we are now so in love with our own cleverness we actually tend to oversimplify. In ancient times this would have meant more time to solve other problems, or breed. Perhaps this is just another maladapation to modern times, like trying to remember hundreds of people’s names.
… Or as the “back in my day” crowd say, “lost sight of the woods for the trees”. It’s an Interesting question though … How much of an influence on the result is “gaming” (and therefore adding a simple version of the uncertainty principle into the mix, such as in the sales example), and how much is completely misdiagnosing the initial relationship, or confusing causation/correlation?
I think we are looking at things the wrong way.
Let’s consider the meetings/sales case. By construction, what we noticed was: “your new client call metric is highly correlated with sales performance”.
But we know, and I believe you has reminded us of this at times, correlation does not imply causation. In the example this maxim was ignored, with undesirable results.
Let’s think a little. Two variables can be correlated for diverse reasons. One can “cause” the other (but we don’t know which is which). Maybe both variables are “caused” by a third variable (I’m using “cause” in a stochastic sense). Maybe there are variables acting that we are not aware of.
Or maybe it’s just a temporary fluke.
On the basis of the information presented, I suspect we cannot conclude with certainty which of these possibilities is behind this particular case.
But if we cannot say what happened in the simpler case, can we use it to understand the much more complicated money supply/interest rate case?
Just a slight explantion of “in love with our own cleverness”. If confronted with a problem like “improving the quantity of acceptable glass” from a factory you think “I’ll set the target in terms of volume” and if at that point you get a shot of dopamine or serotonin then you will feel like the solution you have come up with is correct. You will relax (ie. lose the anxiety of having to solve a problem), enjoy the experience, and most importantly, not try any further: the problem is solved – there is nothing more to do. Later when confronted with the failure of this solution to the problem you will wonder why you thought it was solved at the time. The answer is you got a little shot of high chemicals that told you it was.
I have no idea how this works from a physiological point of view but I admit I must enjoy this sort of experience several times a day in terms of my day job of solving problems. I definitely get a bit of (neurochemical I am sure) kick out of solving problems. I am also pretty sure this is why I ended up doing the job I do.
Terence Tao has an interesting toy model to illustrate Goodhart’s Law in the context of measuring academic performance in terms of publication count. He concludes
@Apj,
I didn’t read your message before, but I think you are getting the thing.
And don’t forget the Hawthorne Effect http://en.wikipedia.org/wiki/Hawthorne_effect
As always an excellent read. I agree, I feel this law in effect in our business all the time.
Perhaps independent central banks targeting consumer price inflation is a recent example of this?
Pingback: GOODHART’S LAW—ONE ENORMOUS NAIL. | Pater Familias
Pingback: Goodhart’s law: on not going by the numbers « Phil Ebersole's Blog