The business literature is filled with many instances in which lunch and skill are lumped together. Learning to sort luck from skill will not only help you make better bets, it will also help you analyze you work and that of your team from a fresh perspective.
In Think Twice, Michael J. Mauboussin says:
“we have difficulty sorting skill and luck in lots of fields, including business and investing. As a result, we make a host of predictable and natural mistakes, such as failing to appreciate the team's and the individual's inevitable reversion to the mean.”
[...]
“the idea is that for many types of systems, an outcome that is not average will be followed by an outcome that has an expected value closer to average.”
Daniel Kanehman “captured the idea that any system that combines skill and luck will revert to the mean over time.” In measuring outcomes, we don't consider that often activities are a combination of luck and skill -- the amount of each depends on the activity.
Mauboussin provides examples for investing, but this is a principle that holds true for business overall. He says that when we ignore the concept of reversion to the mean, we make three types of mistakes:
- we think we're special
- we misinterpret what the data says
- we don't focus our feedback on the part we can control
“Feedback based only on outcomes is nearly useless if it fails to distinguish between skill and luck.”
We compound these mistakes by falling for the halo effect.
“The halo effect is the human proclivity to make specific inferences based on general impressions.”
Performance is relative and in The Halo Effect Phil Rosenzweig showed this mistake is pervasive in the business world. There is a tendency to look at a company's overall performance and to make attributions about its culture, leadership, values, and more. Our thinking is prejudiced by financial performance. So in good times, companies are praised and their success is attributed to a variety of internal factors. While in bad times, companies are criticized and these same factors may be attributed for the failures. The reality is more complicated and dependent upon uncertain and unpredictable factors. Says Mauboussin:
“the media often perpetuates the halo effect. Successful individuals and companies adorn magazine covers, along with glowing stories explaining the secrets to their success. The halo effect also works in reverse, as the press points out the shortcomings in poor-performing companies. The press's tendency to focus on extreme performance is so predictable that it has become a reliable counter-indicator.”
Mauboussin suggests a four-point checklist to avoid the mistakes associated with reversion to the mean:
1. Evaluate the mix of skill and luck in the system you are analyzing
A simple test to tell if an activity involves skill is to ask if you can lose on purpose. Table 8-1 in the book illustrates some examples.
we “should be careful when we draw conclusions about outcomes in activities that involve luck -- especially conclusions about short-term results. We're not very good at deciding how much weight to give to skill and to luck in any given situation. When something good happens, we tend to think that it's because of skill. When something bad happens, we write it off to chance. So forget about the outcome and concentrate instead on the process.”
2. Carefully consider the sample size
“Daniel Kahneman and Amos Tversky established that people extrapolate from small sample sizes.”
“The more that luck contributes to the outcomes you observe, the larger the sample you will need to distinguish between skill and luck.”
[...]
“Jerker Denrell, a professor of organizational behavior at Stanford Business School, has shown the link between the sample size and learning. In his paper, Why Most People Disapprove of Me: Experience Sampling and Impression Formation, Denrell argues that the first impression you have of a person or organization can determine your future degree of interaction.”
3. Watch for change within the system or of the system
“One obvious example is individual changes in skill level.”
While our first inclination is to think in terms of acquiring new skills, we should also consider how we lose some skills over time. Think athletes, for example.
“Further, the system itself may change.”
4. Watch out for the Halo Effect
“A whole cottage industry, including business school professors and consultants, is working hard to offer business people tidy solutions for their problems.
[...]
But any time you see an approach offering secrets, formulas, rules, or attributes to achieve success, you can be sure that someone is selling you a nostrum.”
Tidy stories, while appealing, do not take into consideration specific circumstances and context.
“spotting the halo effect requires discipline”
[image above CC0 Public Domain]