The Halo Effect and the Difficulty of Projecting Prospects
By Kent Wilson
10 years ago
I noted in an article recently that scouting is more pseudo-science than science right now and as such a lot of hokum is able to masquerade as useful analysis. This isn’t merely because mankind has always been plagued by capering conman who claim they can predict the future, but also because there are so many factors that determine a kids future success (or failure) in pro hockey that it is incredibly difficult to determine causality.
Idiosyncratic traits like skill level and personal development mix with environmental factors like luck and opportunity in slightly different ways for every hopeful who gets his name called at the entry draft. Understanding what separates a star from a dud five years down the road means disentangling a spaghetti knot of variables and understanding to the degree to which, if at all, each contributed to the ultimate outcome.
This seems to be one of the reasons that teams apparently value "character" so much in prospects. The Flames in particular have been very public about the high value they place on a kid’s attitude and hockey sense. It seems like a no-brainer – a heavy weight factor that will determine whether the guy will become a player or not. Find youngsters with good heads who understand the game and can be coached. That suggests high adaptability, meaning less headaches for the teammates and more chance he will be able to learn the game at a high level.
Covering prospects – particularly Calgary prospects – has nevertheless made me skeptical of this line of thinking, if only because you’d need both hands to count the number of Flames picks who have been described as "high character" individuals over the last decade or so – a period during which the team has also been one of the worst drafting organizations in the entire league.
I don’t even think it’s safe to say that character is necessary but not sufficient to be a quality NHLer. Mostly because a lot of jerks and broken people have been great hockey players, both in the recent and distant past. In fact, athletic success in general across all sports is pockmarked with superstars who range from the arrogant to the criminal. To put it simply, you don’t really have to be a good guy to be good at sports.
Of course, that isn’t necessarily an indictment of looking for kids with good hockey sense, but these sorts of fuzzy concepts seem to get wrapped up together all the time. There is a cognitive bias called the halo effect which is a person’s judgements of another can be positively skewed by a good "overall" impression, even if the factors in question are independent. So, for example, if you think prospect X is a good guy, you’re more likely to rate his other abilities highly. The devil effect can work the opposite way – having a bad impression of a guy can make you minimize or downgrade other judgements about him. A good first impression and all that…
Beyond the halo effect, the other issue with assessing psychological factors like character and hockey intelligence is accurately measuring them. From what I’ve seen and read, many decision makers seem to go with "I know it when I see it" paradigm of assessment for subjective factors like those in question. Unfortunately, without a stable, accurate method to judge and score this stuff, you’re left at the abominable will of whim and chance. Guys can like and dislike other guys for all sorts of different reasons and all or none of them could be relevant to whether someone can actually play hockey or not.
A real world example: In his book "Thinking, Fast and Slow", the Noble Prize winning psychologist Daniel Kahneman shared his experience in the Israeli Army. A review of the book sums it up this way:
In 1955, when Daniel Kahneman was twenty-one years old, he was a lieutenant in the Israeli Defense Forces. He was given the job of setting up a new interview system for the entire army. The purpose was to evaluate each freshly drafted recruit and put him or her into the appropriate slot in the war machine. The interviewers were supposed to predict who would do well in the infantry or the artillery or the tank corps or the various other branches of the army.The old interview system, before Kahneman arrived, was informal. The interviewers chatted with the recruit for fifteen minutes and then came to a decision based on the conversation. The system had failed miserably. When the actual performance of the recruit a few months later was compared with the performance predicted by the interviewers, the correlation between actual and predicted performance was zero.Kahneman had a bachelor’s degree in psychology and had read a book, Clinical vs. Statistical Prediction: A Theoretical Analysis and a Review of the Evidence by Paul Meehl, published only a year earlier. Meehl was an American psychologist who studied the successes and failures of predictions in many different settings. He found overwhelming evidence for a disturbing conclusion. Predictions based on simple statistical scoring were generally more accurate than predictions based on expert judgment.…Having read the Meehl book, Kahneman knew how to improve the Israeli army interviewing system. His new system did not allow the interviewers the luxury of free-ranging conversations with the recruits. Instead, they were required to ask a standard list of factual questions about the life and work of each recruit. The answers were then converted into numerical scores, and the scores were inserted into formulas measuring the aptitude of the recruit for the various army jobs. When the predictions of the new system were compared to performances several months later, the results showed the new system to be much better than the old.
What is interesting in the book itself is Kahneman’s description of the conventional evaluation process – even when he used it himself he was personally very confident about his grades of each recruit and the chances for his evaluations to predict their future success. He was disappointed to find out his predictions fared poorly, just like others before him. Unlike others before him, however, he set out to formalize the process to some degree and then measure its success against future results in order to gauge it’s usefulness, which eventually resulted in the improved method detailed in the above. The fact that prior evaluators were consistently sure of the obviously faulty interview process gave rise to his concept of the illusion of validity – which, I think, applies to other conventions in the hockey world, but is worth its own post at some point in the future.
None of this is to say the Flames shouldn’t be looking for intelligent, high character kids or anything. The brass just needs to be careful not to let cognitive biases or a lack of understanding of formative causal variables to overly weight the importance of such factors. Like my continued disagreement with the "tough=good" theory of hockey analysis, the point isn’t a binary one – it’s not that factor A is either all good or all bad – it’s that things are often much more complicated than that in real life. Seemingly common sense rules of thumb can lead decision makers down the entirely wrong path because they don’t accurately portray things as they really are.
There are a lot of tough guys who are lousy hockey players. And there are probably a lot of bright kids who can give a good interview who will be complete busts five years from now as well. If you’re set up to persistently value toughness or character regardless of other relevant variables, your predictions won’t be terribly accurate.
Around the Nation
Recent articles from Kent Wilson
