The Halo Effect and the Difficulty of Projecting Prospects



Advertisement - Continue Reading Below

I noted in an article recently that scouting is more pseudo-science than science right now and as such a lot of hokum is able to masquerade as useful analysis. This isn’t merely because mankind has always been plagued by capering conman who claim they can predict the future, but also because there are so many factors that determine a kids future success (or failure) in pro hockey that it is incredibly difficult to determine causality. 

Idiosyncratic traits like skill level and personal development mix with environmental factors like luck and opportunity in slightly different ways for every hopeful who gets his name called at the entry draft. Understanding what separates a star from a dud five years down the road means disentangling a spaghetti knot of variables and understanding to the degree to which, if at all, each contributed to the ultimate outcome.

This seems to be one of the reasons that teams apparently value "character" so much in prospects. The Flames in particular have been very public about the high value they place on a kid’s attitude and hockey sense. It seems like a no-brainer – a heavy weight factor that will determine whether the guy will become a player or not. Find youngsters with good heads who understand the game and can be coached. That suggests high adaptability, meaning less headaches for the teammates and more chance he will be able to learn the game at a high level.

Advertisement - Continue Reading Below

Covering prospects – particularly Calgary prospects – has nevertheless made me skeptical of this line of thinking, if only because you’d need both hands to count the number of Flames picks who have been described as "high character" individuals over the last decade or so – a period during which the team has also been one of the worst drafting organizations in the entire league.

I don’t even think it’s safe to say that character is necessary but not sufficient to be a quality NHLer. Mostly because a lot of jerks and broken people have been great hockey players, both in the recent and distant past. In fact, athletic success in general across all sports is pockmarked with superstars who range from the arrogant to the criminal. To put it simply, you don’t really have to be a good guy to be good at sports.

Of course, that isn’t necessarily an indictment of looking for kids with good hockey sense, but these sorts of fuzzy concepts seem to get wrapped up together all the time. There is a cognitive bias called the halo effect which is a person’s judgements of another can be positively skewed by a good "overall" impression, even if the factors in question are independent. So, for example, if you think prospect X is a good guy, you’re more likely to rate his other abilities highly. The devil effect can work the opposite way – having a bad impression of a guy can make you minimize or downgrade other judgements about him. A good first impression and all that…

Beyond the halo effect, the other issue with assessing psychological factors like character and hockey intelligence is accurately measuring them. From what I’ve seen and read, many decision makers seem to go with "I know it when I see it" paradigm of assessment for subjective factors like those in question. Unfortunately, without a stable, accurate method to judge and score this stuff, you’re left at the abominable will of whim and chance. Guys can like and dislike other guys for all sorts of different reasons and all or none of them could be relevant to whether someone can actually play hockey or not.  

A real world example: In his book "Thinking, Fast and Slow", the Noble Prize winning psychologist Daniel Kahneman shared his experience in the Israeli Army. A review of the book sums it up this way:

In 1955, when Daniel Kahneman was twenty-one years old, he was a lieutenant in the Israeli Defense Forces. He was given the job of setting up a new interview system for the entire army. The purpose was to evaluate each freshly drafted recruit and put him or her into the appropriate slot in the war machine. The interviewers were supposed to predict who would do well in the infantry or the artillery or the tank corps or the various other branches of the army.

The old interview system, before Kahneman arrived, was informal. The interviewers chatted with the recruit for fifteen minutes and then came to a decision based on the conversation. The system had failed miserably. When the actual performance of the recruit a few months later was compared with the performance predicted by the interviewers, the correlation between actual and predicted performance was zero.

Kahneman had a bachelor’s degree in psychology and had read a book, Clinical vs. Statistical Prediction: A Theoretical Analysis and a Review of the Evidence by Paul Meehl, published only a year earlier. Meehl was an American psychologist who studied the successes and failures of predictions in many different settings. He found overwhelming evidence for a disturbing conclusion. Predictions based on simple statistical scoring were generally more accurate than predictions based on expert judgment.

Having read the Meehl book, Kahneman knew how to improve the Israeli army interviewing system. His new system did not allow the interviewers the luxury of free-ranging conversations with the recruits. Instead, they were required to ask a standard list of factual questions about the life and work of each recruit. The answers were then converted into numerical scores, and the scores were inserted into formulas measuring the aptitude of the recruit for the various army jobs. When the predictions of the new system were compared to performances several months later, the results showed the new system to be much better than the old.

What is interesting in the book itself is Kahneman’s description of the conventional evaluation process – even when he used it himself he was personally very confident about his grades of each recruit and the chances for his evaluations to predict their future success. He was disappointed to find out his predictions fared poorly, just like others before him. Unlike others before him, however, he set out to formalize the process to some degree and then measure its success against future results in order to gauge it’s usefulness, which eventually resulted in the improved method detailed in the above. The fact that prior evaluators were consistently sure of the obviously faulty interview process gave rise to his concept of the illusion of validity – which, I think, applies to other conventions in the hockey world, but is worth its own post at some point in the future.

Advertisement - Continue Reading Below

None of this is to say the Flames shouldn’t be looking for intelligent, high character kids or anything. The brass just needs to be careful not to let cognitive biases or a lack of understanding of formative causal variables to overly weight the importance of such factors. Like my continued disagreement with the "tough=good" theory of hockey analysis, the point isn’t a binary one – it’s not that factor A is either all good or all bad – it’s that things are often much more complicated than that in real life. Seemingly common sense rules of thumb can lead decision makers down the entirely wrong path because they don’t accurately portray things as they really are. 

There are a lot of tough guys who are lousy hockey players. And there are probably a lot of bright kids who can give a good interview who will be complete busts five years from now as well. If you’re set up to persistently value toughness or character regardless of other relevant variables, your predictions won’t be terribly accurate.

Around the Nation

  • Great article Kent, thanks.

    Is there some sort of.. “devil horn” effect? As in, teams view a player negatively overall, despite all their good traits, because of one bad trait such as being ‘small’ or ‘soft’?

    An example in my head is Theoren Fleury. Late, late draft pick. Nobody thought he’d be any good because he was ‘too small’. Went on to score a boatload of goals. Same with Martin St Louis.

    It’s a shame that things like arrogance scare teams away. Nichushkin being a recent example. Ovechkin was apparently cocky as all hell, but I’m pretty sure there isn’t a team in this league that wouldn’t like to have him. It would be nice to have an aggressive, “out to prove something” player, as opposed to the Stajan “I just want to do whatever they tell me and I’m not very good” type player.

  • Great article Kent, one of the more intriguing pieces I have read in some time. I sure wish that i knew Feaster had this knowledge in hand (not to say he doesn’t)

    So theoretically one would just really rely on the numbers and if you came up with a tie between two equal players you might use something like “Character” or “Tough” to make that final decision?

    • Burnward

      Here’s a dilemma.

      Even if you were aware of this bias you had, correct or not, are you going to intentionally pick the player you don’t want to attempt to right your potential mistake?

      I think all you can do is look at the factors in front of you, weight them out the best you can…and hope you make the right choice.

  • please cancel acct

    Maybe the best place to start the discussion is with a proper definition.

    What is character?

    How do you define character within a person?

    Is the same sort of definition of character that you would use for interviewing for a real world job or dating your daughter the same definition that should be used within the realm of a highly competitive physical environment?

    I am going to say that sports should have a different definition of character and that suggesting that because somebody is a good person and teachable will make them a better player is crazy.

    When a sports team, regardless of the sport, defines character they have to be focusing the definition on the desire of the player to want to win and excel. Watching a player in practices and games should give you a pretty good read on how the player is wired to succeed. That said, the evaluation of a player would need to be done by 3-4 scouts with at least five different viewings. Which brings it back to your statistical variance suggestion that enough data points will create a proper average grouping and that will probably be the most accurate.

    Back to the definition of character. I fully believe that players like Messier, Phaneuf, Gilmour, Cooke, Clutterbuck, Fleury, etc are the exemplary models of what teams should be looking for when they are looking for character in a player. To me, character means a burning desire to win and succeed. The innate wiring in the subconcious that says anything to win, and that the end justifies the means.

    If you have a player that is willing to do anything to win, than you have somebody that will find a way to succeed. Somebody who will make the most of their opportunities. Somebody that will help the team win, because they will do anything not to lose. These players tend to be home team fan favorites, and on the most hated lists of other teams, but they do what they need to do to win.

    You combine this character with hockey sense/IQ and with hockey skills, skating ability, and size and you have a superstar. You take away some of the abilities and you have third line grinders, or fourth line muckers. But you still have players with a desire to succeed and that bring an unmeasureable intangible to the ice because they will create fear or at least awareness in the opposition.

    The question those is where on that line of character to draw the line before you hit psychotics that go too far or that self destruct?

    • It’s a tough question. We don’t even really know what personal characteristics predict better outcomes in the future to tell you the truth. Intuition and experience might guide folks to some common answers, but without proper definition, tracking and correlation to future performance, it’s all guess work.

  • See, this is why I kinda like having Tod Button in the scouting department. He knows how players are being evaluated AND how they were in the past, so he may have critical insights into where things went wrong before.

  • Parallex

    I’ve always kind of believed that intangibles like toughness or character or whatever should be used, essentually, as tie-breakers. Assemble the best options you can based on the tangibles and then use the intangibles to decide amoungst them.

    I dunno, I kind of think that since amateur hockey stats are limited to the standard boxscore numbers that NHL teams almost have to put added value into “scouting only” traits. The numbers that you have are the same numbers that everyone else has (I mean what are you going to do… hire 1000 interns to watch every single OHL, WHL, QMJHL, USHL, KHL, SM-Linnga, Eliteserien game to assemble Corsi/ZS/Qualcomp scores?) so there is no comparative advantage to be had from them.

    I think the problemn lays in teams allowing themselves to overvalue traits whose value is purely subjective to the point that they allow it to become a core trait when it should be tieritory.

  • Jeff Lebowski

    What about the over valuation of hope? In terms of draft picks. Trading known quantity player x for unknown, rather overblown, quantity draft pick.

    The draft picks of themselves have a skewed heightened value because of the chance or potential of amazing pay off. The realization of this hope or potential is actually using the pick to draft a guy who pans out. The above article details how difficult or multi variable that is.

    However many dissociate that complexity from the actual pick. So much so that the value of the pick, driven by the potential it could be, outstrips actual players value, who have beat the large odds and actually made it to regular NHL player.

    It’s as if people see picks as all being future elite, record breaking players. I fall victim of that too.

    We kind of see what we want to see in the benign pick. We see the warts in the player. How many possession sink hole players were once highly touted and highly drafted?

  • please cancel acct

    These kids are drafted for there previous accomplishments. Problem is ,that’s a small window of information for a eighteen year old.

    IMO the success of drafting would increase significantly if they were nineteen and even more if they were 20. More info ,better results.

    Teenager’s are rarely consistent about anything ‘or maybe that was just me and my friends

    • Jeff Lebowski

      I’m firmly against increasing the draft eligible age. Why? Because all it would accomplish would be to render 6th and 7th round picks completely useless. In order for the percentage of successful vs unsuccessful picks to increase, more players would have to leave the NHL permanently every year. So no, raising the draft age would not improve the accuracy of drafting in general. It would simply cause the number of successful picks to gravitate even more directly with draft position.

      • please cancel acct

        I would not support raising the draft age either. I was just trying to make the point that teenagers are hard to predict, but I still believe the accuracy of first round picks would increase with another year of evaluation. More information equals more power

  • Jeff Lebowski

    The issue goes both ways. Many players have been judged solely on hockey ability and become duds. Let’s use Daigle as an example. Based on pure ability, Daigle could still be playing in the NHL and should be a lock for the HOF.

    So all we really know is that it’s a combination of talent and character. I agree with Shutout that the character of a successful athlete differs from that of other professions or areas in life.

    That said, we know that the scouts actually are getting it right to some degree, because of the accuracy of the 1st round. It’s not like 4th or 5th rounders are routinely becoming stars with the majority of 1st rounders never even playing a single game.

    Excellent article.