I stumbled across a very interesting article by the Chief North American Central Scout Mark Seidel today. Entitled "Why NHL Teams Fail at the Draft" much of what he discusses resonates with my own thoughts on prospect evaluation. It’s also relevant for the Calgary Flames, a team that clearly fails at the draft.
1. Overvaluing performances in high-profile games
Certain tournaments such as the world junior championship, the Memorial Cup and specialty "all-star games" like the CHL Top Prospects Game contribute greatly to the mistakes that NHL teams make — especially when it comes to first- and second-round picks.
The reason is that these high-profile events tend to attract NHL generals managers and presidents who, by watching just a small sample of a player’s performance, can fall in or out of love with a prospect.
For example, I know of one team that had its GM go to the Prospects Game and absolutely fall in love with a player that the whole staff had agreed wasn’t worthy of being a first-round pick. His off-ice character was sketchy, he was a bad teammate, and everybody knew he was a very risky pick. The scouting staff was in its first season with the team, and the more the GM raved about the prospect, the quieter they grew because they didn’t want to disagree with their boss.
Even the head scout started to hedge his bets. Eventually, he started to try and find positive things to say about the prospect and, on draft day, the team found the player still available at their pick in the 20s. They practically ran to the podium to scoop him up. The prospect never panned out, and the team wasted a valuable first-round pick.
His first point is perhaps the most instructive, for two reasons: the first, that sample size matters. It’s something I emphasize in my own numbers-based thoughts on player performance, but it’s just as important on the qualitative side of things. Player performance is highly variable and the bell of the ball one night can be the ugly sister the next. Only over repeated viewings can one get a true sense of a players mean (or true) level of ability.
The second is the description of how decision making can be skewed in rigidly hierarchical organizations like a hockey organization. Seidel’s anecdote sounds suspiciously like groupthink…a type of thought within a deeply cohesive in-group whose members try to minimize conflict and reach consensus without critically testing, analyzing, and evaluating ideas.
Some symptoms of groupthink include rationalizing away conflicting information, excessive optimism and self-censorship by group members. It is especially prevalent in top-down directed organizations in which there is a homogeneity of social background and ideology amongst the members. It is at this point that I should point out that the Flames could probably change their name to the Sutters (and their logo to an image of Darryl’s scowling face) and it wouldn’t be completely unexpected.
Of course, I can’t know that groupthink has influenced the Flames own shoddy drafting under the Sutter regime: hell, the team was terrible at it before he took office. However, with a monolithic "Sutter culture" apparently fully entrenched in both the brains and bowels of the organization, I can safely say that it’s certainly possible.
2. The ‘big fish, small pond’ effect
Another important situation for teams to avoid is falling in love with a prospect who is playing against inferior competition. This can be true for kids playing high school or Tier II junior because a prospect that is just OK can look like Wayne Gretzky against weak opponents.
This is another point I try to make when looking at prospects, especially about kids playing in highschool (they play about 25 games a year) and lesser leagues like the BCHL. It’s especially problematic if the guy is playing on a line with an even more dominating teammate.
Seidel’s other two red flags are "poor character" and "ignoring age effects". The former I can’t speak to because I don’t interview players, but the latter is something easily missed at the junior level and especially in kids playing in college. You’ll notice a lot of highly ranked guys don’t boast impressive totals in the NCAA. Usually that’s because college hockey has a lot of older guys (20 and above), making it much more difficult for teens to compete or even be granted much ice time as freshmen. It’s therefore very important to read a player’s results in context of his age cohort.
Anyways, these are just the broad strokes of the article and I recommend reading it through fully.