Evaluating Player Evaluation: Scouting


hockey scouts
– pic via brooke bina


The most prominent method in evaluating players – both established and emerging – is by scouting them in-person. The method has its particular strengths and weaknesses, as you would expect. Over the past two seasons I’ve been contributing to the NHL Draft coverage over at The Hockey Writers and I’ve developed some observations based on my limited scouting experiences.

First and foremost, trying to keep track of more than maybe two or three players in a game is tricky. In my first couple outings to Hitmen games to see opposition players, I figured I could get notes on everybody. Generally-speaking, the more players you try to follow, the less you will actually see. Consequently, the opposite is also somewhat true. I developed an appreciation for Morgan Klimchuk’s two-way play during a fairly one-sided Calgary/Regina tilt where I only paid attention to Klimchuk. As a result, I tend to only “target” one or two players per game.

Secondly, if you want to figure out how good a player really is, you need to see him over and over again and in different situations. The draft-eligible player I saw the most of this season was Calgary Hitmen forward Greg Chase. I saw him on the power-play, short-handed, at even strength, in games that Calgary won handily and in games where the Hitmen just played bad. On the other hand, I saw Eric Roy and Morgan Klimchuk much less often. Thus, I feel less comfortable making broad statements about them as I do about Chase – who was a steal for the Oilers in the seventh round.

Chase’s situation was fairly unique, in that he played top-six minutes for most of the year, but was behind a few older players on the Hitmen depth chart. As such, he didn’t get sheltered as much as other players were, but he got the high ground more often than, say, Jake Virtanen did. This is all a round-about way of noting that a lot of times, you need to see a player in person to know what they can do, and it’s only really useful if you see them in a lot of different circumstances and over the course of a season, especially if a team’s circumstances change. For instance, the Hitmen changed how they deployed players when Jake Virtanen was at the U-17s last year, as well as when they had a slew of injuries during the stretch drive.

Does scouting tell you something? Definitely. In fact, it’s the most data rich scouting tool available.

But compared to statistics, scouting takes a long time to get right, particularly given how much circumstances can skew things and it can be complicated by certain subjectivitiesand  biases. If anything, that’s the beauty of having the two approaches – they can complement each other rather nicely, if used and framed properly. The folly is claiming that either approach is superior, as they can’t really be compared because they tend to tell you different things about different parts of a player’s game. Stats can tell me how good Max Reinhart’s NHLE was throughout last year, but scouting can tell me that he became a huge part of the Abbotsford attack after the lockout took Sven Baertschi, T.J. Brodie and Roman Horak out of the Heat line-up, or provide some insight into his on-ice tendencies in different game situations. Both are valuable tools for assessing, evaluating and projecting players.

Evaluating Player Evaluation Series

  • RexLibris


    This is why I have been an advocate of a bridge between the pure analytics and the “seen him good” perspectives.

    Sway too far one way or the other and you can lose perspective. A good scout finds a way to internalize both and make the right call, or at least a better call than his peers.

    • Parallex

      I don’t.

      Sure it sounds like a great job (Get paid to watch sports!) but I imagine it means a lot of nights on the road travelling from rink to rink sleeping in crappy beds and eating greasy food while wishing you were home to tuck your kids into bed, writing a lot of reports (yay paperwork), and you won’t even get to watch actual hockey because since you’ll be watching only a few guys you’ll miss the rest of the game.

      Last thing I would want to do is turn watching hockey from a entertaining diversion into a chore.

      • McRib

        Professional scouts are mostly all former players. Not to mention amateur scouts make much less than you would expect!!! Unless of course you are a former star NHLer like Steve Yzerman that can jump into a GM role immediately after retiring. Its a lot of paying your dues where most people start out working for next to nothing at a Private Publication scouting the Junior Ranks then…

        Once you make it most entry level amateur scouts only start off as Part-Time (expect to give up any free time you planned) working for $15,000-20,000 (Not sure if you have looked up the airfare to Russian lately but this money doesn’t get much further than expenses). When if your good you can expect a whopping full-time salary of $60,000-70,000 for years… Until you are made a regional scout/head scout where you might get $80,000-100,000.

        Depending on your career decisions the private sector is much more profitable career in the short term and long term, unless your one of the very few to make it to Asst. GM or GM levels. Personally know two people who are amateur scouts both married Lawyers and are stay-at-home Dads when they aren’t on the road 24-7. Scouting is not nearly as glamorous as one would think. For me it sounds a lot like the struggles of trying to be a journalist or writer.

      • Demetric

        You have a point on the travel, I have been there and done that for my current position, it was as bad as 1 week out 1 week in on average for a couple years. It does get old, let me tell you and more so when there is family (no kids yet, but soon)

        Also, fun into work but whats that saying, if you love what you do its not actually work.

        though, I would have to agree, sounds great on the surface but having to focus on a few players at a time i think would take away the enjoyment, but never tried, so who knows.

  • Arik

    I think the big thing this article overlooks is how very very bad humans are at managing large amounts of information and accounting for cognitive biases at the same time.

    When we rely on memories, what we frequently perceieve as a players quality is basically a collection of highlights and lowlights. The meat of a player is going to be neither. Sure, it’ll be good and bad play, but often it’s not so spectacular you’d remember it as defining.

    • ChinookArchYYC

      I agree with this and I agree with Rex’s view that a balance between both good in-person scouting and good analytical research is required these days (or should). The later is particularly important, because the stats need context (presumably from viewing players) and they need adept scouts that can interpret the data in a meaningful way.

  • Being a professional scout sounds like one of the toughest jobs in the world to me. Hard travel, bad hours and most of your work will be tossed on the scrap heap, either because the kid you liked got picked by someone else or because 95% of prospects don’t amount to much anyways.

    tough gig.

  • ChinookArchYYC

    I always wonder how much group-think sways individual scouts. Look no further than the number of publications that choose Seth Jones to go #1 this season. How much of that was driven by the wind changing after the WJC. Frankly, I wonder how many scouts read their favourite sports rag and allow themselves to see the things their reading about.

    Here’s what the word was on Daniel Tkaczuk in 1997:

    Centre, Barrie, Ontario Hockey League

    Tkaczuk Rank: No. 3, North American Central Scouting
    Shoots left, 6-0, 191
    Born: June 10, 1979, Mississauga, Ontario
    GP: 62 G: 45 A: 48 Pts: 93 PIM: 49

    Scouting report: He is a high-scoring, two-way forward in his league. He is potentially a top two-way forward in the NHL. He has the skills to be a point-per-game player in the NHL. He will need to raise his play to an even higher level offensively next season, if he is going to be a legitimate offensive threat in the NHL.

    Scouting notes: Offensive ability is very good; excels at offense through a good mixture of hard work and natural ability … defensive play is very good; is a complete player including in his own end … skating ability is good; skates well enough to effectively play his style of game … physical play is good; became more physically aggressive as the season progressed … attitude and work ethic is excellent; is a quality person on and off the ice.


    I hate this pick! I just hope it’s a 3rd time lucky with #6 and not the complete set of 666.

  • In terms of cognitive biases, arguably the most important thing the Flames have done under Feaster and Weisbrod is change the marching orders given to scouts. Instead of saying, “Find another Rhett Warrener,” or the like, they’re told to look for specific attributes and then rank players based on that.

    • RexLibris

      This doesn’t necessarily close the door on potential biases or serious errors though.

      If the people assessing “hockey IQ” aren’t competent determiners of talent or ability then the list is built with errors.

      Also, while the marching orders could be simple such as “find me a player with a great shot differential, someone who really pushes the river” if scouts focus in on that ability without taking into account other factors they may mis-prioritize certain prospects the same way that earlier generations of talent assessment misjudged the Robbie Schremps and M.A. Pouliots of the CHL world. Can you tell I’m speaking from painful personal experience? 😉

      • SydScout

        But Rex, I think the idea here is that scouts (aka humans) can only focus on a few things or one thing at a time. Trust me, my missus can’t drive and talk on the phone at the same time but still tries to and puts the fear of God into all other drivers on the road (sorry for the religious reference to anyone offended)

        So it’s a case of what’s the most effective – scouts given a mandate to look at the whole package (unlikely to be effective when scouting large numbers at one time) or being asked to focus on certain attributes that the management team have identified as most crucial. If that’s what Weisbrod and Feaster have requested, I give them credit

        • SydScout

          Scouting is not an exact science, and it is really a difficult job to see 100 games a season and not get caught up with all the other scouts opinions. Sometimes the best way to scout a player is the first time just watch the game and see who sticks out based upon your criteria, that is often who you came to see anyways,if not you watch you guy periods 2 and 3. You might then come back and see the mystery player at another time. No offence taken.

  • BitGeek

    I think Analytics can help tell you which players you need to go look at in person.

    It would also help if more games at the lower levels were recorded and made available to more people to look at.

    I can’t imagine a scout not bringing a recording device to the games he watches to make sure he doesn’t miss something while watching the game.

    I think watching a player definitely brings context to the stats you have on that player. It should fill in the missing pieces that stats can’t always tell you.

  • McRib

    Interesting comment made by Weisbrod in the Hearld “Sieloff’s a guy that could, conceivably, be playing hockey very soon.”

    The Flames seem to just love this kid its starting to sound like he is at the top of the totem pole for Junior Defenseman. I think he ends up with Abbotsford next year, as the AHL is a better place for defensive defenseman to develop than run-and-gun Major Junior.