Asking the Right Questions – The imprecise and unattainable



"Not many people have grace."
"Well, you know, grace is a tough one. I like to think I have a little grace. Not as much as Jackie O…."
"You can’t have a little grace. You either have grace or you don’t."
"Okay, fine. I have no grace."
"And you can’t acquire grace."
"Well, I have no intention of getting grace."
"Grace isn’t something you can pick up at the market."
"Alright, alright, look, I don’t have grace, I don’t want grace, I don’t even say grace, okay?" 


Previously we discussed how asking the wrong questions is the first misstep management of lackluster teams can take in the off-season, with a particular focus on being skeptical of small outbursts of success amidst a general sea of mediocrity. This time around, I want to look at how fans and decision makers can get wrapped up in psychoanalyzing their players or agonizing over fuzzy, non-specific but apparently plausible factors.

In his year end press conference, Brian Burke noted he wanted the Leafs to get bigger and tougher going forward. Tyler Dellow has noted Steve Tambellini’s singular and bizarre fascination with the term poise. In Calgary, the implicit, operating assumption for the last three or four years has been the players are good enough to win on the ice, but they need more ______ (fill in the blank with whatever term you care to use. Leadership, will to win, cohesion, consistency of effort, grace etc.).

Although looking for toughness or poise or leadership sounds desirable (and their lack makes for a plausible explanation for failures), the problem with those concepts is their subjectivity and imprecision. During such press conferences, I always wait for someone in crowd to start asking for specifics when GM’s start throwing out the ol’ "we need our men to be decidedly more manly" quotes. For example:

  "How much bigger and tougher do you need to get? Is that in parallel with other qualities (speed, intelligence), or in favor to?"

 "Do you expect a magnitude increase in toughness? How will you  measure that?"

 "Do you expect more toughness to lead to less goals against? Or more goals for? Both? How much do you expect this increased truculence to improve your team’s goal differential? Why?"

 "How much do you expect to pay for this increase in muscle? How do you determine that this is an efficient use of resources?"

 Of course, such questions would be mostly facetious. While concepts like toughness/leadership/poise are easy to visualize as useful, tangible factors, they can’t actually be identified or quantified in any meaningful sense. As such, they also can’t be applied in any sort of rigorous analysis in terms of how "generally accepted positive quality X" affects goal differential (ie; wins) – except in the most rudimentary way (they probably help somehow).

The problem is that the relationship between stuff like leadership and poise to success is indirect, assumed and impossible to measure. Which isn’t to say wanting good guys in the room or lots of poise (however you want to describe that) is necessarily bad, it’s just that managers who make grand, conventionally valued but ill-defined qualities their primary objective can end up chasing ghosts and gossip.

It’s that type of thinking that leads you to trade a third round pick for an overpaid Steve Staios at the deadline or sign a doddering, gin-soaked Nikolai Khabibulin for $2 million too much and two years too long. The idea that reputation and attitude in the room trumps on-ice performance or talent level is what so often leads to bad bets or gross inefficiency.

If you’re a manager or fan of a bad team, your club probably has fundamental deficiencies in terms of talent in various places on the roster that lead to either too many goals against or not enough goals for (or both). Decision makers would be well advised to concentrate on shoring up those areas by asking questions of a players performance.

For instance –

"Does a player drive shots on net? Who dos he play with? Against? Does he produce on the PP? Is he a good bet to sustain that performance?"

Rather than –

"Is he good guy? Has he ever been on a cup winning team? Can he punch other players in the face?"

The former set of questions will get you closer to a players measurable value to the team. The latter are pale proxies of his abilities and value based on crude assumptions. The former should therefore always be considered before the latter in questions of player acquisition and retention.

  • using either: “Does a player drive shots on net? Who dos he play with? Against? Does he produce on the PP? Is he a good bet to sustain that performance?” or “Is he good guy? Has he ever been on a cup winning team? Can he punch other players in the face?” as measuring tools would have kept Stajan from receiving his extension…

    which begs the question about just what the Flames are using, doesn’t it?

    good post, liking the series.

    • A lot of the conventional questions about leadership and such naturally come from a time when folks didn’t have a lot of data to work with – goals, assists, games played, wins, plus some observation. That’s it.

      So naturally the mind fills in gaps when there isn’t information available with short-cuts and rules of thumb.

  • This is the typical manager trap. Qualitative attributes are easy for managers to spew, since they require no effort by either the managers nor the workers to justify.

    I’m always amazed that Flames fans (but I guess sports fans in general) fall for this drivel whenever their team starts lacking.

    • It’s entirely human. Trying to carefully investigate a guys results is hard and time consuming. Answering questions such as “do I like this guy” or “does he seem to exude certain positive qualities” is a lot easier, so it comes to mind more readily and is therefore more seductive.

  • T&A4Flames

    “Is he good guy? Has he ever been on a cup winning team? Can he punch other players in the face?”

    These are the questions that us hockey fans should ask, not managers. Especially, and my favorite…”Can he punch other players in the face?” I love that. I found myself doing that a lot this past year, asking “why can’t he just punch him in the face!!!”

    Too funny Kent.

  • RexLibris

    I remember reading about scouts who had interviewd both Turgeon and Shanahan before their draft year (1987) and one scout in particular said that when he sat across the kitchen table from Turgeon it was like the kid had no idea why he was there. He was almost waiting for the scout to ask him something and then tell him how to answer. Shanahan wasn’t as flashy a prospect but scouts got the impression that “there was more there” in the player.

    I think that advanced stats are beginning to illustrate what the better old-school scouts have been able to determine through observation and subconcious evaluation. A good scout may have watched a player and been impressed by his play without having quantifiable values to suggest that the play and puck went the right way when this player was on the ice and so on.

    For instance, a good manager can be one that hires the right people because they are an excellent judge of character and can determine through the interview not only a candidate who is qualified but is also has the drive and work ethic to be a good employee.

    In Tambellini’s case he has said that this is what he has done with his scouting staff and even involving the owner, Katz, in the process when it came to their impressions of either Hall or Seguin.

    So while I agree that, in the Oilers’ case, poise is a near impossibility to quantify, I don’t think that it is entirely a red herring for a management group to consider.

    I will also agree that as fans we tend to put more stock into these things than sometimes they merit. An example from my personal experience would be the Hall/Seguin debate. I was firmly in the Seguin camp due to position and his playing on an inferior team and posting equal numbers. Until I saw a photo from the combine where Seguin had a tattoo of his own last name on the back of his bicep. That signalled to me that he may be a more selfish player, one that might not have the same depth of character and humility that most teams should look for in a franchise player. It was entirely superficial and based upon my own prejudice (not against tattoos, but against habits that celebrate oneself), yet in the end it may turn out to be prescient based on some media reports about Seguin’s work habits in his first few years.

    As for Burke’s comments, I care as much about Brian Burke’s opinions as he does of mine.

  • RexLibris

    I don’t disagree that Kent’s analysis is fair but I would point out two caveats:

    1. These are public statements and, as such, I think managers know they will be subject to scrutiny. While I don’t doubt that many decisions made by managers will involve discussions with all of the terms listed (see caveat number 2 below) if managers made public disclosure of the more precise analysis it would be subject to both easy scrutiny and demonstration of its falseness. I can prove easily how much possession or save percentage a player had, I can’t prove how tough he is.

    2. Almost all of the data a GM has is historical. Almost every decision a GM has to make is a predictive one. While the historical data must be utilized, a large part of a GM’s job is a guess at whether the historical trends are going to apply to this player or not. Otherwise, assuming all teams spent the same amount of money, all the teams would be perfectly statistical equals to each other. I would allow for GM’s (or other talent evaluator’s) in making that bet to incorporate some analysis of the non-tangibles Kent listed.

    So, if a GM who has watched thousands of hockey players is looking to make a bet on two players that have similar objective measures indicating expected future performance, I think if he bets on the one that he thinks is tougher or has more heart being the one that will match or exceed that expected performance, that is what he gets paid to decide.

    I think that many GMs put far more emphasis on these facets, as Kent suggests in his article.

  • RexLibris

    Most places worth their salt use behavioural-descriptive interviews to determine the suitability of a candidate (basically, ask the interviewee to bring up past examples of what they did in situations).

    In regards to Seguin, dude got 7 points in 13 playoff games last year. We might never know with Hall 🙂

  • RexLibris


    re: Seguin, he was also suspended from play after being late a second time for practice. The interesting thing was his own roommate “neglected” to wake him up after his alarm didn’t go off.

    There’s something else going on there.

    The kid is talented, certainly, and I would love for the Oilers to have him, but I have seen a lot of character in Hall and they are fairly close in points despite the fact that one plays on a vastly inferior team and a different position altogether.

    I’m not opening the Hall/Seguin debate again, I just wanted to use that example of relying on personal bias and perception as a fan to inform on a pair of choices (neither of which were mine to make).

      • RexLibris

        And you are going to credit that to Seguin? Had the Leafs retained that pick and drafted Seguin, do you think he would have taken that team into the post season?

        I admit, I’m speculating about Seguin’s character. That was the whole point of the story, to say that we, as fans, often extrapolate from limited and circumstantial evidence. Media often do the same thing, except in their case they have business model that encourages it. Us fans do it out of amateur (as in, for the love of it) desire.

        • Sobueno

          So uh Rex, when are they going to make you an official contributor at the network?

          Hard to admit (considering your team preferences), but I do enjoy your informed and well articulated arguments/responses.

          Just one question: how does an Oiler’s fan know just as much if not more about my team than I do? Other than the fact that I spend my life toiling away at school in the library rather than learning what’s going on in the “real” world.

          In regards to the so-called character decisions… when it comes to making a call between two potential draftees that are, on paper at least, identical, it seems like a moot point who gets chosen. There’s no measurable way to decide the better of the two, no crystal ball to see who will be more successful, and much like in the Datsyuk example, if GM’s/scouts rely too heavily on impressions you can make a poor call anyway (ie. it can likely go either way; essentially it is a coin flip regardless of how one may like to convince themselves it’s not). I guess in that situation you can factor in team needs and pick based on position.

          Thus I conclude that the Oiler’s positively made the wrong decision in drafting Hall! Well, maybe Hall will be better. Or is it Seguin? Seguin definitely has the greater truculence factor. I’m feeling like an NHL GM…

          • RexLibris

            Thanks Sobueno,

            I’d love to be a contributor but I’m not sure that Vintage’s blood pressure could handle it. 😉

            Thank you for the compliment about my info about the Flames, though. I try to have some background information on the topic before spouting off about it. I’ve been paying some close attention to the team for about six years now in part because I wanted to see if Sutter could actually maneuver them closer to the cup. Then later, when he hired Keenan I was pretty sure that things were going to come unraveled and kept an eye on the team to see what would happen. So a large part of it could be said to be fueled by schadenfreude, but also I liked the tone of the articles on FN and after reading for two years decided it was time to weigh in a little.

            And I would hardly call the sports world a “real world”. More like a juvenile abstraction fueled by vicarious sentiments. The library is probably a far more productive place to be.

            My dilemma example there was essentially asking Kent if there was a method he would propose to help break that deadlock, or if there were a set of statistical data that he might prioritize to try and end the stalemate. Kind of like the breakdown that the NHL goes to in the event of a tie (wins, wins against opponent, goal differential, etc).

            If the Oilers had selected Seguin then Hall would have won a cup and the Oilers be two deep at centre. They didn’t and now they have are still two deep at centre (though Gagner is not as talented as either Nugent-Hopkins or Seguin) and have a winger who is perhaps a season removed from becoming the team’s captain. That draft year, with two exceptional franchise forwards so close together, was a rare thing.

          • Vintage Flame

            I’d love to be a contributor but I’m not sure that Vintage’s blood pressure could handle it. 😉

            It’s always nice having you around Rex. You’re one of the few Oilers fans that don’t refer to me with four letter fun words… and I always enjoy your comments around here.

  • RexLibris

    @Kent Wilson (sorry for the formatting, the reply link isn’t connecting for me)

    So then based on the Willis/Cult of Hockey article, Kent, do you think there is any background information that a scouting department can gather by interviewing past coaches, former teammates, and such? Particularly if these are people who no longer have a vested interest in the performance of that player and so might be less inclined to manufacture a favourable answer.

    After all of the stats and quantifiables are taken into account, how is a scouting staff to determine a preferable candidate between two identical options? Imagine this scenario, in extremis obviously, but nevertheless, a GM has a choice at the draft table between one of two twins both of whom have identical stats and play the same position. I don’t know that anyone would feel comfortable having to fall back on the coin flip method, so what is there left to act as a determining factor?

    In a more realistic approach, barring another set of Sedins in the near future, looking at the Hall/Seguin debate their points were nearly identical, they were about the same size and aside from positions and playoff success there was little to differentiate them. Leaving out the draft-by-position factor because that can vary depending on the team (the Bruins admitted that they would prefer to draft a winger that year, while the Oiler’s rebuild was in obvious need of a centre) the playoff performances and unmeasurables of Hall having been a catalyst in his team’s comeback from a 3-0 game deficit seemed to be a determining factor.

    I understand that if the CHL kept more advanced statistics (TOI, even, would be a good place to start) the call might be easier to make, but I wonder what a manager can do if, after exhausting all other avenues of interview and analysis, there is almost no significant separation between two draft candidates.

    On a final note, I agree with the article, and I believe one of your points, in that an interview to “gain insight into the character of a player” is a flawed instrument. I think of the story about Pavel Datsyuk’s pre-draft interview and his body language (shy) probably turning off a lot of scouts and GMs.

  • RexLibris


    Thanks, I’ve really enjoyed much of the dialogue we’ve all had this past season. Seriously, if you ever need a temporary hack to crank out a contribution and make you regular writers look that much smarter you know who to ask.

    Four letter fun words? I don’t know any of those.

    You and I may disagree, but I respect the emotion you bring to your arguments. And Kent and I, I think, would probably agree on some overall conclusions, although we disagree on the different paths taken to get there.