Random Thoughts – On Process vs Results



The recent struggles of Sean Monahan is a topic of interest for me, in part because it sparked a discussion about the nature of analysis, particularly regarding valuing results and processes. Let’s get to the Monahan stuff first and the rest will follow. I’ll talk about Mikael Backlund (yes, again) here as well.

Monahan’s struggles

Since returning from injury, the Flames rookie center has been getting absolutely killed. Not including last night, Monahan had been outshot (including blocks and missed shots) 48 to 15 or -33 over just three games at even strength. That’s a corsi % of 24%(!!). To put that ratio in context, the worst regular skater in the league by this measure so far this year is the Leafs Frazer McLaren at 36.6%. Brian McGrattan’s corsi% is 41.1% so far (worst forward on the Flames).

Keep in mind, this is while playing with Curtis Glencross and Jiri Hudler and getting fairly generous zone start bumps – the kid’s offensive zone to defensive zone faceoff ratio in those three games was 80%, 55.6% and 60%, respectively. So circumstances don’t explain him getting run over.

Even in such a small sample of games, these are worrying possession numbers and suggest something has gone terribly wrong. Either Monahan has hit the dreaded "rookie wall" or he has come back from injury too early. Either way, his night in Detroit began the same way, if you’re wondering why the kid was benched in the third period.

Anyways, my noting the Monahan’s dreadful outshooting numbers after the Rangers game (22 shots against, 6 for, -16 corsi) led to some discussion on Twitter. I was reminded that the Monahan line scored two goals despite being grossly outshot and that those are the only results that matter. Two goals for and zero goals against means a good night of work, all other considerations be damned.

Well, not really. As I note here continually, the best long-term predictor we have for outscoring in the NHL is outshooting (ie; positive possession). I think even the most innumerate amongst us can also recognize that getting outshot 22-6 but coming out with a 2-0 goal differential is more horseshoes than ability.

This brings me to the discussion of results vs process, which I’ll illustrate with a personal anecdote:

Years ago some friends and I used to get together monthly for a night of texas hold’em poker. This particular evening I was fortunate to be one of the last two players at the table. The game had gone on for a long time, it was a weeknight and my stack was beginning to dwindle. We also gave out a portion of the pot to the second placed guy, so with the guarantee of winnings I simply decided to throw the game and go home.

On the next hand, my two hold cards were a 2 and 9, off-suit. Perfect, I went all-in before the flop. My adversary called with something much stronger – a couple of face cards or something similar. He was much more likely to win, which was my intent anyways.

Except the flop happened to come up 9, 2 other. I had two pair and I went on to take the hand. I got a good result despite a terrible, low percentage play. It happens in cards.

And it happens in hockey. One of the reasons I was drawn to better, more predictive metrics in the early days of my own hockey writing is because I found myself too often chasing my tail in terms of my assessments of teams and players. I frequently found that the "numbers that matter" – ie the results like goals and wins – didn’t seem to accurately predict who was actually the better player or club down the road. I found myself bouncing around and flip-flopping, resigned to deploying shopworn, retroactive narratives like confidence and other pop psychology to explain away things when they didn’t accord with what had seemed to be solid conclusions.

Like a bad poker player who chases flushes, projects his betting habits and doesn’t understand how to bluff, bad hockey teams and players can nevertheless experience pockets of success. Sometimes you get the cards. Sometimes the puck bounces off a skate and goes in or your goalie stands on his head. Sometimes luck can favor you for longer than a game or hand. But, eventually, playing lousy catches up to you.

The objective in hockey is to score more goals than the other guy. When that happens, it can be seductive to assume that the game was played well. But it can be a false signal. Process matters. That’s why, despite winning big with a 9-2 off suit that one time, I still fold the hand whenever it pops up.

Backlund’s Ascension

Which brings us to Mikael Backlund, whose recent spate of good play has ironically made him a player of contention in some quarters again.

Let’s first break down his last few games like we did with Monahan – in the last three, Backlund’s raw possession numbers have been +33 -28 for +4 or a ratio of 54%. This is especially noteworthy because the Flames have been soundly outshot in each of those contests – Backlund’s corsi rate relative to the rest of the team in those games was +15.7%, +32.9% and +13.3%. Unlike Monahan, Backlund wasn’t starting more shifts in the offensive zone either – his o-zone to d-zone faceoff rate was 22%, 23% and 45%, respectively.

So that’s some quality work and the reason why Mickis has lead the Flames forwards in ice time recently. You have to go back to the end of November to find the last time Backlund was scored on at even strength (against the Ducks), which is a function of good play and luck (100% ES SV%!). Still, this is Backlund’s under appreciated talent – the puck moves the right direction when he’s on the ice. He doesn’t do it by hammering opponents with big hits or making Datsyukian dekes through the opposition. It’s just quiet, dogged puck pursuit and an ability to read and react to the play. Which is why he’s an uninspiring player to many pundits and fans – we tend to notice and remember the profane and spectacular and not much else.

One of those pundits is Dean Molberg (Boomer) on the FAN960 in town. A recent exchange between him and friend of the program Ryan Pinder on the radio this past week sparked me to write this section…

The argument was over Backlund, with Pinder’s defense couched in possession metrics. Boomer, a staunch disbeliever in corsi, stated that whatever the numbers say, Backlund seems too passive and uninvolved to be a player of any value. The argument devolved into Molberg more or less telling Pinder to step away from the "nerd stats" and watch the game with the implication being that Backlund would fail the eye-test absent the crutch of arcane measures. 

Let’s first establish that the legitimacy or utility of evidence is not subject to any given person’s subjective appraisal or incredulity. Information is information, even if what it implies is personally hard to digest or fathom. Every major discovery or advancement of knowledge in human history was initally met with disdain, contempt and dismissal by people in general and established authorities in particular. Truth is immune to popularity.

Furthermore, let’s also establish that anyone who has to muster a smear or stigma to invalidate evidence is immediately behind in an argument. One of the easiest ways to rationalize away uncomfortable or counter-intuitive information is to label it as somehow heretical and therefore not worthy of serious contemplation.

An analogy…the theory of evolution is based on lots and lots of data and evidence. It’s the best understanding we have of how life grows, changes and persists on this planet. But there is a large swath of so-called "creationists" who dismiss evolution out of hand because its conclusions do not accord with religious dictum. So the evidence behind evolution is labeled blasphemous and redacted from the minds of true believers because it is rationalized as inherently wrong. 

My intent is not to call Dean the hockey version of a creationist (seriously…I’ve met Boomer and like him), but to point out how  cognitive dissonance can make people act and think in less than rational ways.

Check Your Premises

The irony is that suspicion of numbers is not unwarranted. As I mentioned above, people most certainly should be suspicious of the value of outcomes like wins, goals, points, etc. (particularly in small samples). One wonders if the established hockey orthodoxy grew distrustful of stats over time because so many of them (*ahem*plus/minus) didn’t paint an accurate picture of a team or player.

The problem is, no one treats their observations with a similar level of suspision. We’re hardwired to believe what we see and, further, to take subjective impressions and compelling anecdotes as implacable indicators of truth and reality. No matter how much you respect the power of a given metric or series of numbers, they never seem as viscerally real as the instant judgements and gut feel we get from watching tha action.

Which is why it’s sensible to respect and investigate quantitative evidence and admit, at times, that subjective evaluation might not completely line-up with reality. Humans are good at naturally detecting patterns in data, but we’re terrible at telling when those patterns represent real causal relationships or not.

I’ll finish by noting that the current disruptive "new stats" like corsi and PDO that give so many established MSMers (and some authorities in the league) fits weren’t constructed by theorists in a lab far away from the game. They were stumbled upon and tested in a series of iterative studies by passionate, invested hockey fans and analysts – a sort of crowd-sourced empiricism where the utility of possession metrics and percentages was tested and re-tested and then built upon when they proved to be relatively robust. To put it another way, these aren’t numbers cooked up by people who don’t watch hockey and are simply looking to completely up-end current conventions because "screw you jocks!"

For my part, I think the point of discussing and disseminating this sort of analysis is to remind folks to have humility before the facts. Or, to put it another way, to check their assumptions now and then. Experience can lead to people creating mental short-cuts (rules of thumb) that are easy to apply and generally represent a reasonably good proxy of the truth. The problem is,  when these heuristics either get raised to the level of principles or ossified into dogma.

With rules of thumb, sometimes the nature of the relationship between the variables is lost until all that is recalled is a fuzzy correlation. For instance, good goalies make it much more likely their team will win because they inhibiting the opposition’s ability to score more goals. Therefore, it’s a good heurisitic to say "good goalies have good win totals". On the other hand, goaltenders have no control over how many shots their teams give up or how many goals their skaters score, which are also key factors in winning, making it entirely possible to be a good goalie but have an uninspiring W-L record. 

So the proper expression of this heuristic is: "Good goalies will probably have more wins, but not necessarily." What that so often gets simplified into is: "Good goalies get wins," to the degree that the direction of the relationship is inverted: lots of wins means a goalie is good, which is at times completely false and can lead to poor analysis and bad decision making.

This relates to my recent discussion of size and toughness. All things being equal, it’s usually preferable to have bigger, tougher players. It’s important to keep that qualifier in mind, though, because all things are rarely equal. Bigger = better is another one of those rules of thumb that too often becomes over-simplified and deployed as if it’s a principle rather than a rough guide with caveats.

  • Derzie

    You can drive the play/possession all night. But if you make 2 mistakes on the PK in your own end that turn into goals, your stats still look great but you were front and center as to why the team lost. Stats can be interpreted to suit most any hypothesis. Boomer and Pinder are both right. Backlund has some nice defensive numbers and he is prone to turnovers and fails the eye test a lot.

    • SmellOfVictory

      Every player makes turnovers. Backlund makes less of them than anyone on the Flames, from what I’ve seen, outside of perhaps Brodie.

      As to “stats can be interpreted to suit most any hypothesis”, that is only true if you’re talking to someone who doesn’t understand statistics or you use inconsistent parameters. Where is the interpretation with Corsi? Better shot attempt differential (taking into account offensive zone time) = better player. Where’s the manipulation in that? Or how would it be interpreted otherwise? The only time I’ve ever seen it misinterpreted to serve someone’s pre-existing bias is when stat-haters use the argument “some players take tons of low percentage shots which makes them look better than they are, ergo Corsi is stupid”, which is such a minor/infrequent issue that it’s not even worth consideration.

  • SmellOfVictory

    Monahan will have his ups and downs, especially after injury. He responds well to challenges so he will work his way out of this slump..

    I don’t overly like him playing with Glencross. I love Glencross when he’s healthy and scoring. But he isn’t good at breaking out of our zone because he makes bad decisions when trying to get the puck out.

    I think playing with Glencross hurts his possession numbers. I dunno the CORSI on Glencross but I’d be interested to know what the numbers do in fact say.

    In regards to Backlund I think he has been playing amazing lately. You can really see his abilities with and without the puck. He is extremely good at maintaining possession by stickhandling or reading the play before it happens.

    I have always like him and want him on this team, especially more so than keeping Stajan.

    But I think when everyone talks about not caring about what the numbers say its because at the end of the day you need results. In this business results are goals, assists, hits, wins ect.

    Backlund is a great player when he has confidence and is contributing offensively. But when he, so often does, struggle he isn’t as useful.

    Part of his struggles have been from injury which is out of his control but that’s possibly why having bigger players, who can drive the puck north, is a better idea and what Burke is looking for.

    If Backlund can be a second line center there is room for him here. But if he is going to continually have consistency issues he won’t be.

    I actually like the idea of having a bigger bottom six.

    I hope Backlund is part of the rebuild. Its really in his hands, if he plays like he has been, for the rest of the season, he will be retained.

    Backlund is really the epitome of the debate with advanced stats. The numbers say he has supreme value. But if he isn’t producing offense he doesn’t have value on Burkes team because he is too small to play a bottom six checker role.

    I’m fine with that, even though I like Backlund a lot.

  • I would like to apologize to the FN community. In an earlier post out of haste i made one of those silly flaws I was referring to where i applied directionality to cause and in doing got the direction of the cause wrong.

    We were having the debate about theory in the scientific method sense. One of the beautiful things about the english language is how something in everyday language can mean something completely different in another forum, in this case science.

    Someone replied and suggested that theories cannot be proven. This is not strictly accurate in a scientific sense. In the regular sense this is true a theory is a form of speculation about how something works but may not necessarily have the backing of evidence. A theory in science, however, is a testable and repeatably observed phenomenon. However, all theories can at any time be falsified by continued scientific learning. That is why science is so great and why it’s important for scientists to publish results.

    Theories, hypotheses and the scientific method operates on the premise of rejecting the null hypothesis. A theory is developed to explain the observations tested by hypotheses. In some cases a theory is formed from a series of hypotheses. Theories really are explanations for what we observe.

    Here is where I got the directionality wrong. I implied in an earlier comment the Theories which are repeatedly observed should be promoted to laws. This is simply the not the case. In fact, Laws, for example, gravity inform the theories. The law of gravity for instance states that objects which are not opposed by any other forces will fall (In simple terms anyhow). A law is just simply a repeated observable fact. The law of gravity serves to inform the theories that were set out by Newton and later Einstein. Einstein’s theory of relativity didn’t disprove the newton theory but instead expanded on it giving us a better understanding of it. In that sense Einstein didn’t falsify Newton’s law of gravity but added more dimension to it.

    The theory of gravity could always be later disproven by further evidence. Though, currently it’s safe to say that a theory which has been repeatedly tested with the scientific method is not so much a theory in the common sense but in fact a true part of our world. Evolution follows this course.
    So why should i bring this up in a hockey blog?

    Advanced Statistics are like the laws that inform our larger theories about player development, skill and quality. Corsi, Fenwick, PDO are akin to taking a feather and a book to the top of a high rise downtown and repeatedly dropping it to see which if either will land first.

    The statistics are then used to formulate theories about the progress of a player. For example. if we had follow adv. stats 5 or 6 years ago as closely as we’re following now. One might have looked at Patrice Bergeron and said you know this kid is impressive in the defensive zone and a monster in the face-off. He has the potential to become one of the best 2 or 3rd line Centremen in the league playing against the hardest opposition on a regular night basis.

    In my mind there is no doubt that advanced stats have served to inform the analysis of certain players and teams performance. What we need now is to move towards testing these observations/stats against the results. Unfortunately, this requires continued longitudinal study in order to gain a large representative sample size.

    EG. Backlund either just turned or is turning 25 in the coming months. What is performance at the end of this year and what will it be over the next 3 seasons.

    The adv. stats side of things are saying; Backlund will likely be in the top 10 NHL mould of defensive first offensive centre’s. Let’s see if that comes true. Will he become as valued as Bergeron in the coming few years?