Let’s talk about stats vs traditionalists
By Kent Wilson
7 years agoThe advanced stats vs conventional hockey people fight is probably getting old for anyone watching the argument from the outside. Nevertheless, it rages on, flaring up whenever a new school person flagrantly denies conventional wisdom or an old school person balks at nerds analyzing the game with spreadsheets.
Many might be surprised to find out that the two schools of thought actually agree more than they disagree. The problem is, there are fundamental areas of friction between the two sides that may never be resolved. Let’s talk about these issues, from the view of a new school outsider with a background in psychology (i.e.; me).
So sum up what you’re talking about here. Why can’t we get past this annoying sniping between the two sides?
The problem is there are intractable differences between traditional hockey thought and newer, evidence-based evaluation. Some things that are considered irreducible primary truths or axioms by traditionalists don’t seem to have basis. Or at least in the current environment of the NHL when it comes to, you know, things that help you win games. Culture can do this – inculcate and insulate beliefs that are widely held and wildly wrong all at once. That said, culture can also bury deep seated truths in long-held beliefs and rituals. The problem is, it’s difficult to separate the truth from myth, especially when some beliefs become sacrosanct or considered too “obvious” to be challenged.
So while there is probably a lot of overlap (coaches actually tend to make decisions that are largely in agreement with corsi, for example), some of the key friction points are almost irreconcilable between the two sides.
So what are these “friction points”?
We’ll start with an area almost everyone struggles with: randomness. Traditional hockey guys (and almost everyone else) don’t think probabilistically and don’t understand randomness. Like, at all. The traditional mindset is very much “100% causal” in that what happens is considered inevitable. Everyone has full control over their results.
As a result, they are endlessly chasing results and tend to overweight a player’s previous accomplishments, even if they were mostly team based. They also seem to fall prey to the fundamental attribution error, a bias that supposes personal factors are the primary determining factors in all outcomes. Would the Canadiens consider Andrew Shaw “a winner who hates to lose” if he had played for Oilers rather than Hawks, for instance? Probably not, but the player didn’t really have control over what team he played for and his perception as a winner is almost wholly dependent on that variable.
How do you come to this conclusion?
While corsi gets the most play in the media, one of the biggest leaps forward in hockey analysis was actually PDO and the subsequent acknowledgement of the sway of randomness over results in the NHL. For years, analysts on the new school side of things have watched as narratives and stories about teams and players have been fabricated to explain the random swing of the percentages. Winners and leaders one year become losers and malcontents the next.
The culture of the league also bears explicit antipathy towards a luck/circumstances mindset in players. No one likes a guy who blames his coach, teammates, GM or luck for bad results. Similarly, no one wants to accept that success can be due to good fortune. Our brains prefer the story, which seems satisfactory and obvious in the aftermath.
What’s next?
Traditionalists frequently mistake what they like with what is good (i.e.; what is useful) and that dissuades them from looking for actual relationships between winning and factor X. See: blocking shots, being “tough” and hitting and how they don’t actually tend to correlate to winning.
Again, this is a common psychological bias. It’s hard to know how much a certain action or metric actually influences winning, so instead of answering that question, the brain substitutes something easier – and seeing a guy who is willing to sacrifice his body to block a shot, fight an opponent, or body check opponents is observationally obvious, easy to recall, and superficially commendable.
On the other hand, stats analysts tend to treat each assumption as a theory that requires verification and testing. And here’s the thing: the stuff that turns out to be the most counter-intuitive is what tends to get the most attention and interest – because it’s controversial, noteworthy and points to a potential market inefficiency.
Thus you get the conflict of “this is how the game has always been played” versus “but it’s not right.” You have guys who haven’t played at the highest levels telling ex-players, coaches and longtime fans what they have always believed as not only true but plainly obvious.
Why do you keep talking about psychology?
Because this is, at its base, a conflict about ways of thinking about the game and not actually the particular facts and squabbles that erupt. One of the problems with eschewing careful stats work and evidence-based analysis is that you can be prone to the narrative fallacy.
That makes people prone to reductive, “story”-based thinking when it comes to attributing success or failure to certain variables. So, for instance, Aaron Ward’s take that Carolina’s unique culture was what led them to win a Cup in 2005-06.
That’s an anecdote, not proof of the role culture has in winning.
No one mentions the teams with great togetherness or culture that don’t win anything (and they certainly exist, because only one team wins the Cup every year). Moves that are made to improve morale, leadership, culture etc. are rapidly forgotten if nothing notable happens after them. Also, a good culture can often trail winning (rather than predict it) because it’s a lot easier to have a good dressing room when the team enjoys success. This is true of “confidence” as well. Disentangling these issues is a lot more complicated than looking at the standings and assuming one team, coach or player simply “wanted it more”.
So culture doesn’t matter? That’s ridiculous.
No, I wouldn’t say culture doesn’t matter. Managing a room of people to ensure everyone pulls in the same direction is a challenge in every walk of life, so it’s likely a factor in hockey as well. The problem is, culture and similar fuzzy concepts (like leadership and heart) are often wielded as fudge factors or deus ex machina devices that explain away certain outcomes after the fact when it comes to this stuff.
Here’s the problem from a stats/evidence perspective: culture and such is almost always offered as post hoc rationalization and not as a useful, predictive factor in advance.
To understand the role and effect of culture on winning, it would take careful, systemic study of what aspects of teamwork and leadership actually lead to improved outcomes. This is possible, but NHL teams and traditionalists haven’t done this (as far as I can tell) – they tell stories about what happened and assign roles to actors as heroes, villains etc. in the events afterwards. This is prone to misattribution, scapegoating and over- or underweighting of things because no one can really be sure of what matters and what doesn’t.
In short: culture no doubt matters and it might matter a lot in certain ways. But we don’t really know what, how or why or how to use it to predict success.
Is there anything else to this feud? How about the jock vs nerd divide?
Yeah, there’s another aspect to things like toughness, hitting, size, tough to play against – i.e.; the various factors that still tend to be relatively overweighted in the league. They are part of a particularly masculine “might is right” psychology that underpins all competitions of this nature.
Sports are a proxy for war, sociologically speaking, so hockey men tend to eschew things that make them seem “lesser” – small, weak, vulnerable to physical intimidation – and value things that make them seem dominant – size, strength, aggression, toughness. This bias operates in society in general. Taller men are more often chosen as leaders and tend to make more money (and get more dates).
Being big, tough and mean subconsciously grades a guy as a warrior. Being smaller, less aggressive, more “childlike” marks a guy as someone vulnerable and in need of protection in our lizard brains. That’s why small guys have to prove they can play hockey and big guys have to prove they can’t in the NHL.
So only the stats guys are “right”?
No. I’ve mostly represented that side of the debate because that’s where I come from and best understand. It’s also notable that hockey coaches and decision makers have been making decisions in the absence of “big data” for a majority of the league’s existence, so they created methods and norms that probably best suited what was, or rather has been, available.
Next, know that the advanced stats side isn’t a single, monolithic group (there are on-going battles amongst new school guys going on all the time, but those don’t tend to extend beyond the community), so what is “right” is often in contention on that side of aisle, too. Remember, also, that new school guys don’t have perfect data or knowledge either.
GMs, coaches and players are acting on different information and often under different sets of incentives and pressures. For example, minimizing the role of “luck” in personal and team accomplishments in hockey is probably an adaptive response – believing you are in the arbiter of own your fate instills an internal locus of control, which helps motivate individuals to try hard, strive for me, opt for self-improvement, etc.
The stats side may also have to come to accept that the ideal team on paper is entirely impossible to build for practical reasons that include the human, messy, intangible side of the equation. Personalities, demands, desires, egos and expectations can hopelessly complicate things in the ground but be completely imperceptible from the air.
As mentioned, “culture” tends to convey embedded truths and experience, though the challenge is separating dogma and the credulity of blind convention from behaviour and beliefs that are actually useful. This is noteworthy because what was once true for a people, tribe, culture etc. may not necessarily apply anymore because things inexorably change and evolve. This will also apply to the stats side of things as “corsi” and such become part of conventional wisdom – it’s easy to stop evaluating your assumptions when they are widely established as true.
The next phase of evolution for both sides would be to identify what is useful from the other side of the divide. The team (or analysts) that can effectively view the forest and the trees will be able to sidestep common errors, identify market inefficiencies and integrate empiricism (test and prove, test and prove) throughout the organization, from scouts and GMs to the coaches and players.
Maybe at that point we won’t see these fights anymore. Not that fans, coaches and analysts won’t always have different impressions and opinions about the game, but maybe we won’t have to talk about the fundamental ways in which we approach and think about it.
Recently By Kent Wilson
Recent articles from Kent Wilson
