58
Photo Credit: Gary A. Vasquez / USA Today Sports

The case for firing Glen Gulutzan, Part 2: Process vs. results

When offered the opportunity to write an article for FlamesNation, I told Ari that the research may take me down a rabbit hole or two before it was all over. This is one of the rabbit holes I found myself buried in.

(Here is Part 1.)

Why using results and goal differential is how a team should be evaluated

An early Google search produced an article written by Rob Found from Edmonton, and posted online at The Sports Journal, “Goal-based Metrics Better Than Shot-based Metrics at Predicting Hockey Success.” It is a statistical evaluation of 10 years of NHL data to compare goal-based versus shot-based metrics to see what methods are best at determining team success in the NHL. It was written in 2016.

Here is a summary: 14 hockey statistical models were used in the study, 10 based on shots (Corsi) and four based on goals. Some of the models used a single statistic (i.e. shots for), while others were “composite” in that the model used two or more combined statistics (i.e. goal differential).

Based on his research, Found concluded that goal-based models outperform shot-based models in predicting team success, both in terms of team results and individual contributions to a team’s success, and that defence-based models are better than offence-based models (goals allowed is superior to goals for and shots allowed is superior to shots for). As for individual contributions to team success, only forward contributions were statistically significant, and only when using goal-based models. Contributions by defenceman were not statistically significant using goal- or shot-based models. 

I took the statistics from the 2016-17 season and applied them to the standings. While a simple evaluation in comparison to the research done for the article, the results are very good at showing how effective certain stats are at predicting where a team will finish in the standings.

2016-17 NHL standings relationship to shot- and goal-based statistics (data from Natural Stat Trick)

Standings Team In Playoffs Points CF% CF% Rank Shot Differential SD Rank Goal Differential GD Rank PDO PDO Rank
1 Capitals Yes 118 51.66 6th +213 4th +84 1st 1.027 1st
2 Penguins Yes 111 49.80 16th +80 9th +49 4th 1.015 5th
3 Blackhawks Yes 109 49.89 15th -66 22nd +28 7th 1.013 6th
4 Blue Jackets Yes 108 49.57 18th +52 12th +54 3rd 1.020 2nd
5 Wild Yes 106 49.79 17th +63 11th +57 2nd 1.020 3rd
6 Ducks Yes 105 49.57 19th -4 18th +23 9th 1.010 7th
7 Canadiens Yes 103 51.36 8th +31 15th +25 8th 1.009 9th
8 Oilers Yes 103 50.44 11th +131 7th +36 6th 1.010 8th
9 Rangers Yes 102 49.06 21st -27 19th +37 5th 1.016 4th
10 Blues Yes 99 49.50 20th +4 16th +17 13th 1.007 11th
11 Sharks Yes 99 51.38 7th +187 5th +19 11th 1.001 15th
12 Senators Yes 98 48.56 24th -3 17th -4 19th 0.998 18th
13 Bruins Yes 95 54.48 2nd +524 1st +23 10th 0.990 21st
14 Maple Leafs Yes 95 50.11 14th -56 21st +16 14th 1.008 10th
15 Flames Yes 94 50.39 12th +33 14th +3 16th 1.000 17th
16 Predators Yes 94 51.10 9th +85 8th +18 12th 1.004 14th
17 Islanders 94 47.53 27th -136 24th +1 17th 1.005 12th
18 Lightning 94 51.05 10th -38 20th +6 15th 1.004 13th
19 Flyers 88 52.15 4th +251 3rd -19 22nd 0.983 27th
20 Jets 87 48.96 22nd -100 23rd -9 20th 1.000 16th
21 Hurricanes 87 52.29 3rd +173 6th -18 21st 0.986 25th
22 Kings 86 54.77 1st +434 2nd -2 18th 0.983 28th
23 Panthers 81 51.69 5th +48 13th -26 23rd 0.989 22nd
24 Stars 79 50.34 13th +64 10th -38 25th 0.982 29th
25 Red Wings 79 48.52 25th -175 25th -46 26th 0.988 23rd
26 Sabres 78 47.19 28th -324 28th -32 24th 0.998 19th
27 Devils 70 47.07 29th -294 26th -61 27th 0.985 26th
28 Coyotes 70 44.93 30th -516 30th -67 29th 0.991 20th
29 Canucks 69 48.17 26th -335 29th -63 28th 0.986 24th
30 Avalanche 48 48.65 23rd -299 27th -111 30th 0.966 30th
Playoff Prediction Success 60.00% 66.67% 93.33% 80.00%
Average Error in Predicting Standings 8.4 7.13 1.8 3.53

The results speak for themselves. Corsi is horrible at predicting team success. Shot differential is better, but still ineffective. On the other hand, goal differential correctly predicted whether a team was in or out of the playoffs for 28 of 30 teams. PDO was also very effective. If you believe the Corsi proponents, this would suggest that playoff appearances are nothing more than luck.

But what is most impressive is that both PDO and goal differential determined who the top nine teams were, although not necessarily in exact finishing order. Goal differential also predicted the worst eight teams, although not perfectly in order.

In the case of PDO, Found’s evaluation determined that PDO did not even out over 10 years of NHL data, as proponents say it will. He accounted this to some teams taking better shots when compared to other teams, teams allowing better shots than others, and teams having better goaltending than others. PDO is not luck.

On the subject of goaltending, Found pointed out that of the 40 teams to reach the Stanley Cup finals in the 20 years prior to his study, 28 teams had a goalie that received Vezina Trophy votes in the season they appeared in the final, while the other 12 had goaltenders good enough to earn Vezina votes in other seasons. In Smith we trust.

Corsi was so far from being accurate, that four of the top five CF% teams missed the playoffs, and ranked Boston as second despite finishing 13th in the standings. Boston was only one point from missing the playoffs: hardly an endorsement for Corsi being a good tool.

Found best evaluated the results of his findings as such: “There are meaningless shots, but no meaningless goals…”

My point in going down this rabbit hole? Shot-based statistics and goal-based statistics are tools to measure how the “process” is working in achieving team success. But at the end of the day, results come down to wins, losses, and overtime and shootout losses. In other words it comes down to points, and more specifically, points in relation to the teams in the division, and conference.

Increasing shots for and decreasing shots against – controlling possession – is the “process” the Calgary Flames have chosen to use to improve team success. We cannot confuse “results” with “process.”

The Gulutzan Flames: Results and significant statistics

In the NHL, just under 12% of all regular season games go into overtime. Based on this, the average points per team is just under 92. Setting the bar correctly, that means that evaluating teams by the correct standard is to relate everything to 92 points, not 82 and a .500 record.

The 2016-17 Flames finished the season with 94 points, 12 games over .500, but only two games over the average. The 2017-18 Flames are on pace for 97 points, five games over average. The good news: they are better than average.

The playoffs are played and won at 5v5 and on special teams. Relying on 3v3 overtime is not a good strategy, but it has become the norm for Calgary this year. The Flames have been to overtime 17 times this season, second most in the NHL behind Arizona, and in 30% of all games played. They are truly playing for a tie.

In terms of regulation wins and losses in the regular season, the Flames were 32-33 last season. They were the only team to make the playoffs that was below .500 in that category; no other playoff team was fewer than six games over .500. Two Eastern teams missed the playoffs despite being above .500 in regulation.

This season, the Flames are 20-19 in regulation wins and losses, 16th in the NHL. The Gulutzan Era Flames are just above .500 in regulation, and when you include the playoffs, they are three games below .500.

This season, in terms of regulation wins, the Flames have 20, tied for 18th. Vancouver and Chicago are only one win back at 19. They are 11th in regulation wins in the Western Conference.

Even the worst team in the NHL last season, Colorado, is 24-21, three games over .500 and four regulation wins better.

Setting expectations: Roster upgrades

The Flames were upgraded in the summer. On defense, replacing Dennis Wideman and Deryk Engelland with Travis Hamonic and a re-signed Michael Stone made the defensive core one of the strongest on paper. Up front, the core remained intact, with a few minor changes to the bottom six.

The biggest improvement has been the goaltending. Mike Smith was the biggest question entering the season, but he has done better than anyone could have hoped. The backup question has also been answered: although it took some time to figure things out, David Rittich has save and goals against numbers that are better than Smith’s.

There is little argument that Smith has been an upgrade over Brian Elliott and Chad Johnson. As for the rest of the roster, you may disagree that there have been improvements. What we think does not matter: Brad Treliving believes the roster has been upgraded, so much so he leveraged the future to bring in players he believes can take this team to a better finish in the regular season, and on a playoff run in the spring.

Setting expectations: Points

Last season, the Flames finished 45-30-5 for 94 points. They came out of the gate very slow then as they adjusted to Gulutzan’s system. Roughly one-quarter of the season in, they were 8-12-1, and were beginning to show signs of improvement. From that point on, the team ran up a 37-21-3 record, 77 points in 61 games, a 0.631 win percentage. Once the kinks were ironed out, this is the level of play last year’s team was capable of. It translates into a 104-point season.

There is no excuse for a slow start this year; of the core, only Hamonic is new to the system. With the offseason moves, 94 points would be a step backwards. At the very least, this year’s team should be capable of matching the level of play they achieved last season, or 104 points.

Setting expectations: Pacific Division standings

If you look at the division, most of the other teams are in a period of decline or rebuild mode. Whether it has been making cap room by moving out players, teams starting the season with key players on injured reserve, or aging players on the decline, only the Flames made significant upgrades. For the most part, things are playing out very close to what should have been expected.

2016-17

Rank

Team 2016-17 Points Projected

2017-18

1 Anaheim 105 94
2 Edmonton 103 75
3 San Jose 99 100
4 Calgary 94 97
5 Los Angeles 86 97
6 Arizona 70 56
7 Vancouver 69 73

Only Los Angeles has shown significant improvement of those in the hunt for a playoff spot. A new coach and improved play following a disappointing season does make this a predictable result. It should not be enough to catch this edition of the Flames.

If we have set the expectations correctly at 104 points, and a 100-point season is what even Gulutzan predicted was possible, then the Flames should be leading the division. To be third of the perceived contenders, and in a dogfight for their playoff lives, is disappointing. A 97-point pace is a step backwards.

And then there was Vegas. That they are in the conversation at all for a playoff spot is the biggest surprise of the year; to be leading the Western Conference is just insane. It has also put a playoff spot in jeopardy for the Flames.

Vegas is on pace for 113 points, a testament to what is possible with solid coaching. At worst, the Flames should be in second, on pace for 104 points and home ice advantage in the first round of the playoffs. But that was last years team’s potential. Improvements were made. This team should be battling with Vegas for first in the Pacific.

The team is underperforming. Next week, I will look at what is causing the team to be sitting below expectations.

  • Greatsave

    While I appreciate the time and effort that went into researching the data that would support your thesis, I was underwhelmed by both the cited paper and the premise for analysis in general. I hope to lay out why in a later post tonight, time permitting.

    • Greatsave

      To start, I’d like to reiterate my appreciation for the time and effort put in to research your piece. I may not agree with much of it, but I will not deny that it was a labour of love.

      The article cited a 2016 paper published in The Sport Journal by a Rob Found at the University of Alberta. I looked up this Rob Found. He appears to be a Ph.D. In Biology. Specializing in wild elk personality. This is the one and only paper he has published in sports, nevermind in hockey analytics. There is zero evidence to suggest that he is knowledgeable in hockey analytics and reliable as a source of insight. I have not found any online articles or blog posts by him either. If anyone can point to evidence of his activity in, say, OilersNation, please do so. This is not meant to be an ad hominem attack, merely to point out his lack of established credibility as an authority in hockey analytics.

      With Found’s paper itself, I was not impressed. I believe it is flawed in that he has confused “process” and “result”, as well as “predicts” and “correlates”. (Bearing in mind though that I don’t work in a field that requires statistical analysis, I’m prepared to be shown how my understanding is incorrect.)

      “There are meaningless shots, but no meaningless goals.” This quote from Found’s paper was praised both in the article and in the comments section. While it sounds great, it boils down to a truism that appeals to the rules of the game: “The team that scores more goals, wins.” What Found has done is basically confusing “process” and “result”. He’s seeing Wins and Losses as the “result” that teams are interested in, but he fails to see the connection between Goals and Wins/Losses (as stated in the rule above), and he incorrectly categorizes Goals as “process” instead of “result”. Put another way, if we restated the objective of the game–“to Win”–as “to score more goals than the opposing team”, as per the rules of the game, we would see that it does not make sense for Goals to be part of the “process”, since in that case the process would simply be “to score goals” (and the corollary, “to not allow the opposing team to score goals”). The quote sounds great, but it is itself actually meaningless.

      This confusion contributes to the flawed idea of Goal-based Metrics being a good Predictor of Wins/Losses. Since goal-differential leads directly to Wins/Losses, is it not obvious that a team with many wins will have a corresponding high goal-differential? And vice versa? The only reason aggregate goal-differential over a season does not “predict” Wins/Losses with 100% accuracy is because there are different margins of victory. The idea that Goal-based Metrics better Predict Wins/Losses over a season is simply the same truism extended over a season: “If you score more goals than you give up over a season, you will win more games than you lose.” Again, obvious and meaningless.

      This leads to the next confusion: Do Found’s models “predict” Wins/Losses? Or do they “correlate” with Wins/Losses? Now, as I mentioned before, I’m not trained in Statistics, so I may be missing the nuances of what these terms mean in proper Statistics work. But given the meaningless truisms stated above, to say that “Goal-based Metrics Predict Wins/Losses” is like saying “The goals columns on the scoresheet predict who wins a game.” It’s nonsense. The goals columns of the scoresheet *tell* you who won the game. There’s no prediction involved; it is strictly correlation.

      Given the confusion of “process” and “result”, and of “prediction” and “correlation”, I’m not sure Found’s models are worth much. Using one “result” (Goals) to “predict” another “result” (Wins/Losses) is pointless.

      Regarding Found’s discussion on PDO, he makes two points: a) Because Shooting% and Save% varied over time and correlate with Winning%, this proves Shot-based Metrics are meaningless and have little predictive power in comparison to Goal-based Metrics; and b) Because PDO did not regress perfectly to 100 over time, it cannot be attributed to Luck but must be explained by other factors such as individual and team skills.

      For point (a), I honestly think he’s got it backwards. It is absolutely true that PDO varies over time; sometimes even wildly. And because PDO correlates with Goals, i.e. a “result”, we can expect it to correlate accordingly with the subsequent “result” of Wins/Losses. There’s no surprise here. But there is *no explanation* for why PDO varies over time, or what teams can do to maintain a high PDO or reverse a trend of low PDO, other than “convert more of those shots into goals.”

      For point (b), while it is true to PDO does not regress perfectly to 100 over time, it can be easily seen that as sample sizes increase (in number of games played or seasons), the range of PDO values for teams shrink to a tiny sliver around 100. Single-season PDOs could get as low as 96 or as high as 103, but over multiple seasons, teams rarely fall outside 99-101. Hence I don’t understand what he’s saying about PDO “not even[ing] out over the 10 season span.”

      Now, I’m prepared to accept a more complex or nuanced explanation of PDO variances than simply Luck. Some players are better shooters, some teams generate dangerous chances more effectively, not all goalies are made equal, and teams certainly adjust and adapt to opponent tendencies over time. But Found has provided no concrete evidence to support any of these propositions.

      Found’s observation of good goaltending on Cup finalists is also silly. Are we at all surprised that goaltenders on good teams are considered for the Vezina? But how do we leap from that to “the amount of shots your team allows is largely irrelevant if you have a good goaltender”?

      Now, onto the article itself. At the end of the first section, you said, “We cannot confuse ‘results’ with ‘process’.” Bravo, and yet Rob Found does exactly that, and by extension the article does too, where it follows his train of thought. “Controlling possession is the ‘process’ the Flames have chosen to use to improve team success.” An absolutely fair statement, and the Corsi stats show that the “process” has been improved, but the “results” haven’t followed (in terms of Wins/Losses), at least in relation to the expectations justified in the following sections. But on the basis that you yourself said, “We cannot confuse ‘results’ with ‘process'”, and the signs that the paper cited clearly violate that premise, I must ask that we throw out the paper and reject all conclusions that follow from it.

      Insofar as the distinction between “process” and “results”, I agree that results are what ultimately matter in sports. But “results” cannot, by and large, be directly controlled by players and coaches. They can only control the “process”, by means of which they hope to bring about the “results”. Sometimes the “process” works; sometimes it doesn’t. Sometimes the better team loses. It’s even in the Bible: “The race is not to the swift or the battle to the strong… but whoever happens to score more goals on the night.” Should we be concerned when the results do not meet our expectations? Maybe. Is it because the coach is incompetent? Not necessarily. Especially when statistics show improvements in the process in the direction that he was specifically hired for.

      I look forward to your analysis next week into “what is causing the team to be sitting below expectations.”