logo

The case for firing Glen Gulutzan, Part 2: Process vs. results

alt
Photo credit:Gary A. Vasquez / USA Today Sports
Skylardog
6 years ago
When offered the opportunity to write an article for FlamesNation, I told Ari that the research may take me down a rabbit hole or two before it was all over. This is one of the rabbit holes I found myself buried in.
(Here is Part 1.)

Why using results and goal differential is how a team should be evaluated

An early Google search produced an article written by Rob Found from Edmonton, and posted online at The Sports Journal, “Goal-based Metrics Better Than Shot-based Metrics at Predicting Hockey Success.” It is a statistical evaluation of 10 years of NHL data to compare goal-based versus shot-based metrics to see what methods are best at determining team success in the NHL. It was written in 2016.
Here is a summary: 14 hockey statistical models were used in the study, 10 based on shots (Corsi) and four based on goals. Some of the models used a single statistic (i.e. shots for), while others were “composite” in that the model used two or more combined statistics (i.e. goal differential).
Based on his research, Found concluded that goal-based models outperform shot-based models in predicting team success, both in terms of team results and individual contributions to a team’s success, and that defence-based models are better than offence-based models (goals allowed is superior to goals for and shots allowed is superior to shots for). As for individual contributions to team success, only forward contributions were statistically significant, and only when using goal-based models. Contributions by defenceman were not statistically significant using goal- or shot-based models. 
I took the statistics from the 2016-17 season and applied them to the standings. While a simple evaluation in comparison to the research done for the article, the results are very good at showing how effective certain stats are at predicting where a team will finish in the standings.
2016-17 NHL standings relationship to shot- and goal-based statistics (data from Natural Stat Trick)
StandingsTeamIn PlayoffsPointsCF%CF% RankShot DifferentialSD RankGoal DifferentialGD RankPDOPDO Rank
1CapitalsYes11851.666th+2134th+841st1.0271st
2PenguinsYes11149.8016th+809th+494th1.0155th
3BlackhawksYes10949.8915th-6622nd+287th1.0136th
4Blue JacketsYes10849.5718th+5212th+543rd1.0202nd
5WildYes10649.7917th+6311th+572nd1.0203rd
6DucksYes10549.5719th-418th+239th1.0107th
7CanadiensYes10351.368th+3115th+258th1.0099th
8OilersYes10350.4411th+1317th+366th1.0108th
9RangersYes10249.0621st-2719th+375th1.0164th
10BluesYes9949.5020th+416th+1713th1.00711th
11SharksYes9951.387th+1875th+1911th1.00115th
12SenatorsYes9848.5624th-317th-419th0.99818th
13BruinsYes9554.482nd+5241st+2310th0.99021st
14Maple LeafsYes9550.1114th-5621st+1614th1.00810th
15FlamesYes9450.3912th+3314th+316th1.00017th
16PredatorsYes9451.109th+858th+1812th1.00414th
17Islanders9447.5327th-13624th+117th1.00512th
18Lightning9451.0510th-3820th+615th1.00413th
19Flyers8852.154th+2513rd-1922nd0.98327th
20Jets8748.9622nd-10023rd-920th1.00016th
21Hurricanes8752.293rd+1736th-1821st0.98625th
22Kings8654.771st+4342nd-218th0.98328th
23Panthers8151.695th+4813th-2623rd0.98922nd
24Stars7950.3413th+6410th-3825th0.98229th
25Red Wings7948.5225th-17525th-4626th0.98823rd
26Sabres7847.1928th-32428th-3224th0.99819th
27Devils7047.0729th-29426th-6127th0.98526th
28Coyotes7044.9330th-51630th-6729th0.99120th
29Canucks6948.1726th-33529th-6328th0.98624th
30Avalanche4848.6523rd-29927th-11130th0.96630th
Playoff Prediction Success60.00%66.67%93.33%80.00%
Average Error in Predicting Standings8.47.131.83.53
The results speak for themselves. Corsi is horrible at predicting team success. Shot differential is better, but still ineffective. On the other hand, goal differential correctly predicted whether a team was in or out of the playoffs for 28 of 30 teams. PDO was also very effective. If you believe the Corsi proponents, this would suggest that playoff appearances are nothing more than luck.
But what is most impressive is that both PDO and goal differential determined who the top nine teams were, although not necessarily in exact finishing order. Goal differential also predicted the worst eight teams, although not perfectly in order.
In the case of PDO, Found’s evaluation determined that PDO did not even out over 10 years of NHL data, as proponents say it will. He accounted this to some teams taking better shots when compared to other teams, teams allowing better shots than others, and teams having better goaltending than others. PDO is not luck.
On the subject of goaltending, Found pointed out that of the 40 teams to reach the Stanley Cup finals in the 20 years prior to his study, 28 teams had a goalie that received Vezina Trophy votes in the season they appeared in the final, while the other 12 had goaltenders good enough to earn Vezina votes in other seasons. In Smith we trust.
Corsi was so far from being accurate, that four of the top five CF% teams missed the playoffs, and ranked Boston as second despite finishing 13th in the standings. Boston was only one point from missing the playoffs: hardly an endorsement for Corsi being a good tool.
Found best evaluated the results of his findings as such: “There are meaningless shots, but no meaningless goals…”
My point in going down this rabbit hole? Shot-based statistics and goal-based statistics are tools to measure how the “process” is working in achieving team success. But at the end of the day, results come down to wins, losses, and overtime and shootout losses. In other words it comes down to points, and more specifically, points in relation to the teams in the division, and conference.
Increasing shots for and decreasing shots against – controlling possession – is the “process” the Calgary Flames have chosen to use to improve team success. We cannot confuse “results” with “process.”

The Gulutzan Flames: Results and significant statistics

In the NHL, just under 12% of all regular season games go into overtime. Based on this, the average points per team is just under 92. Setting the bar correctly, that means that evaluating teams by the correct standard is to relate everything to 92 points, not 82 and a .500 record.
The 2016-17 Flames finished the season with 94 points, 12 games over .500, but only two games over the average. The 2017-18 Flames are on pace for 97 points, five games over average. The good news: they are better than average.
The playoffs are played and won at 5v5 and on special teams. Relying on 3v3 overtime is not a good strategy, but it has become the norm for Calgary this year. The Flames have been to overtime 17 times this season, second most in the NHL behind Arizona, and in 30% of all games played. They are truly playing for a tie.
In terms of regulation wins and losses in the regular season, the Flames were 32-33 last season. They were the only team to make the playoffs that was below .500 in that category; no other playoff team was fewer than six games over .500. Two Eastern teams missed the playoffs despite being above .500 in regulation.
This season, the Flames are 20-19 in regulation wins and losses, 16th in the NHL. The Gulutzan Era Flames are just above .500 in regulation, and when you include the playoffs, they are three games below .500.
This season, in terms of regulation wins, the Flames have 20, tied for 18th. Vancouver and Chicago are only one win back at 19. They are 11th in regulation wins in the Western Conference.
Even the worst team in the NHL last season, Colorado, is 24-21, three games over .500 and four regulation wins better.

Setting expectations: Roster upgrades

The Flames were upgraded in the summer. On defense, replacing Dennis Wideman and Deryk Engelland with Travis Hamonic and a re-signed Michael Stone made the defensive core one of the strongest on paper. Up front, the core remained intact, with a few minor changes to the bottom six.
The biggest improvement has been the goaltending. Mike Smith was the biggest question entering the season, but he has done better than anyone could have hoped. The backup question has also been answered: although it took some time to figure things out, David Rittich has save and goals against numbers that are better than Smith’s.
There is little argument that Smith has been an upgrade over Brian Elliott and Chad Johnson. As for the rest of the roster, you may disagree that there have been improvements. What we think does not matter: Brad Treliving believes the roster has been upgraded, so much so he leveraged the future to bring in players he believes can take this team to a better finish in the regular season, and on a playoff run in the spring.

Setting expectations: Points

Last season, the Flames finished 45-30-5 for 94 points. They came out of the gate very slow then as they adjusted to Gulutzan’s system. Roughly one-quarter of the season in, they were 8-12-1, and were beginning to show signs of improvement. From that point on, the team ran up a 37-21-3 record, 77 points in 61 games, a 0.631 win percentage. Once the kinks were ironed out, this is the level of play last year’s team was capable of. It translates into a 104-point season.
There is no excuse for a slow start this year; of the core, only Hamonic is new to the system. With the offseason moves, 94 points would be a step backwards. At the very least, this year’s team should be capable of matching the level of play they achieved last season, or 104 points.

Setting expectations: Pacific Division standings

If you look at the division, most of the other teams are in a period of decline or rebuild mode. Whether it has been making cap room by moving out players, teams starting the season with key players on injured reserve, or aging players on the decline, only the Flames made significant upgrades. For the most part, things are playing out very close to what should have been expected.
2016-17RankProjected
2017-18
Only Los Angeles has shown significant improvement of those in the hunt for a playoff spot. A new coach and improved play following a disappointing season does make this a predictable result. It should not be enough to catch this edition of the Flames.
If we have set the expectations correctly at 104 points, and a 100-point season is what even Gulutzan predicted was possible, then the Flames should be leading the division. To be third of the perceived contenders, and in a dogfight for their playoff lives, is disappointing. A 97-point pace is a step backwards.
And then there was Vegas. That they are in the conversation at all for a playoff spot is the biggest surprise of the year; to be leading the Western Conference is just insane. It has also put a playoff spot in jeopardy for the Flames.
Vegas is on pace for 113 points, a testament to what is possible with solid coaching. At worst, the Flames should be in second, on pace for 104 points and home ice advantage in the first round of the playoffs. But that was last years team’s potential. Improvements were made. This team should be battling with Vegas for first in the Pacific.
The team is underperforming. Next week, I will look at what is causing the team to be sitting below expectations.
Team2016-17 Points
1Anaheim10594
2Edmonton10375
3San Jose99100
4Calgary9497
5Los Angeles8697
6Arizona7056
7Vancouver6973

Check out these posts...