The Problem With +/-

Justin Azevedo
January 08 2012 12:45PM

 

Many stat-heads, like myself, dislike +/- as a form of talent evaluation. For me, there are four main reasons why I don't like the stat.

Goals are Random

Think about the amount of shots taken in a season. Last year, there were 74559 shots taken. Only 6721 (9.01%) actually bulged the twine. Basically, you're evaluating players based on 9% of available evidence. Then think about how many shots are deflected-which is an intentioned action designed to get the puck in the net-or otherwise get past the goalie without any control or intention from the player who took the shot-odd bounces off of opposing players, deflections off of parts of the body, et cetera that cannot be counted as intentioned actions to score.

Goals are Rare

Like I said above, only 9% of shots taken are goals. That’s a terrible sample size to draw conclusions from especially when we have so many other ways to measure the quality of a player. As an evaluator, you should want to have as large of a sample size as possible, as generally that ensures more accuracy in results and evaluation.

A Player's Situation isn't Taken Into Account

A couple of games ago, Jay Bouwmeester was a -5 despite one goal being on a 3-on-1 (where he was the 1), one goal coming on a double deflection and one goal going into an empty net. Thus, he was empirically not at fault for 3 of his 5 minuses. Would anyone really get on his case if he were -2 in a 5-goal game? Furthermore, he was only on the ice for 5 scoring chances against and he faced the toughest competition the other team had to offer in the 25+ minutes he played. Logically, players that face the other team’s best players will have more scoring chances against. More scoring chances against means more goals against in the long term. Another good example here would be the Canucks’ Manny Malhotra-he’s a large reason the Sedin twins get the advantageous starts they do, yet his -8 on the season (at time of writing) would be taken by many as evidence of poor play.

Luck isn't Factored In

I mentioned things like deflections and bounces earlier, but bad shift changes from teammates all too often lead to a minus against. That's not the fault of the player getting the minus, it's the fault of the guy now sitting on the bench-but the guy on the bench doesn't get a mark against in the scorer's book. What about a goal scored one second after a Power Play? Even though the teams are technically at even strength, the numbers in the zone are probably still 5 versus 4. The penalty killers repelled an attack for 2 minutes and then they get a minus a second after the penalty ends, while still at a disadvantage? Absurd.

Now, I won’t completely dismiss +/- as I’m sure it serves some purpose-but it has to be taken in context with other evidence, just like every other stat. It’s inherently flawed, which limits its effectiveness. Unfortunately, people get hung up on the number and as a result draw incorrect conclusions about players.

A9d138d0e612f28cd46f9b7057ed715d
Justin is a 23-year-old Flames fan who also happens to be pursuing a double major at the University of Calgary. He has played hockey at high levels, enjoys wearing shorts and tends to drink far too much Grasshopper. Please don't hate him.
Avatar
#1 Vintage Flame
January 08 2012, 01:01PM
Trash it!
0
trashes
Props
0
props

Good stuff Justin..

Personally I have no use for the +/- stat. Like you said, it probably serves some purpose, but for the most part it gives the critics and the TV commentators something to pick on.. "Ah, he was a -5 tonight, he had a HORRIBLE game!"

Luck factor aside, I think the point that you make about "situation" not being factored in, is your most compelling argument.

Avatar
#2 Kevin R
January 08 2012, 01:48PM
Trash it!
0
trashes
Props
0
props

Good read & agree. +-'s only add to further the direction of the analysis. Look at Iggys 500th. How could you fault any of those Minni players & make an opinion of talent using a +- measure that includes goals like that. The only thing is, in a 9-0 game where a dman plays 25 minutes & has a 0 +- like Brodie did, well, I just find that amazing.

Avatar
#3 RKD
January 08 2012, 02:28PM
Trash it!
0
trashes
Props
0
props

Great article, very true. +/- is somewhat superficial. Guys like Brad Richards and Jokinen who are over -90 in their career doesn't mean they liable for all those goals. Really, it means they are on the ice a 1/3rd of the game and unluckily on the ice more often than not when a goal is scored.

9% is a very small sample size, Jay-Bo and Butler face the top lines every game night and we know sometimes Jay-Bo is near the 30 minute mark.

Avatar
#4 SmellOfVictory
January 08 2012, 02:56PM
Trash it!
0
trashes
Props
0
props
RKD wrote:

Great article, very true. +/- is somewhat superficial. Guys like Brad Richards and Jokinen who are over -90 in their career doesn't mean they liable for all those goals. Really, it means they are on the ice a 1/3rd of the game and unluckily on the ice more often than not when a goal is scored.

9% is a very small sample size, Jay-Bo and Butler face the top lines every game night and we know sometimes Jay-Bo is near the 30 minute mark.

I think the major reason guys like Jokinen and Richards are career minus players is that they do a tremendous proportion of their offensive damage on the PP. Neither has been known as a particularly good defensive player, either (Jokinen recently having changed that to some extent at the behest of Calgary's management).

Avatar
#5 Ryan Lambert
January 08 2012, 03:06PM
Trash it!
0
trashes
Props
0
props

This makes me feel so much better about Butler's night in Boston!!!

Avatar
#6 FireOnIce
January 08 2012, 03:11PM
Trash it!
0
trashes
Props
0
props

This doesn't excuse Butler for his -7 the other night. I find it hard to believe that other players skated off the ice right before the Bruins scored, every single time.

In general, however, I agree with your arguments Justin. Players can just skate off the ice in a poor line change and the players who come on get the negative credit for their teammates' untimely decisions. As well, the +/- simply does nothing for bad bounces and fluky shots.

Good article!

Avatar
#7 Derzie
January 08 2012, 04:39PM
Trash it!
0
trashes
Props
0
props

Plus/Minus is a nice aggregate stat. Not to compare players on different teams but to assess the general contribution to the bottom line of a player within a team. The numbers that mean something are the outlyers. Abnormally high or low plus/minus is cause to look deeper at a player. It tells you that there is something right or wrong going on. What that something is, requires further analysis of more accurate stats. I like the stat a lot as a way to catch my attention to look deeper at players other stats.

Avatar
#8 Emir
January 08 2012, 05:47PM
Trash it!
0
trashes
Props
0
props

Sorry Justin I am not into your analysis. Main reason being lucky bounces and timely shift changes are a two way street. So for all the times a guy got on the ice on a shift changed and got a goal against, im sure over the large sample size you mention it surely must mean there is the opposite situation which means it most likely zeros out I've the course of a season. So im not buying your argument.

As for liking plus minus, I do like it. It does say something about ES play, but compared to other metrics it is lacking. Plus minus needs a large sample size to be relevant whereas other measurement methods are more effective in the short term. It seems like you dislike the short term relevance of plus minus more than anything id say, and id agree with you there.

Avatar
#9 Rain Dogs
January 08 2012, 05:59PM
Trash it!
0
trashes
Props
0
props

I don't disagree with your assertion that +/- is a somewhat flawed stat, but I'm curious how we'd willingly discount all of those factors to the point where we'd conclude the above and not have the same conclusions about say... sv%

Each of your points is magnified on goalies even further than on defenders.

We don't count the actual number of saves as a value-added stat (in fact Justin you recently said "if a goaltender is letting in the same percentage of shots in when he makes 100 saves as when he makes 500 it really doesn't matter how many pucks he's saving") ... it's only "error" rate (sv%)

We count "errors" for goalies, not "successes". And we've established above that Goals Aganist are: Random, Rare, Situation based, Luck influenced and subject to the compounding mistakes made in front of the goalie. Breakaway? Anyone?

Re-read this whole article (and comments) with "goalie" substituted for "player", "sv%" substituted for "+/-", "minus" for "goale against".... well, you get the idea: player reference for goalie reference.

Do we have the same desire to apologize for our goalies as we do our defenders?

Life's not fair, just like many stats and "situation" is what goalies look at as "shot quality".

Avatar
#11 Rain Dogs
January 09 2012, 09:00AM
Trash it!
0
trashes
Props
0
props

@Justin Azevedo

that's exactly correct. So, let's backcast from an indeterminable future when we do have a better stat than sv%. Think of how silly we will have been for having such on-going arguments over the certainty of the quality of goalie X vs other goalies on other teams using the "+/-" of goalie stats... and then further diluting it with a GVT formula which certainly isn't going to correct error.

In the current, we need as much additional context as possible:

Shots against, Comparison vs Backups, largest sample size possible, etc.

Re: Shot Quality. It's "situation", like your 3 on 1 example vs JBo.

Do we think Calgary Goalies would have the same sv% if they played only vs Boston, Philly or Vancouver all year? Vs if they played only against NYI, Clb or Minny?

No, they wouldn't. A disparity in how good two teams are is going to affect both "+/-" and "sv%" which is another reason why we shouldn't compare goalies team-to-team for sv% as if all teams are equal. Tim Thomas is not consistently facing the same 'situations' (shot quality against) that Steve Mason is. Now, Thomas may well do better than Mason on Columbus, and Mason worse than Thomas on Boston, but what would be the difference? We can only guess. I can tell you Thomas wouldn't be consistently putting up .931ev%+ on CLB and having unsustainable seasons of .947ev%.

Why can I tell you that? Because Tukka Rask's evsv% is even better than Thomas', just with a smaller career SA sample size. And all the other backups before him were .922ev%+. Now, it's possible, but not probable, that Boston just happens to possess the two best goalies in the NHL, and have possessed a vast selection of the highest performers (like Andrew Raycroft :$) but it's more likely that Boston is just... that... good... today, and was en route a few years back.

Avatar
#12 SmellOfVictory
January 09 2012, 09:04AM
Trash it!
0
trashes
Props
0
props
Rain Dogs wrote:

I don't disagree with your assertion that +/- is a somewhat flawed stat, but I'm curious how we'd willingly discount all of those factors to the point where we'd conclude the above and not have the same conclusions about say... sv%

Each of your points is magnified on goalies even further than on defenders.

We don't count the actual number of saves as a value-added stat (in fact Justin you recently said "if a goaltender is letting in the same percentage of shots in when he makes 100 saves as when he makes 500 it really doesn't matter how many pucks he's saving") ... it's only "error" rate (sv%)

We count "errors" for goalies, not "successes". And we've established above that Goals Aganist are: Random, Rare, Situation based, Luck influenced and subject to the compounding mistakes made in front of the goalie. Breakaway? Anyone?

Re-read this whole article (and comments) with "goalie" substituted for "player", "sv%" substituted for "+/-", "minus" for "goale against".... well, you get the idea: player reference for goalie reference.

Do we have the same desire to apologize for our goalies as we do our defenders?

Life's not fair, just like many stats and "situation" is what goalies look at as "shot quality".

I don't agree with the comparison of a skater's circumstances as being akin to shot quality. Since the lockout, and likely anytime beyond the mid 90s, shot quality has not differed enough between teams to be statistically significant. Combine that with the fact that very few goaltenders stay with the same team their entire career, and we can say with some confidence that most goaltenders who have played a substantial number of games have faced an approximately equal proportion of scoring chances within the number of shots that they have faced.

Contrast this with skaters, where you have players such as the Sedins who, for the past three years, been treated to a 3:1 ratio of offensive zone starts to defensive zone starts; or, to use another Canuck, Manny Malholtra, who's been given almost the exact opposite. There are players who are employed in certain roles, and those who gain a reputation as a checker will be employed as a checker, which means their stat line will suffer significantly.

Avatar
#13 Rain Dogs
January 09 2012, 09:18AM
Trash it!
0
trashes
Props
0
props

@SmellOfVictory

see comment 11.

Avatar
#14 Domebeers.com
January 09 2012, 10:10AM
Trash it!
0
trashes
Props
0
props

'but it has to be taken in context with other evidence, just like every other stat.'

eagerly awaiting the posts on the problems with corsi, fenwick, etc.

Avatar
#15 jeremywilhelm
January 09 2012, 10:29AM
Trash it!
0
trashes
Props
0
props

Totally agree

Avatar
#16 Emir
January 09 2012, 11:37AM
Trash it!
0
trashes
Props
0
props

@Justin Azevedo

Goals are Random: No goals are not random at all actually. If goals were random then we would not see the disparity between the best goal scorers in the league and the worst in the league. There is nothing random about it at all. Good players make good shots which in turn lead to goals and bad players can't make the same quality of shots which in turn leads to no goals. Nothing random about that.

Goals are Rare: Sure 9% of shots went in last year, but that sample size is 100% of all the goals. Therefore you can only argue that shots is a bad indicator of goals due to the 9% conversion rate. Your point makes no statement about plus/minus. Therefore this point is irrelevant altogether.

A Players Situation isn't taken into account: Same point as I made about luck. Yup Jbo has been on the ice for some 3 on 1's against, but he has also been on the ice for 3 on 1's for. I think this is another two way street, and surely over the course of the year it will level off. However, on the 30th place team in the league you will see more of a minus and on the 1st place team in the league you will see more of a plus. Once again, goes to suggest plus/minus as a way to rank players on the same team but not across teams.

I get where your coming from Justin, but the evidence provided does not support your conclusion.

Avatar
#17 SmellOfVictory
January 09 2012, 12:50PM
Trash it!
0
trashes
Props
0
props
Rain Dogs wrote:

see comment 11.

Compelling example, but it's only one example, and I do accept the notion that PK sv% is more variable then ES sv%, so that must be taken into account.

Counter examples (using overall sv% because I'm lazy):

Roberto Luongo: going from Florida (one of the worst teams in the league) to Vancouver (one of the best teams in the league) had almost no change in overall sv%; he actually had his best year in terms of sv% in Florida.

Kari Lehtonen: going from Atlanta to Dallas, almost no change in overall sv%.

Mathieu Garon: 5 different teams, almost no change in overall sv%.

Or you can look at teams that switch goalies partway through a season/play a 1A/1B sequence. Last season for the Hawks, Turco had a 0.897 sv% while Crawford ended up with a .917 sv%. Niemi had a .920 sv% while Nittymaki had a 0.896 sv%.

I'd say it's significantly more likely that Boston has had a short run of excellent goaltenders than it is that teams have the ability, with the relative parity of the current league, to significantly affect shot quality over the long term.

Avatar
#18 SmellOfVictory
January 09 2012, 12:56PM
Trash it!
0
trashes
Props
0
props
Emir wrote:

Goals are Random: No goals are not random at all actually. If goals were random then we would not see the disparity between the best goal scorers in the league and the worst in the league. There is nothing random about it at all. Good players make good shots which in turn lead to goals and bad players can't make the same quality of shots which in turn leads to no goals. Nothing random about that.

Goals are Rare: Sure 9% of shots went in last year, but that sample size is 100% of all the goals. Therefore you can only argue that shots is a bad indicator of goals due to the 9% conversion rate. Your point makes no statement about plus/minus. Therefore this point is irrelevant altogether.

A Players Situation isn't taken into account: Same point as I made about luck. Yup Jbo has been on the ice for some 3 on 1's against, but he has also been on the ice for 3 on 1's for. I think this is another two way street, and surely over the course of the year it will level off. However, on the 30th place team in the league you will see more of a minus and on the 1st place team in the league you will see more of a plus. Once again, goes to suggest plus/minus as a way to rank players on the same team but not across teams.

I get where your coming from Justin, but the evidence provided does not support your conclusion.

Individual goal events cannot be predicted, therefore they are random. That's all that randomness is. If goals weren't random you wouldn't see guys like Iginla or Ovechkin miss a wide open net, but sometimes they do; it just happens less often to them than to bad shooters. They're not perfectly random by any means, but there is a strong element of 'luck' involved. Luck being a combination of everything from the structural integrity of a player's stick to how focused the opposing goaltender is at that exact moment. That's why +/- is a terrible stat; it focuses on something that has far too many factors out of a player's control, and rates them based on those factors.

Avatar
#19 ChinookArch
January 09 2012, 01:18PM
Trash it!
0
trashes
Props
0
props

We talking stats!

Good article Justin

I can't take anymore argument / counter- arguement posts for various imperfections of stats.

Avatar
#20 OldTimeHockeyFan
January 09 2012, 11:58PM
Trash it!
0
trashes
Props
0
props

Butler is the worst defenseman we've had since Kari Eloranta.

Avatar
#21 Rain Dogs
January 10 2012, 11:24AM
Trash it!
0
trashes
Props
0
props

@SmellOfVictory

Yeah for sure. My point isn't that there isn't going to be variability, or that there won't be outliers, or that sample size shouldn't be considered. Or that FLA gives out saves like candy (right Luongo, Vokoun, Anderson, Clemmensen, and Theodore?)

It's context. Justin is advocating for context with "+/-" because of unfair and unpredictable team circumstances (my words) and I somewhat agree. However, if that is true for the defender, then all the same would be true for the goalie as well if you agree with the "+/-" argument. A Goalie is just as "empirically not at fault" for deflections and bad lines changes etc.

A goalies job is to save the puck as much as a defenders job is to ????

Why do we invest $$$$ in defenders? Solely offence? Defenders defend the net (or in other words) reduce the chances of goal scored against. ie. limit shots with high "quality" or percent chance or going in the net. (like a 3 on 1).

Agreed? Then Shot quality/"situations" exist (or whatever semantic fits) but we don't know to what degree.

Therefore, one way of establishing context for goalies (and not assuming all teams, defenses and situations are equal) is to take all the backup data and compare the evsv% vs the "much better?" number 1 goalie. We expect to see a distinction because the #1 is better: Number 1 is 930evsv% and all backups are .915evsv% we can probably assume that #1 goalie is in fact better than #2 thru 8 goalies B(replacement level)

So, what might it mean if goalie 1 is .924evsv% over 7000sa and goalies 2-8 are .924 over 3000?

goalie 1 is much higher than league average but no better than replacement, which is unusually high = scorer bias?, team factors? 8 consistently super goalies (that's nice, but unlikely)

vs

What if goalie 1 is .925evsv% over 10000evsa and goalies 2-8 are .902evsv% over 3000?

Goalie 1 is MUCH better than backups (replacement), but replacement is low vs league = 6+ backups all suck (unlikely)?, Goalie 1 is very, very good?, Team factors create lower replacement level (team sucks)? Team inexplicably plays consistently worse over 8 seasons with backups in net (unlikely)?

Context.

Comments are closed for this article.