Black Box: Goalies



Every week this season we’ll be providing a statistical update on the Flames, broken up into four sections.  First, an OZQoC chart to clarify the roles each player has been assigned, then the team’s success with them on the ice at even-strength, followed by special teams, and finally a look at the goaltending.

Advertisement - Continue Reading Below

The decision of which statistics to include isn’t an easy one, because a lot of the traditional goaltending statistics, like goals against average, shut-outs and wins, aren’t really worth looking at.  Goals against average is as much a function of how many shots the Flames are allowing as the goalie’s ability to stop them, and we already have save percentage to measure that.  Shut-outs are great, but your chance of winning isn’t much different when you allow just one – consistency is far more important. 

Speaking of which, wins are more a measurement of how much the Flames are scoring than how many pucks the goalies are stopping.  Instead we’ll look at two statistics: Quality Starts and Even-Strength Save Percentage.

Quality starts replace wins as a way of measuring how consistently the goalie is giving the team a chance to win.  They’re only awarded when they stop an above-average number of shots (91.2%), or when the team allows very few shots and they play well enough to win by allowing fewer than 3 goals, and stopping at least 88.5%. Here’s how this last section will look, using last year’s data as an example.

Advertisement - Continue Reading Below
Goalie           GS QS  QS%  ESSV%
Miikka Kiprusoff 71 41 57.7% .916
Henrik Karlsson  11  8 72.7% .905

Save percentage is often touted as the ultimate goaltending statistic, but it’s not without its flaws. Even with Quality Start percentage helping us measure consistency, save percentage alone can easily be skewed, especially over such small sample sizes.

We’ll look only at even-strength save percentage to avoid the effect special teams play can have, but there’s little we can do to counter the way shots are recorded differently from one city to another.

Of save percentage, just remember that a lot of transient factors are involved, and it can take years to determine a goalie’s true talent level.  Even Kiprusoff’s even-strength save percentage has varied wildly from .907 to .941 since the lock-out, and that’s with huge sample sizes.

Enough chit chat and preamble! Next week we’ll begin – although with just a single game’s worth of data.

This is hardly your last chance to offer suggestions, as we’ll consider this a living effort, and tune it throughout the season.

  • Interesting to see Karlsson’s high QC percentage. Of course, the sample size is so small, not a lot can be taken from it.

    Is there some way to place Kipper’s QS performance in context of other starters in the league last year?

  • MC Hockey

    Nice job Mr Vollman! I really like alternate ways to measure players and it works especially well with goalies. Look forward to seeing it regularly…can you do it for the Jets too???

  • Only Henrik Lundqvist and Ilya Bryzgalov have seen at least 1800 shots in each of the last three seasons.

    Lundqvist fluctuates from 55.2% to 67.1% (we talk about him in our book).
    Bryzgalov is almost always 60.9% to 61.3%, except once year at 54.0%.

    Kiprusoff is all over the place too, from 50.0% to 63.9%.

    If you look at all goalies who have faced 1800 shots in a season over the past four years, Kipper’s 2009-10 season sits 9th, but he’s also got 2 of the bottom 5 (along with Marc-Andre Fleury, Jimmy Howard and Jonathan Quick), and last year’s was definitely well below average.

    If you ask me, for someone like Kipper the bar is at 60%.

  • Reidja

    Interesting analysis. Should we read into Karlsson’s quality start percentage? How many were because the team allowed very few shots? Could it be an effect of the team playing differently in front of him (i.e. more defensively?)

    Some questions about these stats: 1) Where do the 91.2% and 88.5% thresholds come from? 2) What about the fewer than 3 goals threshold and how many shots is “very few”? 3) Finally, you often hear “the goalie has to be your best penalty killer”. Would there be value in displaying PKSV%? I wonder how well it correlates with QS% as this would include PKSV% from my read.

    • Henrik Karlsson had 8 quality starts in 11 starts, for a great 72.7%.

      We’ve been studying Quality Starts for four seasons. Two others have done the exact same thing: Peter Budaj and Thomas Greiss in 2009-10 (and with much higher save percentages, .928 and .926 vs .904 for Karlsson as a starter).

      The next year Budaj was 12 for 39 (30.8%) and Greiss was a below-average goalie in the Swedish Elite League.

      So yeah – don’t read too much into it.

      Answering the questions:
      1. 91.2% is the league average, 88.5% is just an interesting statistic threshold we observed in our research.

      2. The correlation between Quality Starts and Wins is better when you include this condition. It also seems unfair to stop 18 of 20 and have it called a non-Quality Start even though teams usually win those games.

      For 88.5% of shots, you can allow 2 goals on 18 shots. If you allow 3 goals you need to stop that 91.2% – that’s 35 shots.

      3. Studies have shown that short-handed save percentage doesn’t correlate with even-strength save percentage. It’s basically all luck.

  • Reidja


    I am a goaltender myself, and will be the first to tell you that Kiprusoff’s and Luongo’s outright SV% are more comparable than the difference between the two suggests.

    It is easy to inflate a SV% on a good team (I have done this through minor hockey and sr) and even easier to deflate it on a bad team (I have done this as well, playing very well in junior but underlying numbers dont appear to support the fact)

    The truth of the matter is, the disparity in goaltending throughout the league has become very small. This is supported by the fact Detroit competes with “thin” goaltending and the total number of quality goaltenders on the free agent market.

    I would also support the creation of a shot quality stat based on quality of competition (adjusting save pct. based on the opponents shooting pct. and win pct.)

    • Reidja


      It’s uncanny how much influence their is on goalies by the team in front of them. And most of that influence is completely unintuitive.

      Quality starts is an interesting metric, but it is so heavily swayed by “team” as well.

      You could cherry-pick last years… Oct. 28th? game vs Washington. Something like 4 or 5 PPGA… even if you look at only the even strength aspects, the team in front of the goalie with those kind of score affects, should consider it an outlier.

      I was tracking quality starts, but considering “blow-outs” as outliers if at the hands of a poor PK. I think it gives a more accurate result.

      But hey, I’m a goalie… so I’m a goaltender apologist.

      • Actually, the work being done now suggests that shot quality doesn’t have much influence at all on SV%. Not in the NHL at least and not above natural variance (randomness).

        Tom Awad’s recent inquiry says the split for luck/other factors (skill, quality of team, etc) is about 70-30. Shot quality was the least relevant factor involved:

        One thing he notes is shot counter bias in some rinks (FLA, NJ and BOS are notorious). the NYR counter is also terrible at noting distance from the net as well, so those factors can sully investigations somewhat.

        The guy finds a way to tease apart variance, skill and team effects for goalies will be a rich man I think. He could charge GM’s a ton for that info, because clearly they have no idea as well.

        • BobB

          I didn’t actually say anything about shot quality did I? I talked about team…

          I talked about even strength vs special teams.

          Although, if we think special teams shot quality doesn’t influence sv%/sh% at all….

          why did we just sign Anton Babchuk to 2mil/season?

  • Section205

    I hope it’s ok to refer to Robert’s article at Hockey Prospectus.

    Kipper’s 57.7% is 16th out of 28 goalies with minimum 41 starts. Ahead of Roloson, Lehtonen, Halak, Lundqvist, Vokoun, Howard and others.

    Goaltender GS QS QS% Tim Thomas 55 40 72.7% Roberto Luongo 60 41 68.3% Pekka Rinne 64 43 67.2% Sergei Bobrovsky 52 34 65.4% Antti Niemi 60 39 65.0% Jonathan Quick 60 39 65.0% Jonas Hiller 46 29 63.0% Marc-Andre Fleury 62 39 62.9% Corey Crawford 55 34 61.8% Ilya Bryzgalov 67 41 61.2% Carey Price 70 42 60.0% Ryan Miller 65 39 60.0% Cam Ward 74 44 59.5% Craig Anderson 49 29 59.2% Niklas Backstrom 50 29 58.0% Miikka Kiprusoff 71 41 57.7% Dwayne Roloson 54 31 57.4% Kari Lehtonen 68 39 57.4% Jaroslav Halak 57 32 56.1% Martin Brodeur 54 30 55.6% Michael Neuvirth 45 25 55.6% Henrik Lundqvist 67 37 55.2% Tomas Vokoun 57 31 54.4% Steve Mason 53 27 50.9% Jimmy Howard 63 31 49.2% Ondrej Pavelec 54 26 48.1% Brian Elliott 51 17 33.3% Nikolai Khabibulin 46 15 32.6%

    (Sorry, this table did not paste as well as I expected)

    Kipper’s 41 QS is tied for 4th overall behind Ward, Rinne and Price. Ward had 74 starts and 59.5%.

    Robert also identifies the stat Really Bad Start (RBS) and Kipper had 14 of those. I’ll bet those 14 bombs really hurt Kipper’s save% overall. Perhaps that could be seen as an argument that he was fatigued/overplayed?

    I think there is room for optimism, because Karlsson’s strong numbers over a small sample size is certainly much better than McElhinney and other backups Kipper has had.

    Kipper should be able to rest more often and reduce the probability of fatigue and reduce his number of RBS.

    Meanwwhile,if Karlsson can make about 20 starts with 65% QS (assuming he plays a few weaker opponents) then our goaltending should improve overall to get us 5 or 6 more points in the standings this year.

  • Matty Franchise Jr

    So, let’s see if I’ve got this right.

    If a team is very offensive, meaning they have the puck most of the time and they have it in the offensive zone and therefore regularily allow few shots against, or/p>If a team is very defensive, keeping the puck to the perimeter or out of the defensive zone therefor allowing few shots against,

    then a goalie only needs to maintain a .885 SV% in order to rack up the QS%.

    If this is indeed the case, then it seems that this new metric isn’t really telling us anything that EVSV% doesn’t. I’d like to see Section205’s table include EVSV% to see if this is correct.

    • You’ve got the idea right, but may need to explain why that means “this new metric isn’t telling us anything that EVSV% doesn’t.”

      Even-strength save percentage doesn’t tell you how consistently a goalie is playing. He could throw up a shut-out one night, and then a five-goal night, and his stats look the same as a goalie who lets in 2 one night and 3 the next. One of them is clearly helping their team more than the other.

      The idea behind Quality Starts is when you play well enough for your team to win 75% of the time. If you stop 18 of 20: mission accomplished.

  • In the past 4 years only 10 goalies have gone from starting over 70 games to starting under 70 – but usually due to injury!

    Ryan Miller, 2008-09, 76 GP to 59 GP, .907 to .918, 52.0% to 62.1%

    Roberto Luongo 2008-09, 73 GP to 54 GP, .917 to .920, 63.0% to 57.4%

    Nicklas Backstrom, 2009-10, 71 GP to 60 GP, .923 to .904, 66.2% to 50.0%

    Marty Turco, 2009-10, 74 GP to 53 GP, .898 to .914, 51.4% to 51.9%

    Martin Brodeur, 2008-09, 77 GP to 31 GP, .920 to .919, 71.4% to 63.3%

    Martin Brodeur, 2010-11, 77 GP to 56 GP, .916 to .902, 61.8% to 50.0%

    Jonathan Quick, 2010-11, 72 GP to 61 GP, .907 to .918, 47.2% to 63.3%

    Henrik Lundqvist, 2010-11, 73 GP to 68 GP, .921 to .923, 62.5% to 55.2%

    Evgeni Nabokov, 2008-09, 77 GP to 62 GP, .910 to .910, 66.2% to 61.3%

    Craig Anderson, 2010-11, 71 GP to 51 GP, .917 to .914, 53.5% to 59.2%

    So there’s certainly no guarantee that playing fewer games would improve Kiprusoff’s performance, but it certainly did wonders for Miller, Quick, and Turco.

  • IDK, to be honest, I don’t know how much of this stat is the goalie and how much of it is the team. IE: Where the team lets the opposition shoot on the goalie has a lot to do with how well a goalie plays. I think Clay and Rain Dogs already talked about it.

    I would pair the stat with something else, something that tracked where the opp was allowed to shoot from.

  • wawful

    I think Karlsson’s ice time last season was probably not a good sample of his ability for several reasons.

    1) Too small to give good statistics.
    2) He started against below average teams. (Average finishing league rank of opponents he started against was 19.3.)
    3) He was most frequently deployed during back-to-back games, often ones in which the opposing team was also going back-to-back.

    I would agree with others that it would nice to see a goalie stat that takes quality of opposition into account. There may be a difference between 40 shots from a first-place team when your defence is performing well and 40 shots from a last-place team when your defense has melted down. I’d also like to see something based on scoring chances and blocked shots in addition to just plain old shots.

    It might also be worth looking for stats that correlate with changes in play-style. For example, players block a lot more shots now than they did in 2005. The Flames used to clear the shot-lanes when the opposing team was shooting from the blue line because they thought Kipper would have a better chance of making the save with a clear view of the puck. Now they block up the shooting lanes as much as possible. This probably means that a save percentage calculated purely from shots in 2005/2006 is not equivalent to one in 2010! e.g. In 2005, Kipper faced a lot more shots from the blue line that were easier to stop than the average shot he sees now. This may have caused his SV% to be somewhat inflated before blocking became the preferred team tactic.

  • Section205

    Wow, lot’s of good comments and discussion that gives me some new perspectives.

    There is a lot of fluctuation for some good goaltenders. Is it really so hit and miss?

    Also interesting that 2008-09 Thomas leads with 70.4%, then his teammate Rask leads 2009-10 with 69.2%, then Thomas leads again in 2010-11 with 72.7%. Surely Boston must be doing something right, which puts their goalies in the best position to succeed. I also believe that Boston has been one of the better 5 on 5 teams the last few years, which probably means strong EVSV% too.

  • Section205

    I wish every goalie could have a breakdown of the shots against:

    Shots “expected” to be stopped (E)- 1300 (saves 1260 = .969)
    Shots “not expected” to be stopped (NE) – 500 (saves 370 = .740)

    Total 1800 shots, 1630 saves, .906SV%

    Obviously subjective, like scoring a fight. But if criteria were established for “expected” vs. “not expected” then each shot could be categorized as such.

    Then you could point to poor team play when a goalie faces a relatively high proportion of NE shots.

    The goalie can not control the quantity of “defensive breakdowns” such as breakaways, odd man rushes, giveaways, or missed defensive assignments due to sustained pressure. The goalie cannot clear the crease from screens without compromising his ability to stop the puck.

    It is probably not fair or accurate to assume that every team has the same amount/proportion of these difficult events.

    I notice that Tim Thomas and Roberto Luongo had Shootout Sv% of .526 and .538 respectively (obviously small samples)

    • BobB

      I understand what you’re getting at with this, but inevitably one would use it to compare to other goalies…. and it’d be so subjective it’d be meaningless.

      Think of shots against. A shot itself should be pretty clear, but we see with scorer bias…. over 10,000 shots that bias can make a lot of difference.

      (I’m looking at you Cam Ward and Tomas Vokoun… as prime candidates)

  • Jeff Lebowski

    These stats are very interesting akin to sabermetrics in baseball. However I can see with my eyes that Kipper is the best goalie in the league bar none.

    Kipper makes all the saves other goalies make but he also makes saves that NO one else can.

    The question with him in recent years has been stretches of poor play by him or team play. Somehow he still manages to get W’s.

    If I needed to win one game I’d pick Kipper.

  • BobB

    Occam’s razor is: as the Flames ice an older, less talented, less elite team against the NHL norm, their goaltending is going to suffer. Age, sustained speed and puck pursuit (backchecking) don’t put the defensive core/goalies in a position for success.

    Out-shooting is one thing (fenwick and corsi) and that off. zone time IS important to our goalie, but what you do after opp. transition is the MOST important thing for defense/goalie success.

    Odd-man rushes against.

    Time to set up for a shot because of light back pressure.

    Availability of passing lanes

    Sure, Iggy and Tanguay get two or three shots against team X (Luongo/Niemi/Howard), but team X (Sedins/Joe Marleau/Pavel Zettersyuk) counter attack and only need one three-pass play to bury us, while we’re coasting back, or the goalie is required to make a lateral sliding save like in the photo above^^^ vs getting hit square in the chest.

    It’s why you see the Don Cherry #541 so often: “Goalie makes a big save, other team comes back to score.” Transition shots against are the highest opp. quality/opportunity for. Shot quality models I’ve seen only track proximity (ft. from the net) to determine quality…. not defensive situations, which then renders shot quality meaningless.

    A transition two-on-one SA from 15 ft out is always going to be higher quality than a sustained-zone, strong-side, stuff play into the goalies pads from 1ft, or even more than many breakaways.