CHL Goalies, Decision Making, and Hockey Canada's Failings

Despite a ton of hype and a high draft selection, Zach Fucale is an extremely risky prospect.

Soon after Hockey Canada announced the pre-tournament rosters for this year's World Juniors, I was pretty critical of their selections, most notably the decision to bring Zachary Fucale and Jake Paterson in goal over Edmonton's Tristan Jarry and Tri-City's Eric Comrie. I wrote this article on why I thought this was a massive, inexcusable mistake. If you haven't read that article, basically my argument was this: a goalie's ability to stop the puck is the absolute core of their job, so you want to bring the goalies that are the best at stopping the puck. Relative to their respective junior leagues, Eric Comrie and Tristan Jarry proved to be elite puck stoppers, while Fucale and Paterson were just barely above average. It's Hockey Canada's job to supply the World U-20 team with elite talent, and they bypassed the elite goaltending talent that Canada has at its disposal in favour of two average guys for essentially no good reason.

I received a lot of feedback on the post, most of it positive, but some understandably critical. However, most of the feedback I saw wasn't addressing the Fucale/Paterson vs. Comrie/Jarry debate that the whole piece centred around, but the small add-on at the end of the article where I implied that both Fucale and Paterson had extremely poor chances at becoming regular NHL goalies, let alone star starters. Wading through the usual "watch the games, nerd!" bullshit, a couple of consistent criticisms emerged:

1. The intangible, cerebral aspects of goaltending were too easily dismissed.
2. The WHL, OHL and QMJHL may play very different styles, so the effects of those styles on save percentage made comparison across leagues difficult.

I'll address the first point before delving into the massive pile of data I looked at to come to a conclusion about the second. Eric Tulsky did a really good job here showing that "clutch playoff performance" isn't really a consistent, magical, unexpected thing at the NHL level. Basically, any deviation between "clutch" to "choker" can be explained by simple random variance over small sample sizes. For example, "clutch" Zach Fucale had an abysmal 0.884 save% over the WJCs and pre-tournament games, after a disappointing 5-1 loss to Finland in the semi-finals. Did Fucale lose his ability to win big games after performing poorly in these ones? Nope, sometimes guys have tough stretches. That's just the way goaltending works.

As a result, it's far more important to analyze a goaltender's body of work as a whole in an effort to determine how many pucks you can reasonably expect a guy to stop. A goalie who has stopped more pucks in the past is a better bet to stop more pucks in the future and is therefore a better bet to win you more games down the road - it's pretty simple. Therefore, bringing a goalie with a consistently higher save percentage is a smarter bet than a guy with an average one. Jake Paterson has a career 0.902 save%, and Zach Fucale sports a career save% of 0.900. How do these numbers compare to their peers and are they good bets to become NHL players? I looked at that question in detail here:

Purpose of Study & Hypothesis

As I indicated above, the purpose of this analysis was not only to determine whether Hockey Canada's goaltending choices for the World Juniors were reasonable bets to perform at a comparable level to their WHL counterparts that were left off the team, but to also see if these players were likely to become NHL players based on the measurable attributes of their peers that made the NHL. 

My hypothesis was largely the same as the one I held for the series of defensemen posts I wrote back at the NHL Entry Draft (you can read those here, here and here): I believe that the NHL is really, really good super-concentrated collection of the very best hockey talent on the planet. As a result, you have to be an elite performer in your pre-NHL days to even make the league, and this elite performance is reflected one way or another on the scoresheet. As this applies to goalies, we should expect the guys who make the NHL as regulars to stop pucks at an elite level in junior.

Procedure

First, I defined a "regular NHL goalie" as a goalie who has:

• 20 or more games of NHL experience.
• Played in a minimum of 10% of available games played between their draft+3 and draft+13 seasons (basically, 8 games per year for 10 years).
• Appeared in a minimum of 8 or more games 5 separate seasons if they are no longer in the NHL.

I think this criteria is pretty generous, since it basically includes guys that were rarely-used journeyman backups. Then, data was collected on every goalie who faced at least 1000 shots in the CHL, and their career save percentages were determined. Unfortunately, a lack of data limited the analysis to looking at goalies drafted only as far back as the 1998 NHL entry draft, so Andrew Raycroft was the earliest available NHL goalie who had complete and accurate data to work with.

(Side note: in the next couple of days, I'm going to write a post outlining how and why the data was collected the way it was, but that's much less interesting for most of you I'm sure. However, Josh Weissbock's work was absolutely invaluable in compiling this data, and I couldn't have put this post together without him, so a big thank you to you Josh #statthuglife)

After this was done, I compared each future NHL goalie's CHL performance with all of the goalies that played in his league and the CHL over the time frame when he was active. For example, Carey Price played four seasons in the WHL between 2003-04 and 2006-07, so his career save percentage was first compared to all WHL goalies who were active sometime within one or more of the seasons between and including 2003-04 and 2006-07, and then to all CHL goalies who played during that same period. To level the playing field and provide an accurate comparison across all leagues and all years, a standard score (z-score) was calculated for each goalie, then converted into a percentile for ease of comparison. If a goalie scored 85%, it means that he was better than 85% of his peers. Here are the results:

OHL

Save Percentage of all OHL goalies drafted since 1998 that made the NHL. Jake Paterson included for comparison.

Percentile of OHL goaltender career save percentage.

Percentile difference (percentile minus 50%) from OHL average.

The bottom two graphs are essentially the same, with the middle one organized in chronological order left to right, and the second demonstrating the difference between goalies studied and the average save percentage of their peers. This format is the same for the other leagues as well.

You'll notice that only Michael Leighton performed worse compared to his league peers than Jake Paterson has, and only Alex Auld is within + or - 10%. Given that there were a whole ton of goalies that performed at about the same slightly above average level as Auld and Paterson (~85 in the sample I studied), this would seem to indicate that Alex Auld is the huge outlying data point here, and probably the absolute best-case scenario for Jake Paterson in terms of NHL success.

QMJHL

The Q is interesting, because average goaltending is just currently so much worse than the other two Canadian junior leagues:

As shown by the graph, Quebec goaltending has, on average, been in the tank since roughly 2003-2004. This means that goalies with lower save percentages than in the OHL would score higher in percentile since their competition is so poor. However, if the QMJHL talent still holds its own relative to the rest of the CHL we can theoretically discount at least some of the proposed systemic effects that could depress save percentage. As it turns out, M-A Fleury and Corey Crawford (and by extension, Ondrej Pavelec and Jaroslav Halak) are still elite performers when comparing them to the rest of the CHL, and Jonathan Bernier is strong as well. If there were legitimate style-of-play reasons why QMJHL goalies were seeing lower save percentages, we would expect to see this effect across the board on elite goalies too. Since this is not the case, I'm led to believe that there isn't a quantifiable negative effect that's exclusive to the QMJHL.

Still, some decent talent has come out of the QMJHL in recent years. I also included the two European import goalies that made the NHL out of the Q in just the career save% graph as a small comparison:

Save Percentage of all QMJHL goalies drafted since 1998 that made the NHL. Ondrej Pavelec, Jaroslav Halak and Zach Fucale included for comparison.

Percentile of QMJHL goaltender career save percentage.

Percentile difference (percentile minus 50%) from QMJHL average.

Pavelec and Halak scored similar to Crawford and Fleury, which is to say they were elite QMJHL goalies. Zach Fucale performed at a level similar to but slightly below Kevin Poulin and Pascal Leclaire, neither of whom are or were anywhere close to a legitimate starting goalie, let alone an NHL star. It's also worth noting that Poulin and Leclaire, like Alex Auld in the OHL, have beat monumental odds to qualify as a "legitimate NHL goalie." 85 goalies scored similarly across the CHL, and only these three made the NHL in any capacity. That's a success rate of about 3.5%. Fucale scored closer to league average than Leclaire and Poulin did too, and only Poulin's career save% numbers were significantly inflated by a really strong draft+2 season in the QMJHL.

WHL

The strongest goalies drafted in the past 10 years have all arguably come out of the WHL, which is interesting because the 'dub was awful back in 1998. Then out of nowhere, goaltending improved dramatically and six legitimate NHL starting goalies emerged:

Save Percentage of all WHL goalies drafted since 1998 that made the NHL. Tristan Jarry and Eric Comrie included for comparison.

Percentile of WHL goaltender career save percentage.

Percentile difference (percentile minus 50%) from WHL average.

4 of the 6 goalies that made the NHL out of the WHL all scored in the 90th percentile or better, and a 5th (James Reimer) was just shy. Braden Holtby a bit of a surprise and an outlier for sure, but even his .905 career save% puts him more than 0.5 standard deviations above the mean in the entire data set. While he was better than average among his peers, he also played in a super-competitive WHL and was compared to guys like Price, Reimer and Dubnyk. 

The two WHL guys that were passed over for the opportunity to represent Canada also score exceptionally well here. Jarry's career save percentage is above 94% of his peers, while Eric Comrie is better than 90% of other WHLers. This also comes at a period where WHL goaltending is as strong as it's been since the 04-05 lockout period, so they're not dominating a bunch of scrubs. Just as it did before the tournament, Hockey Canada's decision to bypass these two entirely completely mystifies me.

CHL

Now that we've seen each future NHL goalie against their league in their era, here is each future NHL goalie against the whole CHL in their era:

Percentile of CHL goaltender career save percentage.

Percentile difference (percentile minus 50%) from CHL average.

While Tristan Jarry and Eric Comrie compare admirably to a group comprised mostly of NHL starters including Carey Price, Craig Anderson, Marc-Andre Fleury, and James Reimer, Paterson and Fucale most closely compare to guys like Poulin, Leclaire, and Auld. Michael Leighton still remains a total mystery but then again, it's Michael Leighton who's most generously described as a "replacement-level journeyman backup" at best, so it's not as if a legitimate NHL starter slipped through the cracks. It's still bizarre though.

Discussion

I would be remiss if I didn't hammer this point home at least once in this article: CHL success is no guarantee of NHL success. Even if a given goalie was an elite performer in junior, there was no guarantee that they'd make the NHL as even guys who posted career save percentages in the 84th percentile or above (more than 1 standard deviation above the mean) only had about a 20% chance of becoming an NHL'er, as shown by this table:

69th percentile represents ~0.5 standard deviations

But here's the takeaway from that: a draft pick is a bet. When drafting a player, you're betting that the player you pick will outperform any other player you have the opportunity to spend that pick on. A goalie who has an elite save percentage such as Tristan Jarry or Eric Comrie is historically 5-6 times more likely to develop into an NHL player than a goalie who performs similarly to Jake Paterson or Zach Fucale, so using an early draft pick on a goalie with mediocre numbers is generally a terrible, terrible bet.

And if you look at what is happening to NHL goaltending, the odds of a non-elite junior making the NHL appear to only be getting worse. You'll notice that the majority of guys who made the NHL with non-elite CHL save percentages did so about a decade ago, and basically only Braden Holtby has done so since. I believe this could be due to two related factors: the improvement of NHL goaltending as a whole, and the increasing presence of European netminders in the NHL. Here are those trends in graph form:

NHL ES Sv% by year since 2000.

Courtesy of www.QuantHockey.com.

This growth in NHL average save percentage seems to correlate with the explosion of the presence of Swedish and Finnish goaltenders in the NHL. Keep in mind that every time a Henrik Lundqvist or a Pekka Rinne or a Kari Lehtonen or a Tuukka Rask comes from over seas, they're not just taking a job - they're taking a job for 5-10 seasons. This means that goalie turnover is extremely slow. Maybe you'll only get one or two new faces per season and those one or two new faces have to be better than the guys they're overtaking. As a result, NHL jobs that would previously go to Canadian backup goalies (like Alex Auld, Michael Leighton, and Andrew Raycroft) are now going to top prospects and more talented Europeans - see Robin Lehner, Eddie Lack, Viktor Fasth, Antti Raanta, etc.

The improvement in NHL goaltending and the influx of European goaltenders conceivably means that as a proportion of total goalies that play, fewer and fewer CHL goalies will end up making the NHL as regulars, and those that do will be the absolute cream-of-the-crop elite performers. Given that only ~5% of CHL goalies turn into NHL'ers and there are roughly 60-70 legitimate average or better goalies currently in the CHL, there are probably only 3 or 4 future NHL goalies in the entire CHL right now. So who are they? Aside from Eric Comrie and Tristan Jarry who I've talked about already, future NHL'ers could be Flyers prospect Anthony Stolarz or Columbus prospect Oscar Dansk. They could be overage import stars like undrafted Eetu Laurikainen or Los Angeles' Patrik Bartosak too. Not all of those guys are going to pan out, and every single one has performed better in their CHL careers than both goaltenders Hockey Canada selected to represent the country at this year's World Juniors.

Perhaps the most damning thing about goaltending is that scouts and "hockey people" who claim to be experts on goaltending and forecasting performance are simply completely wrong the vast majority of the time. The following two graphs are courtesy of Matt Pfeffer, who works as a scout/analyst for the OHL's Ottawa 67's. They measure Goals Versus Threshold (GVT), which is essentially a hockey equivalent of baseball's Wins Above Replacement (WAR), against the draft position of a player. The first graph for skaters is essentially what you'd expect: the best players are taken early, then the value of skaters falls off considerably after the first few selections:

By contrast, goalies are all over the map. There's no real relationship between what their value turns out to be and what scouts think of them:

Add to this the fact that 84% of all goalies drafted out of the CHL between 1995 and 2009 failed to become NHL regulars, and you start to get the sense that scouts are, at best, really bad at judging goaltender talent level.

Conclusion

Odds are that any given goalie in the CHL will not make it to the NHL no matter who they are. That's not a slight against any goalie, that's just the hard truth. The NHL is the absolute best league in the world, getting in is ultra competitive and really, really difficult. The vast majority of prospects out of the CHL fail to get in, and those that do have tended to be elite performers when they were in junior. But, some guys stand a better chance to make the league than others and those guys tend to be the Tristan Jarrys and the Eric Comries of the world, not the Zach Fucales and Jake Patersons. Right now, based on what I've found, I would bet against Zachary Fucale becoming the #1 starting goalie that scouts forecast him to be, since goalies that have performed at the same level as he has have missed the NHL entirely 95% of the time. This is not to say Fucale almost guaranteed to fail. I don't know what he'll develop into. But it's far more likely he develops into someone playing in the Swiss A League than he does the next Henrik Lundqvist.

Of course, this whole thing underpins a much larger and broader discussion about Hockey Canada and player evaluation at the highest levels - mainly that the people that are in charge right now are not very good at it. While the goaltenders selected to represent Canada at this year's World Juniors were unquestionably the wrong ones, Zach Fucale and Jake Paterson were absolutely not the reason why Canada will fail to win the gold medal for the fifth consecutive year. 

Canada's roster was filled with "tough" and "gritty" and "responsible" players like Bo Horvat and Scott Laughton and Josh Anderson and Frederik Gauthier and Adam Pelech and Chris Bigras, while ultra-talented but "risky" stars like Max Domi and Darnell Nurse were left at home. In the end, Canada couldn't score and couldn't even carry the play against a far-inferior Finnish team. Canada had 11 1st round picks on their roster, Finland had 11 undrafted players. Player quality and player development clearly aren't the problems, so what else could it be? Well if the players aren't the problem, then scouting, evaluation, coaching, and management are. Canada is still the best country in the world at hockey, so it's time we start giving ourselves the best chances at winning again, and that starts with accurately evaluating who our best players really are.