The Extra Value of Having an Elite Reliever

The Royals have got all of us thinking. It’s not so much about the Royals finding the only way to win, but they’ve definitely found an interesting way to win, and given their accomplishments you can’t just sweep it away as luck. The Royals have won with a bullpen-heavy model, and now it’s going to be interesting to see if other teams respond by putting more resources toward their relief. On the one hand, according to reports, there are a lot of excellent relievers on the market, as sellers try to meet perceived demand. On the other hand, I guess, every reliever addition is also a reliever subtraction, so. The point is, expect bullpens to be in focus.

And bullpens, I think, are something we struggle to measure. So much attention is paid to the difficulty of evaluating defense, but we also run into some trouble trying to value really good or really bad relievers. We’ve got reliever WAR, which takes leverage somewhat into consideration, but there’s an argument reliever WAR is missing something, something that, say, underrates how much a great reliever is really worth.

The goal here isn’t to try to answer everything. Bullpens are complicated and I can’t develop a model for everything in a day. I’ve simplified, to try to address one point. We already have reliever WAR. How have teams with an elite reliever actually done, compared to teams without one?

If there’s nothing extra going on, you’d expect the difference between teams to be equal to the difference in WAR. If there’s something extra going on, you’d expect the difference to be greater than the difference in WAR. I decided to use 1986 – 2015 as my window, to mostly capture the period of modern bullpen usage.

I started with all the individual reliever seasons of at least 50 innings. From there, I established some cutoffs, so I grabbed the top 10% in ERA-, and then I selected those who were also in the top 10% in FIP-. As the last step, I kept just those relievers who had an average leverage score of at least 1.5 when entering. This makes sure I’m looking at high-leverage relievers, instead of successful mop-up types. Of course, this isn’t the only way to define an elite reliever, but it certainly isn’t including non-elite relievers.

I linked those relievers with the teams they were on. (I eliminated the few who were traded.) So this gave me a sample of teams with an elite reliever, and a sample of teams without such a reliever. Very few teams had two qualifying elite relievers, and I decided to include them anyway. I made sure to put everything on a per-162-game scale, thanks to what happened in 1994 – 1995. Finally, we can just get to the fun stuff. Looking at average team WAR:

With elite reliever: 39.5 average team WAR
Without elite reliever: 32.5

Better teams have been more likely to have elite relievers than worse teams. That’s not a surprise — better teams have better players than worse teams, and worse teams have less use for elite relievers. What’s most important is, based on this, we’d expect a seven-win difference in actual results. Here’s what’s observed:

With elite reliever: 88.6 average wins
Without elite reliever: 79.9

We expected a difference of 7 wins, based on WAR. What we actually get is a difference of 8.7 wins, based on, you know, actual wins. Which leaves 1.7 wins unaccounted for. I decided to examine team bullpen Clutch, to see if it could be of any help — Clutch tends to explain a lot of under- and over-performance. Maybe the bullpens were being leveraged in such a way as to squeeze more out of them than you’d possibly think.

With elite reliever: +1.7 average team bullpen Clutch
Without elite reliever: 0.0

It couldn’t be cleaner. We were missing about 1.7 wins. Through Clutch, we located about 1.7 wins. This appears to explain the over-performance, but given the consistency, it shouldn’t be thought of as over-performance; it should be thought of as a team capability, that WAR misses.

This is evidence that having a shutdown high-leverage reliever can be worth more wins than WAR would suggest. And this is just in the regular season, so we’re not touching on any further benefits in the playoffs. When you have that reliever, you can make sure to use him at the most important times, saving close games and putting out assorted other fires. I think it’s entirely intuitive, so this probably isn’t a shocking result. This doesn’t get into deeper bullpens. And I might be overlooking a bias or two that explains what’s been found. Very well could be there are going to be comments below that make me out to be an idiot, but if they’re right, I’ll appreciate them, since I’d love to have more than one brain considering this.

All of this is looking back. This is reflecting on relievers who were elite. When it comes to team-building, the tricky thing is trying to identify elite relief ahead of time. Of course, Aroldis Chapman looks amazing. Andrew Miller. Wade Davis. And so on. But we’ve long understood that relievers can be particularly volatile, and as an example, a qualifying elite 2014 reliever was Greg Holland. Also Jake McGee. Craig Kimbrel had his ups and downs. Koji Uehara was more vulnerable in 2014 than he was in 2013. Joe Nathan was a hell of a lot worse in 2014 than he was in 2013. Like with any player, when you acquire a good reliever, you’re acquiring present talent, but talent can fade, and injuries can happen at almost any moment. No one has ever been or will ever be a guarantee.

But if you feel like WAR has undervalued the best of the best, you’ve got some support. Teams with elite relievers have been better than teams without elite relievers. And they’ve been better by more than the WAR gap would predict. Seems curious.