The Extra Value of Having an Elite Reliever
The Royals have got all of us thinking. It’s not so much about the Royals finding the only way to win, but they’ve definitely found an interesting way to win, and given their accomplishments you can’t just sweep it away as luck. The Royals have won with a bullpen-heavy model, and now it’s going to be interesting to see if other teams respond by putting more resources toward their relief. On the one hand, according to reports, there are a lot of excellent relievers on the market, as sellers try to meet perceived demand. On the other hand, I guess, every reliever addition is also a reliever subtraction, so. The point is, expect bullpens to be in focus.
And bullpens, I think, are something we struggle to measure. So much attention is paid to the difficulty of evaluating defense, but we also run into some trouble trying to value really good or really bad relievers. We’ve got reliever WAR, which takes leverage somewhat into consideration, but there’s an argument reliever WAR is missing something, something that, say, underrates how much a great reliever is really worth.
The goal here isn’t to try to answer everything. Bullpens are complicated and I can’t develop a model for everything in a day. I’ve simplified, to try to address one point. We already have reliever WAR. How have teams with an elite reliever actually done, compared to teams without one?
If there’s nothing extra going on, you’d expect the difference between teams to be equal to the difference in WAR. If there’s something extra going on, you’d expect the difference to be greater than the difference in WAR. I decided to use 1986 – 2015 as my window, to mostly capture the period of modern bullpen usage.
I started with all the individual reliever seasons of at least 50 innings. From there, I established some cutoffs, so I grabbed the top 10% in ERA-, and then I selected those who were also in the top 10% in FIP-. As the last step, I kept just those relievers who had an average leverage score of at least 1.5 when entering. This makes sure I’m looking at high-leverage relievers, instead of successful mop-up types. Of course, this isn’t the only way to define an elite reliever, but it certainly isn’t including non-elite relievers.
I linked those relievers with the teams they were on. (I eliminated the few who were traded.) So this gave me a sample of teams with an elite reliever, and a sample of teams without such a reliever. Very few teams had two qualifying elite relievers, and I decided to include them anyway. I made sure to put everything on a per-162-game scale, thanks to what happened in 1994 – 1995. Finally, we can just get to the fun stuff. Looking at average team WAR:
With elite reliever: 39.5 average team WAR
Without elite reliever: 32.5
Better teams have been more likely to have elite relievers than worse teams. That’s not a surprise — better teams have better players than worse teams, and worse teams have less use for elite relievers. What’s most important is, based on this, we’d expect a seven-win difference in actual results. Here’s what’s observed:
With elite reliever: 88.6 average wins
Without elite reliever: 79.9
We expected a difference of 7 wins, based on WAR. What we actually get is a difference of 8.7 wins, based on, you know, actual wins. Which leaves 1.7 wins unaccounted for. I decided to examine team bullpen Clutch, to see if it could be of any help — Clutch tends to explain a lot of under- and over-performance. Maybe the bullpens were being leveraged in such a way as to squeeze more out of them than you’d possibly think.
With elite reliever: +1.7 average team bullpen Clutch
Without elite reliever: 0.0
It couldn’t be cleaner. We were missing about 1.7 wins. Through Clutch, we located about 1.7 wins. This appears to explain the over-performance, but given the consistency, it shouldn’t be thought of as over-performance; it should be thought of as a team capability, that WAR misses.
This is evidence that having a shutdown high-leverage reliever can be worth more wins than WAR would suggest. And this is just in the regular season, so we’re not touching on any further benefits in the playoffs. When you have that reliever, you can make sure to use him at the most important times, saving close games and putting out assorted other fires. I think it’s entirely intuitive, so this probably isn’t a shocking result. This doesn’t get into deeper bullpens. And I might be overlooking a bias or two that explains what’s been found. Very well could be there are going to be comments below that make me out to be an idiot, but if they’re right, I’ll appreciate them, since I’d love to have more than one brain considering this.
All of this is looking back. This is reflecting on relievers who were elite. When it comes to team-building, the tricky thing is trying to identify elite relief ahead of time. Of course, Aroldis Chapman looks amazing. Andrew Miller. Wade Davis. And so on. But we’ve long understood that relievers can be particularly volatile, and as an example, a qualifying elite 2014 reliever was Greg Holland. Also Jake McGee. Craig Kimbrel had his ups and downs. Koji Uehara was more vulnerable in 2014 than he was in 2013. Joe Nathan was a hell of a lot worse in 2014 than he was in 2013. Like with any player, when you acquire a good reliever, you’re acquiring present talent, but talent can fade, and injuries can happen at almost any moment. No one has ever been or will ever be a guarantee.
But if you feel like WAR has undervalued the best of the best, you’ve got some support. Teams with elite relievers have been better than teams without elite relievers. And they’ve been better by more than the WAR gap would predict. Seems curious.
Jeff made Lookout Landing a thing, but he does not still write there about the Mariners. He does write here, sometimes about the Mariners, but usually not.
Great article, should be a lot of elite bullpen arms moved this offseason.
Having a dominant reliever is definitely probably a plus but it seems that spending on them unless a truly dominant reliever isn’t a solid choice. Rivera certainly an exception. Maybe Chapman for modern day.
Relievers have high volatility especially since they work in short samples and it seems useful ones can come from anywhere. See the Cubs and Trevor Cahill.
There are a lot of guys that can throw 95 in short stints now. Doesn’t seem like the most effective use of resources.
That’s what I call the fallacy of the saber argument for fungibility of relievers, by bringing up examples where a team finds a good reliever out of nowhere. Using that logic, some people think that you should not pay for relievers. But that’s taking on a lot of luck, that ignores the fact that the team might have to try a number of failures before finding their Cahill. And by then, your playoff chances could be shot. So that’s fine if you just want to find a reliever, you can take that risk, but not so fine if you are a contender or hoping to contend.
As this research shows, having elite relievers is crucial for contending teams, and thus retaining pitchers you think are elite is part of the puzzle of building a playoff team. The Giants were ripped a lot for signing and resigning Affeldt, Romo, Lopez, and Casilla to multi-year big money (for relievers) contracts, but it was this Core Four who dominated the relief innings from 2010 to 2014, and helped the Giants win those championships.
But I agree to a certain extent, there needs to be a balance to this, because elite relievers could cost the team too much. Rob Nenn was great as a closer for the Giants, but after his shoulder ended his career, his $9M per season for three seasons afterward (when $9M was a huge percentage of a payroll), really limited what the Giants could do with their bullpen and we ended up with closers like Matt Herges and Dustin Hermanson, and worse, then they paid for Armando Benitez to be our elite reliever, and he really sucked.
Logically, it makes sense that relievers are valuable because of their high-leverage usage. If you have a tolerable rotation that gives up 3 runs in 6 IP, and a shutdown bullpen, you’ll regularly give up fewer than 4 runs a game. If you have an average offense, you’ll score more than 4 runs a game, making your WP over .500. Not bad for a tolerable rotation with an average offense.
There was a comment in your piece on Clutch about a week ago noting that since ~1990 teams have positive cumulative bullpen clutch scores nearly across the board. It seems that the reliever specialization seen in the last 25 years is not only leading to more innings thrown by relievers but also deployment of relievers in situations where they are set up to realize results greater than their component skills would suggest (that thing about relievers consistently beating their FIP and posting suppressed babip numbers).
So are we basically just putting a number on the degree to which selecting for favorable matchups benefits relievers more than WAR would suggest? Unless an adjustment for relievers getting constant platoon/GB/FB matchup advantages is baked into reliever WAR I am not surprised at all by the gap found in this piece.
Follow up thought: what % of these elite relievers in the sample were capital C closers primarily working the ninth regardless of matchups and how many were 7th/8th inning specialist types?
I had made the comment you reference, and I think Jeff put it well that it’s a “a team capability, that WAR misses”.
“Clutch” is basically the difference between the situational performance weighted by leverage, vs. the situation performance overall. Since pitchers are used in roles that vary a lot by leverage, a lot of pitching “clutch” is out of their hands, but is really a function of how often the best pitchers are used in high leverage situations.
Thanks tz! You are a fantastic contributor to the comments around here.
Thanks!
Get a room you two!
“On the other hand, I guess, every reliever addition is also a reliever subtraction, so.”
Sometimes! But it would be interesting if we see teams crush up more meh starters into elite relievers. One reason could be if they have a better realization of elite relievers’ value. Another reason could be if they get a better understanding of which meh starters convert well.
About time somebody started to figure this out. Good job, Jeff.
Could the extra 1.7 wins that you chalked up to clutch use also be due to some teams with an elite reliever also having more than one elite reliever?
Makes sense. By the form of the clutch equation, it rewards those that don’t blow the hold or save given the same amount of hits. One form of intentional sequencing mentioned by pitchers might be relevant. They talk about becoming more aggressive with a bigger lead. An elite closer with a 3-run lead is probably going to be coming right at a hitter. He knows the odds are he can get 3 outs before they score 3 runs if he doesn’t walk anybody. With a 1-run lead the RP will be more careful. This works if the reliever can actually pitch with command. Doesn’t work if the RP just throws hard and hopes.
And, of course, RP clutch includes “luck”.
Great article, by the way…
Also, lucky teams that have more wins than WAR expected at the deadline are less likely to trade their elite reliever
Also, take a club with a great closer but poor depth. Their WAR will be lower than a bullpen good all the way through. Still, their clutch value could be high, since a lot of runs will be scored in relatively low LI situations. The great closer will make up by shutting down the game when LI is high.
What was the average WAR of the relievers in your sample?
Dying to know that # to give perspective on the 1.7WAR number.
Also what the the N (number of teams) for each grouping?
My rough calculation says about 800 non-elite teams and 100 elite-reliever teams.
Something to keep in mind.
Awesome article.
Nice idea, but using the ERA- cut-off cooks the books somewhat, because ERA for high leverage relievers = high clutch score (on average). ERA is extremely sensitive to sequencing. Would be much stronger if you used only FIP- or, if you want to add more info, wOBA-against.
Of course, if you try to use wOBA-against, you’ll see it’s extremely difficult with FanGraphs, because for some reason you can’t get multiple years team splits to try and stitch wOBA back together again.
Excellent point, and something I should’ve caught. I re-ran using just FIP- and leverage, leaving out ERA-, and no matter where I put my cutoffs, I wound up with a difference in Clutch a little greater than 1.0. Not as high as 1.7, but also not nothing.
Shoot, sorry, I hadn’t seen your reply before I opened by big mouth again. Thanks for taking a look without FIP. Interesting that you still found that much clutch, even if significantly less than before. I’ve been meaning to do a similar analysis using base runs instead of WAR and using wOBA-against instead of FIP/ERA.
A couple other notes here: for your entire population of team seasons, the average bullpen clutch is 0.24, so the actual “premium” provided by the elite relievers you identified is less than 1.0 of clutch. And if you applied your selection criteria for elite relievers to the entire population of relievers meeting your minimum qualifications across all seasons, the group may be skewed towards years with even higher average bullpen clutch. For example, the past 10 seasons have an average team bullpen clutch over 0.4. This means the relative advantage of elite arms may be smaller yet.
The bigger point is about looking at bullpen clutch at all. All else equal, higher is better, but the measure is sensitive to how bad the mop-up guys are, not just how good the high leverage guys are. Clutch is a tricky concept in that respect. It’s a combination of opportunity, usage, talent distribution across the bullpen, and individual player clutch (within their own appearances). There is so much to pull apart here.
Or put differently, you have some of the causality backwards, Jeff. High clutch leads to your sample of elite relievers, not the other way around.
People only starting talking about the Royals so much when they won the World Series, not when they outperformed expectations in the regular season, so it seems to be missing the main point here to be focusing on regular season performance (while not denying that the comments made here about regular season performance are important). To focus on postseason performance, what seems to be most important isn’t specifically having a single elite reliever (although that’s great too), but rather having DEPTH in relieving, having people who can take over in the 6th inning or even earlier and shut the other team down. It can make up for extremely mediocre starting pitching, and the Royals are a perfect example of this. As great as Wade Davis is, he could only pitch at most 2 innings in a game, and that not on consecutive days normally, and the Royals won quite a few games in the postseason when their starters left in the 5th or 6th inning, and on the flip side they made a huge number of comebacks against either mediocre middle relievers or else tiring starters who weren’t pulled because the opposing manager didn’t trust his relievers. This was pretty much the same formula that the Giants used to win the World Series in 2012 and 2014 (other than Bumgarner their starters were incredibly mediocre in 2014), and it can be argued that the reason the Giants won games 4 and 7 vs. KC is the Bochy went to his bullpen sooner than Yost did.
I have an intuitive sense that luck plays a factor in a sport like baseball. However, my impression is that by “luck,” most people re sports mean random occurrences that somehow favor one team more than another. That’s possible in a short-term situation, maybe even a few at-bats per game. But not over the long haul. I’m as despondent as any Mets fan that the Royals won the World Series, but I think they made their own luck, to steal a phrase from Titanic.
I really think the aggressive swinging style of the current Royals lineup generates contact, especially off hard throwers (i.e. power pitchers or late-inning relievers) that is somehow a bit more unpredictable and harder to field. I took some time to go watch Game 7 of the 2014 World Series and was amazed to see that Joe Panik’s incredible double-play arguably made the difference in the game.
That was a typical ball off the a Royals bat, and it just didn’t work out in that case because of the way-above average fielding on the play by Panik. Especially when you consider that against an “unlucky” team (less unlucky than less agile, really) like the Mets, a ball like that would have gone through to the outfield or possibly been botched by Murphy. And even getting one out on that play would have helped SF, but they managed to get a double play thanks to the (correct) challenge.
Of course, many are lauding the Royals as the best team ever, which I think is a bit of hyperbole. Even though they are pretty amazing, it’s possible for an elite power pitcher like Madison Bumgarner to neutralize them with pitch movement that doesn’t allow them to put the ball in play the way they want to.
I honestly think teams should spend a lot of time reviewing those Royals hits to prepare for next season. It seems like preparing for controlled chaos, but that is the way the Royals seemed to me.
Bumgarner isn’t a power pitcher in the classic sense, he doesn’t throw THAT fast, but his delivery is hard to pick up, coming from an unusual angle and with a lot of body rotation, and he throws (or can throw) a lot of fastballs which rise out of the strike zone, and the Royals just couldn’t lay off of them. As games and ABs went on, they just kept on swinging at higher and higher pitches. The Mets pitchers on the other hand kept throwing strikes even when there were two strikes on the hitter, they just threw away their advantage in the count again and again.
I really like this idea. It’s also possible that defenders behind the Mets staff could become a bit complacent or at least less intimately familiar with balls in play scenarios behind a low contact high K staff that induced a lot of easy weak contact. Practice makes perfect and repetitions are key to mastery so they say. How many even semi-well struck balls did anyone have to field behind Familia once he took off in 2015?
Some statcast data on the quality of “bad ball” contact made by the Royals especially in those epic close/late situations could be interesting stuff.
If these are extra wins due to the clever use of elite bullpen arms at the most opportune moment and if we generally accept that the pitcher do not decide when to enter the game, but that the manager decides when they enter, it seems reasonable that at least some of this wins are chalked up to the manager and not the pitcher. (Some of it is the pitcher, because once you enter, you still have to deliver)
So, in that line of thinking it might be that the WAR-valuation for these pitchers is basically correct, and what is missing in our models is a “clever-situational-decision-making”-WAR, which may or may not be sustainable year-to-year.
I generally agree, Jeroen. Although it is less in-game cleverness and more sustainable if you have a lot of elite RPs, like the 2014-15 Royals. The ensemble sample variability should be a lot less than focusing on one pitcher.
perhaps the difference in manager bullpen use is not significantly different among teams though
I would expect trash teams to unload an elite reliever if they happened to roll Yahtzee, and for only in-contention teams to seek them. Since they’re so variable and work on short deals, isn’t this the most icy of icing on a team cake?
Not sure if there is a clean way of looking at it, but is some of the over/underperformance a variability issue in terms of bullpen quality across the seven arms?
A team with more variability (“stars and scrubs”) may be more likely to overperform as the usage becomes more leveraged. A team with a much flatter distribution may see a closer link between team WAR and actual wins.
I recall reading an article which looked at this in terms of bullpen variablity vs over/under performance of pythagorean records and seeing some correlation, but not quite meaningful statistically. Sadly I can’t find the article.
Lot’s of teams have 1 elite RP in their bullpen. The characteristic shared by the Giants of 2010, 2012 and 2014 and the Royals of 2014 and 2015 is that they both had very deep bullpens. It is possible that your findings with 1 elite reliever gets magnified with deep bullpens, especially when you have a manager like Bruce Bochy who deftly mixes and matches as the game develops and is not necessarily captive to defined roles.
Or if the bullpen is deep enough to be Yost-proof (like this year’s Royals) 😉
Definitely (but this was less striking for the 2010 Giants because they also had quite a few good starters, which wasn’t the case in 2012 and 2014). And the effect is much much greater in the postseason, because that’s when managers pull starting pitchers faster, so you can get away with very mediocre starting pitching.
One factor that I wonder about here is whether team quality affects whether a reliever counts as elite. Ken Giles, for instance, has obviously been an elite reliever, but in 2014 he missed the inLI cutoff with 1.18 and in 2015 he barely made it with 1.60. (In 2014 he also missed the innings cutoff.) This wasn’t so much because of his role as because the Phillies were so bad that they didn’t offer a ton of high-leverage relief opportunities. Papelbon also missed the inLI cutoff in 2015 (though he’d also have missed the FIP- cutoff, and was traded).
Eh, it’s probably a minimal effect. A more likely effect in this area is what astrostl says, that bad teams with elite relievers are more likely to trade them, so the sample excludes some bad teams. But the Clutch numbers are very suggestive.
So reliever WAR includes some leverage. Clutch is based on leverage. If reliever WAR simply included all leverage, would that explain the 1.7 wins you’re crediting to clutch? Just seems to weird to count leverage twice and in two different ways when it seems to be the entirety of the answer.
To clarify. I am not advocating giving relievers credit for leverage. To the contrary, I would prefer reliever WAR to be fully context neutral, precisely so it’s easier to separate out the win value of the performance content, which is within the player’s control and a function of his ability, from the win value of the performance context, which, while correlated to his ability, is not a function of it.
Question, Jeff… If clutch is based on WPA, shouldn’t a clutch value of 1.7 be worth 3.4 wins? You only need to add 0.5 WPA to original Win Expectancy (50/50 chance) to win a ballgame.
Echoing a few above comments: What about when you have two elite relievers? Or three?
Does it vary proportionally? Or does it stack in a weird/cool way?
The implication of this article is that the contribution of a team’s relief pitching to total wins is WAR+CLUTCH. By this count, the five best relieving teams are the Yankees (12.04), the Cardinals (10.05), the Orioles (8.36), the Pirates (8.12), and the Royals (6.78). Two of the Royals postseason opponents were among the 5 worst by this count, the Blue Jays (2nd worst, -2.37) and the Astros (5th worst, -0.87). This is consistent with the bullpen failures of these teams against the Royals, aka the Royals great clutch hitting (the Mets on the other hand were actually okay by this count (5.48, 10th best)).
I was thinking about this recently, but much more in terms of bullpen depth. I don’t think WAR is failing to capture the value of relievers as individuals, but in team concepts, there’s some “missing” WAR that comes from having a bullpen with plenty of depth.
It’s mostly about synergy between the bullpen and starting pitching. I’m a Braves fan, so in the past few years I’ve watched bullpens ranging from elite to complete dumpster fires. One difference is that the starting pitching actually performs better with bullpen depth…because if they’re struggling in the fifth or sixth inning, the manager knows he has plenty of good bullpen arms to get through the ninth. And that sometimes is the difference, in a single game, between a starter allowing three runs and five runs, or the difference between four runs and seven runs.
Even if closer usage is less-than-optimal by most managers, it’s still pretty straightforward that you can’t use your closer every night. The Royals, however, had multiple good relief pitchers, as well some good swing-men like Blanton and Medlen.
This research is a nice bookend to BP’s research on extended playoff success, which they published in a book, Baseball Between the Numbers (Chapter on why Beane’s stuff don’t work in the playoffs).
They studied playoff teams since the playoffs system started and use regression to see which team metrics enabled teams to go deeper into the playoffs. They came up with three metrics (the secret sauce was their term for it), one of which was their metric for how good a reliever was (WRXL). Essentially, having an elite reliever was a good indicator of a team capable of going deep into the playoffs (secret sauce in the book is different from the one they poorly used in articles afterward, as the secret sauce was in terms of historic ranking, not seasonal ranking, as they did in their articles).
I find it kind of funny, though, that it took the Royals to win with a strong bullpen to get studies like this done, when the Giants have had strong relievers since 2010, Wilson, Romo, among others. They have also been going with high contact hitters for a number of seasons as well, people scoffed when they acquired Freddy Sanchez, and others acquired after him, but they have been building up a high contact team for a while now.
Thing is a top reliever is EXTRA valuable in the playoffs because of the ability to pitch more innings.
There should be a study on how a reliever’s importance might increase if you assume he can join any individual contest for at least one inning if necessary.
Is the difference more than 1.7?
If teams with an elite reliever outperform their expected win total, doesn’t that mean that teams without one will underperform?