Fun with Shutdowns and Meltdowns
Yesterday David introduced a couple of new stats, Shutdowns and Meltdowns, to the site. It’s fashioned after saves/blown saves but is vastly superior, because it’s a metric that uses WPA as a substitute of the brainless, archaic save stat and the rules that guide it.
A team essentially has something like a 98% likelihood of winning the game with a three-run lead with no outs, yet a manager will trot out his ace reliever in that situation about 98% of the time for the sake of save. But when the game is on the line and it’s non-save situation, we often see managers make some of the most bizarre choices in their bullpen usage.
Take for instance Tuesday night’s Phillies–Cardinals game that ended in the 10th on a walk-off homer by Carlos Ruiz. While there’s no real “ace” in the Cardinal bullpen, the inexperienced Blake Hawksworth isn’t the guy you normally would want on the mound against the Phillies in such a high leverage situation, but it appears Tony La Russa held back his closer because it wasn’t a save situation.
Anyway, with any luck this catches on. Just to recap:
A Shutdown is when a reliever accumulates greater than or equal to 0.06 WPA in any individual game.
A Meltdown is when a reliever’s WPA is less than or equal to -0.06 in any individual game.
What I thought would be interesting is to look at the “Meltdowniest” pitchers of the past three seasons, as well as the ones who we could say have ice water in their veins. The pitchers with the most meltdowns are usually the ones fans want to ride out of town on a rail, along with their manager, while the pitchers with the smallest meltdown rate we tend to feel pretty comfortable with, even in the highest leverage situations.
These are the pitchers with the highest percentage of relief appearances that resulted in a Meltdown:
| Name | G | SD | MD | Meltdown% |
| Aaron Heilman | 229 | 55 | 52 | 22.7% |
| Luis Ayala | 163 | 33 | 37 | 22.7% |
| Shawn Camp | 149 | 36 | 33 | 22.1% |
| Brian Bass | 93 | 18 | 20 | 21.5% |
| Juan Rincon | 143 | 27 | 30 | 21.0% |
| Scott Linebrink | 178 | 61 | 37 | 20.8% |
| Sean Green | 215 | 52 | 43 | 20.0% |
| Mark Lowe | 136 | 47 | 27 | 19.9% |
| Zach Miner | 111 | 39 | 22 | 19.8% |
| Cla Meredith | 217 | 45 | 43 | 19.8% |
Aaron Heilman is our King of Catastrophe. Cub fans are glad to see him take his act elsewhere. That four-year, $19 million deal given to Scott Linebrink was given by the White Sox is looking like one the more awful contracts given to a middle reliever in recent memory.
So what pitchers have ice water in their veins, or in other words, the pitchers who have experienced the fewest rate of blowups per appearance? I excluded anyone who didn’t average at least an average pLI of 1.2.
| Name | G | SD | MD | SD% | Meltdown% |
| Joe Nathan | 206 | 104 | 18 | 50.5% | 8.7% |
| Jonathan Papelbon | 192 | 85 | 17 | 44.3% | 8.9% |
| Joakim Soria | 172 | 86 | 16 | 50.0% | 9.3% |
| Mariano Rivera | 197 | 104 | 20 | 52.8% | 10.2% |
| Takashi Saito | 164 | 57 | 17 | 34.8% | 10.4% |
| Jose Valverde | 191 | 89 | 21 | 46.6% | 11.0% |
| Francisco Cordero | 206 | 101 | 23 | 49.0% | 11.2% |
| David Aardsma | 145 | 63 | 17 | 43.4% | 11.7% |
| Rafael Soriano | 162 | 63 | 19 | 38.9% | 11.7% |
| Bobby Jenks | 175 | 76 | 21 | 43.4% | 12.0% |
No major surprises here, especially the from the top four.
What about the highest Meltdown % by pitchers with 30 or more saves the past three seasons?
| Name | G | SD | MD | SD% | Meltdown% |
| Chad Qualls | 207 | 88 | 36 | 42.5% | 17.4% |
| David Weathers | 210 | 73 | 36 | 34.8% | 17.1% |
| Kevin Gregg | 218 | 89 | 35 | 40.8% | 16.1% |
| Brian Wilson | 155 | 84 | 24 | 54.2% | 15.5% |
| C.J. Wilson | 190 | 74 | 29 | 38.9% | 15.3% |
| Fernando Rodney | 159 | 66 | 24 | 41.5% | 15.1% |
| Matt Capps | 182 | 75 | 26 | 41.2% | 14.3% |
| Brad Lidge | 205 | 83 | 29 | 40.5% | 14.1% |
| Brandon Lyon | 199 | 78 | 28 | 39.2% | 14.1% |
| Trevor Hoffman | 164 | 86 | 23 | 52.4% | 14.0% |
The unpopular trio of Chad Qualls, Kevin Gregg and David “Stormy” Weathers lead the list. We also see the Phillies favorite bipolar pitcher, Brad Lidge make this list. Interestingly enough, Brian Wilson had the highest percentage of his outings end in either a Shutdown or a Meltdown of any reliever.
Again, I hope this catches on. Shutdowns and Meltdowns are a more practical and intuitive way at evaluating reliever performance, and I think a more fun one, too.
Erik Manning is the founder of Future Redbirds and covers the Cardinals for Heater Magazine. You can get more of his analysis and rantings in bite-sized bits by following him on twitter.
This is actually kind of interesting. I sort of expected Kevin Gregg to show up in your third list, but I’m actually kind of surprised to see Trevor Hoffman in there too. What about a SD to MD ratio? Maybe it’s not just important to see how many times in general someone “shuts down”, but it’s also important to see what the ratio of great games to bad games is.
Like I suggested elsewhere, though, I’d like to scale WPA to outs to be able to better estimate the impact of the usage of lesser relievers in smaller situations. How much does a LOOGY contribute to the bullpen when used specifically in those situations? Perhaps a secondary statistic based on SD can be devised that measures the effectiveness of relievers when used in lower-out situations, or SD can be expanded.
So really, why not scale the amount of WPA added to the amount of outs the pitcher gets? If you get a positive WPA in just one out, that means you’re doing your job as a LOOGY, right? You can set the tolerance for this stat pretty low because it wants to measure minute changes.
We know that the +/- .06 WPA is essentially over three outs. What about +/- .04 WPA for a 2 out situation and +/- .02 WPA for a 1 out situation? This is obviously just throwing an idea out there, because I’m not (yet) a statistician.
A LOOGY should be able to still get a shutdown if he comes in for one or two outs in certain situations, especially if it’s a close game and there are runners on base. The WPA changes enough in close games in the later innings that if the LOOGY comes in, gets one or two outs, he should still be able to get a shutdown even though he didn’t pitch three outs.
But if a LOOGY comes in to get a single out earlier in the game when the score isn’t close, Im not sure that reliever has earned a shutdown, as I’m not sure how much that really helped the team get closer to winning
Yeah, take for example the Cards. Their LOOGYs are 2nd and 3rd in SDs because they have a decent amount of appearances in high lev situations
However, I do think the SD/MD ratio could be very useful.
It might be interesting because it would be good to quickly rate the consistency — and value — of people with decent shutdown percentages at a glance.
Case in point: In the past three seasons, Kevin Gregg has an SD/MD of 2.54, Joe Nathan has an SD/MD of 5.78 and Trevor Hoffman has an SD/MD of 3.74. You already knew Nathan was better than Kevin Gregg anyway, but it’s nice to have a quick metric that tells you what’s up. You can also obtain an average league SD/MD and quickly determine who is above or below average in this aspect.
Honestly, I originally thought of looking at it this way, but for some reason or another I settled the percentages of their games that were either shutdowns/meltdowns or neutral. Looking back, I think I’d just do the SD/MD for all the reasons stated. Maybe for another post I can span SD/MD ratios over the decades, see who was the king of each categories for aughts, 90’s, 80’s…
Of course, I listed the SD/MDs in the tables so you can still eyeball it.
Erik, it’s not like you can’t have both. It’s not a bad idea to be able to see how many games a reliever came into in which they didn’t really wildly affect the outcome of the game, so maybe having SD, MD, NE (neutral effect) and SD/MD could be the four things we’re looking to determine.
I agree having both would be good to see, as the SD% and MD% show how often the pitcher is used during important situations in the game. The SD/MD could be a better indication of performance overall.
The win value of an out, when LI = 1, is +.027 wins. So, if you can get two outs when the LI is 1.2, you get your shutdown. To get it for one out, you need the LI to be 2.3.
So, definitely possible to be a LOOGY and earn a shutdown under the current construction.
So in other words, Renyel Pinto’s SD/MD must be like .1 or something
Man, those graphs are long! 😉
Not graphs, tables.
Yeah, the front page doesn’t like tables, for some reason. Should be fixed now, though.
Wow, just awesome work FG!
Pretty sure it was Tango and some commentators over at his site that put the wrapping paper on this one. The conversation on naming the stat was pretty great – like trying to figure out what to call a new dog
The Wilson thing might be that he can’t get a save without putting a couple guys on first. This makes for some super-high leverage strikeouts with runners on in the ninth (shutdown), and when it goes south, he gets a meltdown.
The WPA will come out to be the same thing if you start the ninth with a 2 run lead and allow no runs, whether it’s 1-2-3, or you load the bases first and then get 1-2-3.
To expand, this is because it is the NET change in WPA. If he allows some baserunners in the inning, his WPA will be negative at that time, but if he gets through the inning without anyone scoring his WPA will always be positive. It only matters in someone comes in during the inning to replace him.
after being at last night’s Rangers/Royals game, my gut is very familiar with the Meltdown statistic. see Nippert, Dustin; Ray, Chris; and Francisco, Frankie
of course, Mr. Soria wasn’t so hot last night either 🙂
That should have been a 🙁 face
Regarding the Hawksworth example, I think LaRussa left him in because memories of the twenty inning Mets game were still fresh, and he didn’t want to burn through relievers if they were still pitching well, as Hawksworth had in the 9th.
I’m new to the game so I apologize if this is already out there, but is there a stat that improves upon the “quality start” like this does for saves?
they have something called game score but isn’t used here.
I’m working on it…
It would be nice to have both descriptive and predictive evaluations for starting pitchers. Descriptive evaluations–trying to record how they actually did– would do something with WPA or WPA/LI. Predictive evaluations–looking at how pitchers “should” have done–might combine FIP with Innings Pitched. If Justin Verlander has a 12 K, 1 BB, 1 HR, 120-pitch, 6-inning start, and Roy Halladay has a 7 K, 1 BB, 1 HR, 120-pitch, 8-inning start, Verlander has a better FIP and xFIP than Halladay (I think, right?). But Halladay has had a better start, because you should credit Verlander with 2 IP of average relief work that Halladay avoided. Now, that may end up judging pitchers somewhat on their manager’s decisions, but it’s better than throwing our hands up and saying, “We can’t evaluate this!”
If the WE actually decreases even though the pitcher got an out, who is the WPA increase/decrease actually attributed to? If its negative to the pitcher I think the stastic can be unfair to a 7th inning reliever where the win probabilities can actually decrease even with an increase in outs
For example, the home starter leaves a mess with the home team up by 1 in the bottom of the 7th with no outs and men on first and second. Reliever comes in and the WE prob is .885 at that point. He K’s the next guy for the 1st out: WE=.859. Next guy pops out: WE .809. Lets assume the manager puts in a loogy to face the final hitter of the inning. The 7th inning guy just got a blowup @ -.076 for shutting down 2 hitters in a row.
You are highly misinformed. In no way does your example make sense. You are flipping the signs, or looking at the WE from the batting team’s perspective, or something.
If you make an out, and no runner advances a base, the batting team ALWAYS (without exception) reduces its chances of winning and the fielding teams ALWAYS increases its chances of winning.
I figured out what I was doing wrong – I was looking at the bottom of the 7th from the perspective of the home team – obviously the home team is hitting – not pitching! I knew something was severely odd. Thanks!
The fact that Heilman is ranked #1 on Meltdowns confirms that this is the most accurate reliever stat of all time. Even with his solid traditional stats in 07, you could hear the collective groans of 50k fans when he came into close games.
I love the Shutodwn/Metldown stat. Love the name too.
Just thinking out loud…
-Can this be translated or applied into a predictive nature?
-How dependent would this be on the management (or mis-management) of the bull pen or is that an unanswerable question?
-Does x number of Shutdowns/Meltdowns correlate to a pitchers WAR?
Rivera’s postseason Shutdown/Meltdown ratio has to be absurd.
I used Shutdowns and Meltdowns to come up with a stat called ROL (short for Rolaids because Rolaids spells relief). The formula is (IP*SD*gmLI)/(GM*MD*xFIP), where IP is innings, SD is Shutdowns, gmLI is leverage index entering a game, GM is games, MD is Meltdowns, and xFIP is expected field independent pitching. I ran it for 2009. You can check it out if you want… http://www.themartinezreport.com/2010/05/rolaids-spells-relief-2009.html I have no idea if it has any merit.
This is really some great stuff. I’d love to see Fangraphs collaborate with Martinez on this ROL stat.
Thanks. I think I’m onto something, but who knows. I hope people check it out.
Erik, I think this is a great stat. This is definitely one i want to see catch on.
Brian Wilson probably has the highest shutdown/meltdown frequency because he’s completely insane and his pitching reflects whichever bipolar mood he’s in that day.