Are Starters Improving Relative to Relievers?

If you read about baseball on the internet these days, it’s tough to miss pieces on the changing relative skill levels of relievers and starting pitchers. As Sam Miller pointed out on Effectively Wild, relievers have a higher ERA this year than starters. Not only that, but the strikeout rate advantage relievers have traditionally had over starters is plummeting. Look at almost any statistic, and the historical edge relievers have had over those in the rotation has diminished.

At the same time, relievers are setting volume records left and right. For five straight years, relievers have set a new record for largest share of pitches thrown. In 2012, Rockies relievers struck out more batters than Rockies starters for the first time in baseball history, and you could convince yourself that it was a Coors field oddity. In the next six years, however, four more teams did it. From a bulk perspective, relievers are pitching more and more innings, carrying ever more of baseball’s pitching workload.

Clearly, these two effects are correlated. You’ve undoubtedly heard of the times-through-the-order penalty, the concept that starters fare worse each time through the batting order. In tandem with the innings spike by relievers, the number of pitches starters throw their third time through the order is plummeting. It’s not rocket science — the third time through the order is the time when starters are weakest, and those weak points are disappearing. Of course starters’ stats look better!

Given the changing roles of starters and relievers, it’s probably not right to look at unmodified splits to figure out whether starters are actually getting better relative to relievers. Even if the talent level of every pitcher remained exactly the same, cutting out a chunk of third-time-through plate appearances from starters’ cumulative totals will change their statistics. To actually see whether relievers are getting better relative to starters, we’ll need to do something a little fancier.

With copious database assistance from Sean Dolinar, I came up with an idea. What if we could just look at starters and relievers their first time through the order, and strip out non-representative at-bats from each of those samples? Giving starters credit for blowing away pitchers defeats the purpose of this analysis, for example, because relievers basically never get to face pitchers. To handle this discrepancy, we looked only at pitchers’ performance against the first seven batters in a lineup (as a few teams bat their pitcher eighth), and only the first time through that lineup.

Now, as you might expect, this narrows the gap between starters and relievers considerably. Limiting our sample to the first time through the order barely changes relievers’ stats: in 2018, for example, relievers only faced 3,459 batters a second (or third!) time, as compared to 71,212 batters the first time through. Starters, on the other hand, get to discard a ton of challenging situations. Right there, you’d expect to see a large narrowing in skill differential.

In fact, if you don’t look too closely, you might be inclined to say that the times-through-the-order penalty was the only reason relievers looked better than starters all along. This year, for example, starters have allowed a .313 wOBA the first time through the order against 1-7 hitters. Relievers, on the other hand, have allowed a .319 wOBA over the same split. There’s just one problem, though. Starters haven’t always been better than relievers the first time through the order. Take a look at the difference between wOBA allowed for the two cohorts over time. One note: in this and all following analysis, I’ve stripped out intentional walks.

For the most part, relievers enjoyed a consistent advantage over starters, even controlling for times through the order. Last year, though, that edge disappeared completely, and this year starters have taken the lead.

It’s not just wOBA, either. Take a look at strikeout rates using the same data set.

Though it’s not quite as dramatic, the last two years represent the smallest advantage relievers have had over starters in our sample. Walks are a bit noisier, but while relievers have always walked more batters than starters, the gap is at its widest point in our sample in 2019.

So, what does this all mean? Are starters just magically getting better relative to relievers, even after accounting for the conditions they’re pitching under? Well, there are two theories I’d like to explore. First, more and more relievers are getting into games. In 2008, 496 pitchers appeared in relief. Ten years later, in 2018, that number was up to 657. Pitchers who might have been back-end starters or minor league depth are increasingly joining the back end of major league bullpens. What if the declining (or even reversing) reliever edge is just showing the dilution of the reliever ranks?

To handle this criteria, I got creative. I ran first-time-through-the-order numbers for starters and relievers, but this time only for players who met a playing time minimum. I looked at relievers and starters who faced at least 120 batters the first time through the order (40 for 2019), trimming the marginal hangers-on from both lists.

These pitchers, as you might imagine, perform better than the total set of starters and relievers. They’re the pitchers their teams are consciously giving innings to, not Quad-A players filling gaps in the bullpen or rotation. Still, though, the trend of starters performing better relative to relievers remains.

Now, this isn’t a perfect way to control for the talent level of relievers. Still, it makes the talent pools far more uniform, and yet the sudden improvement of starters relative to relievers remains. What else could it be?

There’s another excellent reason the results gap between starters and relievers might be narrowing. Forget the diluted talent pool for relievers; what if starters are approaching their outings differently? For most of baseball history, starters were expected to pace themselves and pitch the majority of each game they appeared in. Recently, though, that model has changed. Many starters now expect to face about 18 batters (two times through the order) and then hit the showers. Maybe they’re just throwing with much more effort to each batter than they used to.

To test for this, I turned to a trusty cohort of pitchers — swingmen who made at least 10 starts and 10 relief appearances in a given year. These are pitchers without a fixed role, bouncing around to whatever spot serves their team best. Maybe they’re banished to the bullpen after a string of bad starts, or promoted to the starting rotation after some effective relief work. Either way, they have enough appearances in each role to understand how to prepare for it. If the way starters approach starting is truly changing, we’d expect swingmen to have smaller splits than they did in the past.

Take a look at the performance gap the first time through the order for swingmen (between starting and relieving). This data goes back only to 2008, the first year I could query on Baseball Savant.

A quick note on the methodology here: in creating this split, I emulated a study in The Book, working out each pitcher’s first-time-through wOBA gap, then taking a weighted average using the smaller of the number of batters faced as a starter or a reliever. I also manually stripped out openers in 2018.

Well, that data is way too noisy. It’s kind of hard to tell anything, as we’ve run into our old nemesis, sample size. There are only 15 or 20 pitchers a year who are true swingmen, which makes it hard to do a year-by-year study. Let’s look at that data again, only with three-year averages.

Okay, there’s maybe a trend there if you want to see it. That’s the problem with these sample sizes (1000 to 1200 PA a year): they’re small enough that noise is a major component. While this isn’t real evidence of anything, it’s at least pleasing to see it move in the correct direction.

So, what can we say overall about the seeming improvement of starters relative to relievers in the past handful of years? As far as we can tell, it’s not just an artifact of starters going through the order a third time less often. Even focusing just on pitchers’ first time through the order, the last two years show a marked change in relative skill level.

The change also probably can’t be explained by a dilution of reliever talent — even looking only at high-volume starters and relievers, the talent change over the last two years shows up. This effect merits further study, but it doesn’t seem like an obvious culprit.

The best evidence we have is the diminishing advantage swingmen get when pitching in relief. Maybe, just maybe, starters are improving relative to relievers because they are preparing more like relievers, putting in maximum (or closer to maximum) effort on every pitch. The data isn’t conclusive, but it’s at least suggestive.

No matter how you slice it, starters’ performance is getting better relative to relievers. Whether it’s changing preparation, the changing talent levels of pitchers becoming starters, or something else I didn’t consider, you can’t deny the effect. Whatever the cause, baseball looks different now than it did 10 years ago. I can’t wait to see what the next 10 years bring.