Strike Three?! Let’s Check in on Umpire Accuracy

About a year ago, I wrote about how umpires have improved at calling balls and strikes throughout the pitch tracking era. They have gotten a whole lot better, especially at identifying strikes. While everyone appreciates a more consistent and accurate zone, that has made things a bit harder for hitters overall. More importantly, their progress didn’t seem like it was showing any signs of slowing down. With the 2023 season in the books, it’s time to check in on whether that’s still the case. The methodology here is simple: check to see if Statcast agrees with the umpire’s decision for each called ball and strike. This isn’t a perfect method, as a pitch right over the heart of the plate is a much easier call than one right on the edge, but the enormous sample sizes (there were 376,635 takes in 2023) mean that things even out over time.

First of all, overall accuracy continued to rise in 2023. According to Statcast, 92.8% of takes were called correctly, up from 92.45% in 2022. In other words, umpires missed a call once every 3.6 plate appearances. This isn’t terribly surprising, as accuracy has risen in every single season of the pitch tracking era. Notably, in the last three years, the improvements have been very nearly identical: 0.37%, 0.33%, and 0.35%:

This graph might make it look like umpire accuracy isn’t in any danger of plateauing just yet. It shows three years of nearly identical improvement and an overall slope that’s more or less unchanged since 2008. However, things look slightly different if we break the takes down by whether or not they were inside the strike zone. It’s not necessarily easier to identify a ball correctly than it is to identify a strike correctly. However, balls are easy to identify more often than strikes are, simply because the area outside the strike zone is a lot bigger than the area inside zone. In 2023, 55.5% of the takes on pitches inside the strike zone were also in the shadow zone. That is to say that when the right call was a strike, it was usually a difficult call. For takes on pitches outside the zone, only 31.3% were also in the shadow zone. That’s a big difference! If you’ve ever adapted a cake recipe for a different size pan or heard a lecture about the relationship between volume and surface area (or if you’ve ever borne the brunt of Michael Baumann’s very strong opinions about what size pizza you should order), you likely know this intuitively. The strike zone is small and (in theory) vaguely rectangular. Outside the strike zone is, well, everything else:

At the beginning of the pitch tracking era, there was a lot more room for improvement on pitches in the strike zone, and progress there came more rapidly. That is no longer the case. Umpires are still improving at roughly the same slow, steady pace on pitches outside the zone. However, inside the zone, their accuracy has actually declined by a hair over the past two years. In other words, it looks like we’re finally learning the limits of umpire accuracy, at least on pitches inside the zone:

Last year, I broke things down by Baseball Savant’s attack zones. This time, I’m going to focus solely on the shadow zone, because umpires have gotten so good at the other zones that there simply isn’t much room left for improvement. Over the heart of the plate, they got 99.26% of calls right in 2023, down just a hair from 2022. In the chase zone, they were at 99.24%, up a hair from 2022. Neither change was more than seven-hundredths of a percent, and neither has deviated by so much as half a percentage point in total over the past five years.

In the waste zone, umpires missed just two calls all season. As you might have guessed, these weren’t normal pitches. Both missed calls came on particularly eephus-y eephuses from position player Ryan McKenna:

I think we can forgive Jordan Baker for letting Adley Rutschman convince him that these two parabolas were actually strikes in the ninth inning of a 12-2 game.

Between the chase, waste, and heart zones, we’re talking about just 1,301 missed calls total in 2023. That’s 1.87 misses per game. I don’t know what threshold for error other people might have, but I can’t really imagine asking umpires to improve much more than that.

More importantly, that means that any changes are happening around the edges of the plate, in the shadow zone. Keep in mind that roughly 42% of all pitches are in the shadow zone, and roughly 80% of all takes. At the beginning of the pitch tracking era, that number was down below 76%, but as umpires improved, both pitchers and hitters got a better sense of the zone. Now more than ever, this is where the action is:

Although the overall slopes are very clear, the lines in this graph are a lot more jagged, which makes sense. These are all close calls! Back in 2008, umpires got them right less than two-thirds of the time. Now they get them right 81.9% of the time, well over four-fifths. That means that umpires are missing roughly nine calls in the shadow zone per game. From 2022 to 2023, overall accuracy in the shadow zone improved by 0.92%, and it has improved at an average of 0.89% for the last three years. In that sense, the improvement shows no signs of slowing down.

That said, take a look at the blue line. It makes a pretty compelling argument that umpires have reached their limit on accuracy, at least for shadow zone pitches inside the strike zone. Over the last four seasons, the accuracy has stayed between 81.33% and 81.87%. That’s a total deviation of just 0.54%. It’s the first time we’ve ever seen that kind of stability in the entirety of the pitch tracking era, so it would be a surprise to see the accuracy suddenly start climbing again. Maybe this is as good as it gets.

However, on shadow zone pitches that are outside the zone, umpires are still improving. In fact, accuracy outside the zone has improved by more than 5% over the last four seasons, and by at least 1.25% in each of the last three seasons. It has surpassed accuracy inside the zone for the first time since 2016. There’s no sign of a plateau here, which leads me to believe that the red line could keep on going up. Still, I can’t imagine there’s that much room for improvement, seeing as accuracy on pitches inside the zone just hit its peak right around the same spot.

Whereas last year I came away from my research thinking that the improved accuracy in ball-strike calls showed no signs of slowing down, I’m starting to lean the other way. The margin for improvement is already vanishingly small, and one of the key drivers of the improvements — shadow zone pitches inside the strike zone — seems like it has plateaued. Moreover, we know that umpiring is getting harder. Pitches are getting faster and moving more, and there are more calls than ever on shadow zone pitches that could go either way. In 2023, umpires missed just under 11 calls per game, and nine of them were in the shadow zone. As long as that red line keeps moving upwards, there’s room for at least a little improvement, but I’m starting to suspect that it won’t be going up for very much longer.