Umpiring Is About To Get Better
For the last few years, I’ve been checking the accuracy rate of the ball-strike calls made by umpires, dividing the number of correct calls by the total number of takes. It’s a blunt approach, but because umpires make so many thousands of calls each year, it yields solid results. On Tuesday, I pulled the numbers for the 2024 season, and I found something I didn’t expect: Accuracy is going down rather than up. In every single season since the beginning of the pitch tracking era in 2008, umpires have gotten better at calling balls and strikes according to the Statcast strike zone. This is the first time I’ve ever pulled the numbers and seen a lower accuracy rate. However, this is also the first time I’ve checked the numbers this early in the season, and it turns out umpires tend to make better calls as the season goes on. Since 2017, accuracy in March, April, and May has been 0.19 percentage points lower than accuracy over the full season (though the difference in 2023 was just 0.03 percentage points). Here’s what that looks like in a graph.
You know how at the beginning of every season, there are a couple blown calls during a nationally televised game (or at least, calls that appeared to be wrong according to the on-screen strike zone), and certain people start complaining that umpires are terrible and they’re getting worse? Those people always catch me off guard. I usually forget about the missed calls when the season ends, but those people somehow manage to keep their umpire anger at a high idle through the entirety of the offseason so that the instant baseball returns, they’re ready to shout about the umpires again without any need to ramp up. I don’t know how they do it without pulling an oblique, but in a sense, those angry people are right. Even though the umpires are always getting better year after year, they’re nearly always more accurate toward the end of the season than at the beginning — so much so that when the season starts, they’re worse than they were at the end of the previous season. For a month or two, the umpires really have gotten worse. We often say early in the season that pitchers are ahead of hitters. It turns out they’re ahead of umpires too.
For each season, I broke down the overall accuracy in two-month increments, essentially dividing the season into thirds. I also broke down the accuracy during spring training and the playoffs, although there are plenty of factors that make those numbers suspect. During spring training, the umpiring pool is much wider. Perhaps more importantly, there are far, far fewer tracked pitches during spring training, both because the number of games is so small and because not every stadium is set up for Statcast. That results in a much smaller, much less reliable sample. The playoffs are also a much smaller sample, but they’re also, at least in theory, selecting for better umpires. Working the playoffs is seen as an honor and a reward for performing well in the regular season. We should expect accuracy to be at its lowest during spring training and highest during the playoffs.
Generally speaking, the results fit our preconceptions. Spring training accuracy is very low and it features the volatility that we’d expect from a small dataset. Umpires are also more accurate in the playoffs. The red line is March, April and May, and as you can see, it’s nearly always below everything but the spring training line. Not only do umpires start getting better in June, but they keep getting better right through the end of the season, which is why the light blue line for August, September, and October is usually above the yellow line for June and July. The trend is a little bit easier to see if we focus just on pitches in the shadow zone, the area that’s one baseball’s width from the edge of the zone on either side.
In the graph above, the dotted line represents that season’s overall accuracy on calls in the shadow zone. Each data point represents the number of percentage points above or below that year’s average. Not only do the calls get better as the season goes on, there’s a definite gap between the first two months and the rest of the season. Umpires are decidedly worse in those first two months. However, 2023 was a real outlier. It was first time since 2008 that umpires were more accurate in the beginning of the season than the end.
With that, I want to bring you back to 2024. So far this season, umpires have gotten 92.46% of calls right, down from 92.81% in 2023 and just two thousandths of a percentage point higher than in 2022. Based on everything I’ve shown you, we should expect umpires to get better over the rest of the season. However, the drop-off from last year is noticeable. Accuracy over the first two months of the season has only fallen once before, from 2009 to 2010, when it dropped by 0.16 percentage points. So far this season, accuracy has fallen by twice that amount: 0.32 percentage points. That’s a tiny change, on the order of one call per game, but that doesn’t make it any less real. We’ll have to wait and see how the rest of the season goes, but perhaps this year really could end up being different. Or, if it follows the pattern of the past decade and a half, accuracy will soon be in its way up.
Davy Andrews is a Brooklyn-based musician and a writer at FanGraphs. He can be found on Bluesky @davyandrewsdavy.bsky.social.
Good insights, good read, and I appreciate the human element of getting better the more reps that one gets in, but man, seeing the headline on Twitter, I thought for sure this would be about Angel Hernandez retiring.
Yeah, talk about ‘bait and switch’.
I sincerely hope that somebody is currently compiling an absurd number of clips for an equally absurd Angel Hernandez career retrospective.
This would be hilarious
That would be a well-deserved live burial.
Check google. A whole lot have been posted in the last 48 hours.
Good read. I would love to see a hybrid approach to ball/strike calls with ABS/umpires. Let ABS control the east/west portion of the zone (a camera above home plate gets this call right 100% of the time) and let the umpires own the north/south portion of the strike zone. This would greatly improve accuracy but still keep the umps involved.
Nothing gets the call right 100% of the time.
you don’t think a camera above home plate always sees if the ball crossed the plate?
Not 100% of the time. No measurement device is perfect; it always comes down to what the limiting precision is.
E.g., suppose there is a camera mounted directly over the middle of the plate. A strike on the corner is going to show a little bit of daylight between the ball and the black, since the line of sight is not exactly vertical. And the amount of daylight depends on the height of the pitch, which in your scheme is measured by the umpire.
To mitigate this error, maybe you decide to put one camera directly over the outside corner, another over the inside. Well now you have to worry, since the strike zone is 3D, about what part (front or back) of the black the camera sees.
So maybe to solve that problem, you want two rows of cameras placed (somehow?) above and along the full edges of both sides of the plate. Even then, there will be marginal calls in which the camera cannot distinguish the occultation of a seam’s width strike from random noise.
And by the time we reach that point, I’d say the impartial human umpire seems like the more elegant solution to calling balls and strikes.
LOLz
It’ll be 99% accurate.
Good, then we agree that the difference in accuracy between humans and robots is one of degree, not a qualitative shift from fallibility to perfection.
But why? If you’re going to have ABS call a portion of the zone why not all? I say all instead of both because if it’s done as you suggested the camera is actually calling 2 of the 3 planes of the plate (the x and z planes), not just one.
At the moment from what I understand, umps using ABS are involved in the pitch calls at the plate because they’re essentially relayed what the call is via headset. Basically anyone who isn’t aware that the ump is using ABS wouldn’t even realize the difference.
I’m in favor of full ABS, but it seems a surprising number of fans actually like the human umpire element. If MLB thinks fully ABS is a bridge too far then I think this is a nice hybrid model that meaningfully improves accuracy.
And how do you get a camera directly over the plate at, say, Wrigley Field?
We all have our own ways of holding the line against the inevitable robot uprising.
First they came for the umpires, and I said nothing…
As an anti-robot fan, I’d rather have all of the dimensions called by a robot than some subset of the dimensions.
Well, it’s a lot easier for ABS to accurately call the horizontal part of the zone that is precisely defined in the rules and is also the same for every single batter, neither of which is true for the height of the strike zone. Thus, I understand the argument for a partial use of ABS in this manner.
The problem with that is the ball moves in three dimensions. Depending on its depth from the front to the back of the plate, it’s possible it could be a high/low strike but an east/west ball at one point of it’s path, and the reverse later.
I think you might be letting perfection get in the way of improvement.
To be clear, what we have now is far from perfection and I think we all accept that perfection is likely not attainable. I really don’t think this is an issue for 99.9% of pitches.
The rough version of the strike zone we get on TV broadcasts is an improvement, and now every fan at home thinks the Umps have it in for their team when a call is made that disagrees with it.
We have (or are working on) the technology for a fully automated strike zone. I feel like implementing a clearly lacking halfway measure as some sort of compromise is just going to make everyone less happy.
I’m not seeing why anyone should have a preference to “keep the umps involved” at the expense of accuracy and cost.
Because it’s just entertainment. The umpire occasionally being wrong is entertaining. Think professional wrestling, but not quite as intentionally theatrical. Getting all wound up in righteous indignation at the Injustice of something as trivial as professional sports… is kind of entertaining.
And, of course, some people just don’t want things to ever change one iota from the perfect state they imagine it existed in when they were younger.
I never find the inaccuracy of human umpires entertaining. Never. Not once. If it benefits my team, still hate it.
I’m for giving the robots a run until we come to despise them too.
I mean…I’m with you. I’m just saying not everyone is, and it’s a defensible position
This is the primary reason I dislike replay. I find it makes my entertainment product less entertaining. I dislike how it detracts from the excitement of the moment.
Of course, I have a preference for more accuracy but I find it immensely frustrating when we watch a slow motion replay from several angles and the answer is still ambiguous.
Ultimately I believe we’re trading excitement for accuracy, and if it were possible to get to a high enough level of accuracy I guess I’d hold my nose and accept the trade. But where we are currently just doesn’t get me there.
What cost? The umps are always going to be there to make the other calls (out/safe, fair/foul, etc.) and enforce the game in general. Good luck getting the umpire’s union to accept an across the board pay cut just because part of their job has been replaced by machines!
There’s no reason for humans to call balls and strikes anymore.
For some of us, it makes the game more interesting and, for lack of a better word, romantic, to have the organic umpire.
We’re not launching space shuttles here. I don’t need everything in a *game* to be precise and accurate. Baseball admits all kinds of variance that has nothing to do with the athletes, and at least to some of us it is preferable to tolerate (and argue about) the occasional blown call than it is to rely on million-dollar hardware and AI programs in the name of ‘getting it right’, which it still can’t and won’t do 100% of the time.
The justification I can see for going to ABS would require a demonstration that pitchers can locate with greater precision than humans can call balls/strikes, and/or batters can distinguish balls from strikes with greater precision than human umpires can. Only then would we know that variance due to umpire fallibility is actually limiting the quality of the game.
There is nothing romantic about blown calls. It detracts from the game. A player’s skill shouldn’t be undermined by the fallibility of a bad umpire. Everything else you’re saying is convoluted nothingness. All the new system has to do is get more calls right than the current human umps. That’s it. That’s progress. And we have the ability to track that. You don’t need some abstract test of player vs. umpire aptitude. Juan Soto’s player based batter eye doesn’t need to be analyzed against Angel Hernandez’s K zone to achieve any level of understanding here. And I don’t care if Javy Baez thinks he knows the zone better than Laz Diaz. Inconsequential to this discussion. We can implement and track a 3D zone with technology more consistently and with more accuracy than humans. On one hand, you say you don’t need everything in the game to be accurate and precise…only to mention in the same post that you require a demonstration of umpire-based variability to measure the impact prior to change. Contradictory nonsense. We can see with our own eyes how the game is adversely affected by bad calls. And there’s no amount of contrived romance that makes the game better than simply getting calls right.
Since both of us are concerned with the quality of the game, let me try to resolve what you perceived as a contradiction in the post you replied to.
I agree that a minimum level of umpire (whether human or machine) precision is necessary, but I do not find it to be self evident that umpire precision must always be maximized to the best of our technical capability, given the potential for collateral effects on both game strategy and aesthetics. To use my space shuttle comp again, even there the rocket builder doesn’t always use the most precise hardware or implementation available — there is always a notion of minimum requirement, beyond which returns diminish.
(A baseball analogy might be the degree of flatness of the playing surface, which introduces variance due to bad hops. So you need the field to be reasonably flat, but these days few would trade a grass field for astroturf-over-concrete to avoid the occasional bad hop. This is an area where most will accept an increase in variance, due to a non-athletic source, in the interest of the quality of the game.)
So the question before us is whether it is desirable to implement changes to the game in order to increase precision of ball/strike calls. In order to inform that decision, we need to identify the acceptable level of umpire precision, and then ask how that compares to what we currently have.
We can argue about how to determine the acceptable level of umpire precision, but the idea I offered above was based on the idea that we don’t want the noise introduced by umpire to diminish the ‘signal’ from the athlete. So, e.g., if the best control pitcher can locate his fastball only to within +/- 4 inches, then an umpire who measures location to within +/- 1 inch may be adequate. Increasing the umpire precision beyond that might save you the aggravation of seeing the TV computer disagree with the human, but it’s not going to matter, statistically, in the long term.
Fair enough, I don’t agree, but maybe I’m thinking of this too simplistically (while maintaining maybe you’re overcomplicating it). Thanks for the detailed explanation on your end. And I apologize for my prior rudeness in disagreement. Enjoy the game however you want, sir!
Thank you and you too!
I am a week late to the party, and maybe nobody reads this, but this viewpoint is ridiculous. There are no “diminishing returns” here in regard to balls and strikes. What’s the incremental “costs” for each fraction of a percentage of improved accuracy? Did I hear you say “NONE”, because that’s the right answer. Will technology get it 100% perfect? No, but that’s probably an unattainable goal. But if you can go from 95% accuracy, with no significant negatives, why not do it? I’m sorry , but losing the “human element” is not a negative. Losing managers and players arguing with umps and the drama of seeing them ejected and not being able to impact the rest of the game is definitely a negative. I can’t believe anyone would be against getting the call right. Get it right to the highest percentage we can possibly attain. There’s NO RAITONAL argument against this. NONE.
“For some of us, it makes the game more interesting and, for lack of a better word, romantic, to have the organic umpire.”
It’s hard to even imagine these words being written by a baseball fan. I have seen so many games ruined by bad calls. Bases loaded, the slider ends up 8 inches outside and the ump calls it a strike. What does the pitcher do, what does the hitter do? We already know. The next slider is six inches outside but the batter waves at it, strike three. Not interesting, not romantic.
The player and manager are mad, so the ump goes into what I call Defiant Toddler Mode, and maybe throws someone out of the game for being right. (At least 90% of the time players and managers are thrown out, it’s for correctly pointing out that the ump blew the call). Not interesting, and certainly not romantic.
Sorry, I was relying on the ‘for lack of a better word’ to do more work there. It’s not the bad call that is romantic, or how about ‘virtuous’, it is the fact that we can go outside, to a beautiful ballpark, and watch humans play a game that humans have been playing for the past 150 years, temporarily free from intervention by the technology that is overtaking the rest of our lives. The occasional bad call is the price to pay for that benefit, not the benefit itself.
I understand that I am in the minority being willing to pay that price, but it actually kind of hurts to think that might make me seem like not a fan of baseball.
Re ‘interesting’: as is well understood, the robot removes from the game the elements of framing and the game- and season-long efforts to ‘work the ump’ not only by the catcher, but also the pitcher and batter. It replaces those elements with greater precision, but it’s hard to consider as ‘more interesting’ a game that has fewer usable skills and strategies.
It would remove catcher framing from the game and drastically overhaul the value of catcher defense. That could very well cost some veteran catchers their jobs among other things.
Oh well.
Is there a useful measurement of shadow zone consistency among umpires (eg consistently calling balls that are less than one ball width outside rulebook zone to right handed batters a strike?) and rate of calls that are, uh, “very” wrong? Within games and over the season?
Also – gosh – that seems like a really amazing improvement in accuracy since 2008!
Nice insight. But I’m not sure that you’re out of the Angel Hernandez weeds yet. Since the 2024 drop is so very small, could it be the result of a single ump gone bonkers? That is, can you rerun the data taking Angel Hernandez’ calls out of the mix and check the difference?
I’m going off memory here, but I think that Angel is really only 3rd-4th worst as measured by ball-strike call accuracy. Most of his “iconic” moments involve his other behaviors (even his reactions to other people’s reactions to his strike calls), not ball/strike calls themselves.
More interesting questions might be if newer generations of umpires are better than the old timers like Angel, and if there is a rookie effect where newly promoted umps are less accurate for a few months of learning curve.
Yes — as long as we’re on this tangent, it’s been extremely odd to see how the media valedictions are all covering for Angel Hernandez’s reputation by focusing entirely on his strike-zone accuracy, as if that were the problem. This is of course a complete red herring, since even though he was always a bad strike-zone ump it was obviously not the reason people thought he was the worst ump, and it feels almost like it’s been introduced as a deliberate distraction or straw-man argument (though it’s probably just as much a problem of “this is the thing we have good numbers to measure, so it must be the thing we want to talk about”).
Barry Svrluga of the WaPo wrote an entire column about Angel Hernandez’ sin being the Ump Show, contrasting with CB Bucknor (also bad, but not a show) and Jim Joyce apologizing publicly when his bad call ruined Andres Gallaraga’s perfect game
https://www.washingtonpost.com/sports/2024/05/28/angel-hernandez-mistakes/
Ehhhh. C.B. Bucknor’s supercilious smirking is his own spin on Ump Show. Let’s not get it twisted.
Balls and strikes are the calls I give umpires the most leeway on. It’s the hardest part of the job and has the most distracting features.
What made Hernandez terrible, beyond the fact that he was not a good ball/strike umpire, was everything else. The calls on the bases, check swing calls, being conflict-seeking, etc. To me, his worst call of all time was the Rosales home run he refused to flip in 2013. Clear video evidence, no other way to interpret what happened, extremely important play in the game, and he refuses to change an obviously incorrect call. The theory in 2013 was that it was bc video replay was new, and this was his protest against being second guessed. That’s just not a good attitude to have.
I personally am of the theory that Hernandez kicked calls on purpose to get back after perceived slights, in addition to organically missing calls like every umpire is apt to at time. Not something you can have in the game.
Curious if you’ve dug deeper into the data. Was there something driving that drop in 2023? Maybe a pool of new umpires? Or a certain cohort of umpires who’s names might rhyme with Wangel Mernandez who have such bad outliers that they’re skewing the sample.
Just noticing that I didn’t see the comment right before mine…
Am I missing something? If the premise is true, why would playoff umpiring start back at square one around 84% accuracy? Am I reading the graph wrong?
yeah, you’re reading the graph wrong. Each colored line is a different 2-month period (and ST and playoffs) of each season since 2008. Each line generally being slightly higher as the season goes on indicates that umpires get slightly better as the season goes on.
The upward trend of all lines indicates the much bigger effect, which is that umpires have gotten much better since 2008.
With the technology to automate the ball and strike calls to within 99% there’s no sound reason for ball strike accuracy to even be an article topic.
“You know how at the beginning of every season, there are a couple blown calls during a nationally televised game (or at least, calls that appeared to be wrong according to the on-screen strike zone), and certain people start complaining that umpires are terrible and they’re getting worse? Those people always catch me off guard. I usually forget about the missed calls when the season ends, but those people somehow manage to keep their umpire anger at a high idle through the entirety of the offseason so that the instant baseball returns, they’re ready to shout about the umpires again without any need to ramp up. I don’t know how they do it without pulling an oblique, but in a sense, those angry people are right. Even though the umpires are always getting better year after year, they’re nearly always more accurate toward the end of the season than at the beginning — so much so that when the season starts, they’re worse than they were at the end of the previous season. For a month or two, the umpires really have gotten worse. We often say early in the season that pitchers are ahead of hitters. It turns out they’re ahead of umpires too.”
Please hire an editor.
Maybe he could have split that block into two paragraphs, but otherwise what’s so bad about his writing here?
From what I’ve seen, FanGraphs is more interested in allowing writers to “write what they want” instead of strictly following every rule.
On the other hand, after editing non-professional writers myself the past 23 years, there is only so much you can do. A heavy edit makes a piece more about the editor than the writer.
Contributors here are always going to be more “analytical personalities” than professional writers – and that’s okay.