An Annual Reminder About Defensive Metrics

This is now the third consecutive year in which I’ve written a post about the potential misuse of defensive metrics early in the season. We all want as large a sample size as possible to gather data and make sure what we are looking at is real. That is especially true with defensive statistics, which are reliable, but take longer than other stats to become so.

While the reminder is still a useful one, this year’s edition is a bit different. Past years have necessitated the publication of two posts on UZR outliers. This year, due to the lack of outliers at the moment, one post will be sufficient.

First, let’s begin with an excerpt from the UZR primer by Mitchel Lichtman:

Most of you are familiar with OPS, on base percentage plus slugging average. That is a very reliable metric even after one season of performance, or around 600 PA. In fact, the year-to-year correlation of OPS for full-time players, somewhat of a proxy for reliability, is almost .7. UZR, in contrast, depending on the position, has a year-to-year correlation of around .5. So a year of OPS data is roughly equivalent to a year and half to two years of UZR.

Last season, I identified 10 players whose defensive numbers one-third of the way into the season didn’t line up with their career numbers: six who were underperforming and four who were overperforming. The players in the table below were all at least six runs worse than their three-year averages from previous seasons. If they had kept that pace, they would have lost two WAR in one season just from defense alone. None of those six players kept that pace, and all improved their numbers over the course of the season.

The next table depicts the guys who appeared to be overperforming early on. If these players were to keep pace with their early-season exploits, the rest-of-season column would be double the one-third column. Brandon Crawford actually came fairly close to reaching that mark; nobody else did, however, as the other three put up worse numbers over the last two-thirds of the season than they had in its first third.

Just like with the underperfomers, all four of overperformers had recorded defensive marks six runs off their established levels. Replicating those figures over the rest of the season would have meant a two-win gain on defense alone. Again, no one accomplished that particular feat.

A funny thing happened when I ran the numbers for this season. There weren’t any outliers of a magnitude similar to last season or the season before. It’s possible you missed the announcement at the end of April, but there have been some changes made to UZR to help improve the metric.

Here’s the announcement:

For the 2017 season, Mitchel Lichtman has made some improvements to the UZR methodology!

– UZR now uses hit timer data (hang time) rather than hit type designations, which is an improvement on the methodology and thus the results.

– The methodology has changed a little that allows UZR to account for some of the noise associated with imperfect data. The net result of this change is that extreme UZR’s, which were likely caused by, to some extent at least, noise in the data, rather than extreme performance, will be slightly ‘dampened.’ We think that these new values, while very close to the old ones in most cases, more accurately reflect the actual performance of the players in question.

These changes in UZR are currently active for 2017, and will also be rolled out for 2012 – 2016 data in the near future.

-David

To identify potential outliers for 2017, I followed the methodology from previous years, looking at players with at least 3,000 innings from 2014 to 2016 at one position who have also qualified at that position this season. This necessarily eliminates a lot of players, but I wanted to make sure I wasn’t making too many assumptions with the data. When I did that last season, 32 of 56 (57%) had recorded a defensive mark within three runs of expectations and 44 of 56 (79%) within five runs. This season, I narrowed down the list to 44 players. Of those 44 players, 33 (75%) were within three runs of expectations and all but one player (98%) was within five runs.

While we can’t — or, at least, I can’t — say exactly what this means, it would appear that the change in UZR is having the desired effect of limiting extreme UZR figures that resulted from statistical noise. Whether this allow UZR to become a more reliable metric from year to year, more research would be necessary to verify that, but it certainly seems possible.

As for the players this season who’ve exhibited the greatest difference from their previous years of work, I’ve included them below. Let’s start with a quick look at the five players who have a more than four-run difference from what we would expect going into the season, a sample that includes two underperformers and three overperformers.

First, the underperformers. Their current WAR might not completely reflect their skill level if you are looking past the decimal point to determine value.

What we see here are two outfielders who have generally been pretty close to average defensively over the past few seasons, but this year are performing quite a bit below that level. Neither player is particularly young, but neither player is likely one of the worst in the game at their respective positions, as the numbers currently suggest.

On the other side of coin, we have three players who seem to have been overperforming expectations thus far.

Nolan Arenado is quite good at defense, but he’s producing at a rate three times that of his previous three seasons. Justin Upton has been a slightly below-average left fielder the past few years, but his UZR this year would put him among the game’s best. That’s likely to even out as the year goes on. Finally, here’s the thing, Royals fans: you know that -0.8 WAR your starting shortstop is currently sporting? It might actually be overstating his production so far. He’s been an above-average shortstop in his time with Kansas City, but this would put him in Gold Glove territory in the non-Andrelton Simmons category. Despite being a decent-fielding shortstop, there’s a pretty good argument to be made that Escobar is the worst everyday player in baseball.