Last Year’s WAR with True-Talent Defense

I’ll begin by saying I’m not sure what value all of what’s to follow actually has. I know that’s about the least-compelling way to begin a blog post, but I just want that to be very clear. TangoTiger’s recent Building a Better WAR Metric series on the site jumpstarted an idea I’d been kicking around in my head for a while. It’s an idea that mostly exists because I’ve seen people on the internet say they’d want to see something like it, and, at the very least, it could serve as a talking point for another constructive discussion about WAR, and any constructive discussion about WAR is a good thing, because we all admit it’s far from perfect and constructive discussions usher progress.

People don’t really have beef with the offensive side of WAR, I don’t think. As far as sabermetric stats go, wRC+, and therefore wRAA, are about as infallible as they come. Tough to argue with the outcomes of history. I don’t see too many quibbles with the base-running numbers, partially because I think most people think they do a good job, but also because they don’t move the needle much either way, and there’s bigger fish to fry. Some people aren’t fans of the positional adjustments — both the assigned weights, and the entire concept of including them. I’m in the camp that firmly believes in the idea of the positional adjustment, but, like anything, the formula for the weights could always be looked at to see if it could be improved in any way, and Jeff Zimmerman’s work on this topic last offseason was a great place to start.

But the bigger beef, beyond the positional adjustments, is of course defense. Anyone will admit this is the weak link of WAR. It’s probably the weakest link of sabermetrics, as a whole, in 2016. And mostly, what it boils down to is, we know that defensive metrics don’t stabilize until the sample spans roughly three years, or 3,000-ish innings. Meaning, the single-season data is subject to noise, and if we wanted to draw any conclusions from it, we’d have to regress it. Despite that, single-season WAR is powered by noisy, unregressed, single-season defensive metrics. That’s the crux of the beef with WAR.

So, some folks have suggested that the defensive component of WAR ought to be regressed in some way, in an effort to strip out some of the noise that comes with single-year defensive data, or to better capture a defender’s true performance. I think there are a number of flaws in this general line of thinking, but there are a number of flaws with the way it’s being done now, too, so let’s humor one another.

Both ZiPS and Steamer use multiple years of data, giving more weight to the most recent seasons. Multiple years of data to weed out noise? Check. Both also incorporate some form of “scouting” information: Steamer regresses toward the results of the Fans Scouting Report, ZiPS searches for keywords in actual, physical scouting reports and uses those as a means for regression. Eye test? Check. Blend all that together and factor in some aging curves, and you’ve got yourself as good an idea of any player’s true-talent defensive ability as you’re going to find. Sort the Fld column here and I think you’ll agree that these numbers pass the eye test with flying colors.

So let’s imagine a world where, last year, every player performed exactly to their true-talent defensive ability. Everyone hit the same, everyone ran the same, everyone had the same amount of playing time, but defensively, we knew exactly what everyone’s true-talent ability was worth, and no one varied from it.

In that world, last year’s WAR leaderboard, in sortable, scrolling fashion, would look like this:

What can we take away from all this? Let’s get through some of the nerdy stuff first. The mean of the changes evens out to 0.0, and the standard deviation is 0.6 WAR. The changes are normally distributed, and a little more than two-thirds of all players fall within 0.5 WAR of their actual 2015 WAR. We know that WAR comes with error bars, and that, for example, 2.5 WAR and 3.0 WAR aren’t always all that different. The lack of certain in defensive metrics is most of the reason why, and this reinforces that the error bar for any player is likely within one win. Only six players out of 139 see their WAR change by more than two standard deviations — or more than 1.0 WAR.

Gerardo Parra is our only true outlier. His 2015 value, with estimated true-talent level defense, would have been +2.7 WAR. His actual 2015 value, according to WAR with UZR, was only +0.4 WAR. The 2.3-WAR difference is four standard deviations above the mean, which is crazy. Clearly, something is up here. For context, UZR had Parra pegged at -18, while DRS had him at -10. BaseballProspectus’ FRAA had him at just -0.5. Perhaps UZR was overly harsh on Parra last year. Perhaps the projections are being too generous with his current true-talent ability. For what it’s worth, I sought clarity on Parra’s defensive numbers back in November, and found some evidence that suggested Parra’s true-talent level has declined some, though I can’t objectively say whether that means I’d take the over or under on the estimation of the projections.

The next-largest gap between 2015 value and expected true-talent value belongs to Kevin Kiermaier, who comes in at +4.1 WAR with true-talent defense, and +5.5 WAR with 2015’s figures. UZR had Kiermaier pegged as a +30-run defender last year. And while that’s obviously not his true-talent level — we’ve got that estimated at +18 runs, which is still remarkable — I think this discrepancy highlights the issue with the line of thinking prevalent in this post. UZR had Kiermaier at +30, FRAA agreed at +31, and DRS thought those were both light, putting him at +42. Every system available to us suggested Kiermaier’s defense was worth north of three wins last year, yet we know that’s not his true-talent ability. Just like how Bryce Harper posted a 1.109 OPS, yet we know that’s not his true-talent ability.

Kiermaier robbed something like half-a-dozen home runs or more last season. Part of that is because he’s incredible, but he also probably got somewhat lucky in having more balls than usual hit to the exact sliver of space in which a fielder has the opportunity to a rob a home run. We’d never expect him to rob half-a-dozen or more home runs in a year moving forward, but he did, and he ought to get credit for that. Just like how a batter’s BABIP can be inflated by a bunch of balls being hit to just the right spot, a fielder’s defensive metrics can be inflated by a bunch of balls being hit to just the right spot that allow him to make a plus play.

On the flip-side of that coin, consider Juan Lagares, who doesn’t appear in the table because he wasn’t qualified, but whose fielding ability was valued at +3.5 runs last year, and whose true-talent fielding ability would’ve been +7.9 runs. We know Lagares is an excellent fielder, but we also know he was dealing with elbow problems that limited his greatest defensive asset — his arm. In other words, it would be silly to use anything like Lagares’ true-talent defensive numbers for last season, given we have reason to believe Lagares was performing well below his true-talent ability, just as we have reason to believe Victor Martinez was performing well below his true-talent ability at the plate, for the same reasons.

I don’t think it’s all bad, though. Aside from the outliers, this could help clean up plenty of guys’ defensive numbers. The scouting and eye test data we have seems to suggest Mookie Betts may be an even better defender than he was given credit for last year. Plenty of people seemed to think Yoenis Cespedes‘ value was boosted due to overly-optimistic numbers on his defense, and his estimated true-talent agrees and docks him a win.

This methodology has its flaws. Our current methodology has its flaws. I’m certainly not suggesting this is how things should be done — I think that would be a step backward, in fact. But more information can never hurt.