Descriptive and Predictive Metrics

This morning, Buster Olney took to twitter to offer some more thoughts on WAR, which I think we can surmise is not his favorite statistic. At the risk of making a habit out of responding to Buster’s twitter messages with FanGraphs posts, I did have some thoughts about a few of the things he mentioned, and those thoughts are longer than 140 characters, so I’m putting them here. The subject is worth discussing anyway, and hopefully we can articulate some points about WAR and various other metrics in a way that helps bridge gaps that may currently exist. At least, that’s my goal.

Let’s start off with Buster’s comments:

#1: Teams will invest in players based on OBP, ERA, defense-independent P numbers, home/road splits, left/right splits, etc. They don’t use WAR.

#2: GM said: “I don’t need stats that tell me what happened; I already know that. I need stats that tell me what’s going to happen.”

#3: You’re missing my point, too; if the smartest in the game don’t use WAR, don’t care about it, shouldn’t that tell us something?

Whether or not teams are using WAR in their evaluations is an argument with no resolution – Buster will cite the people he talks to who don’t use it, and I’ll cite the people that I talk to that do, and we’ll just end up in a stalemate. That’s not a point worth debating, I don’t think. However, the overall point that Buster’s making here is worth talking about, I think. The assertion (as I read it, at least) is that WAR – a descriptive metric of past events – is not useful to teams as a predictive metric of future events, so therefore it should have limited value to the public as well. However, I think there’s a bit of an unfair criticism being leveled against WAR here, as it’s being assailed for not doing something that it was not designed to do.

The GM is completely correct in stating that, for his job and the decisions that are required, what matters are metrics that can predict what will happen and not just aggregate what did happen. When trying to decide who to trade for or sign as a free agent, he shouldn’t just look at a list of prior season WAR and get the guy at the top. WAR was not designed as a projection system, but instead, as a retrospective look at what a player produced in a given timeframe.

Any team trying to make decisions about roster construction needs to be factoring in myriad variables about multiple players. Past performance certainly has to be a significant part of the calculation, but so does age, body type, skill set, health, how a player would fit in a particular ballpark… there are numerous things that don’t go into WAR that will significantly effect the expectations of future performance. That’s why people like Dan Szymborski invented systems like ZIPS, so that we could take past performance and adjust for the other factors, coming up with a projected line that tells us more about what may happen than simply looking at past results.

Whether they’re using ZIPS specifically or (far more likely) their own in-house version of a projection system built on proprietary data, I guarantee you that the Smart GM that Buster talked to is looking at that kind of information when deciding which players to acquire, because that’s the tool he needs for help in figuring out how to build out his roster. Projection systems are designed to answer that specific kind of question. Prior season WAR is not.

Projection systems can offer you future expected WAR, and that can be something that is valuable to a GM. But, there are times when we’re not asking about what a player’s future projected WAR is going to be, and so looking at past season data is more applicable.

The post-season awards are the obvious area where you’d want the best descriptive metric possible, and you could really care less about a projection of future performance. When determining who should win the MVP, it doesn’t really matter whether Jacoby Ellsbury or Jose Bautista is more likely to produce similar results in 2012. A projection system would be an incorrect tool to use to answer the question at hand.

Just like a GM shouldn’t focus on prior season WAR when making personnel decisions, baseball writers shouldn’t focus on projection systems when filling out their awards ballots. They are asking two very different types of questions, and there are different tools that help answer different types of questions. Some metrics are predictive, some are descriptive, but let’s not lampoon one for not being the other.





Dave is the Managing Editor of FanGraphs.

82 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Mike G.
12 years ago

Excellent, well thought out, and reasoned piece Dave. Thanks for posting.

Garrett
12 years ago
Reply to  Mike G.

lol