Does Projected Team WAR Actually Mean Anything?

I think it’s safe to say we lean pretty heavily on projections here. Now, it’s important to understand we’re all always kind of making projections. The Padres acquired Wil Myers on the basis of a positive internal player projection. When we think about our favorite teams adding, say, Dee Gordon or Nelson Cruz, we’re considering what we expect them to do in the season or seasons ahead. Our enthusiasm for the coming year is based in part on a mental projection of the quality of our team. We all project, and the only real difference is that, around here, we lean on the projections by Steamer and ZiPS, instead of doing things in our heads. FanGraphs makes things really easy. What do the projections think about next year? There’s a tab you can click on. It’s a starting point.

But while projections are handy, it’s only natural to wonder: do they matter? How important are they, actually, with regard to predicting the short-term future? Tons of people have tested individual player projections, but here we also include team projections, based on manually-updated depth charts, and if there’s error in each given player projection, how much error might we see with team projections as a whole? It’s a perfectly reasonable question. It can’t even be answered conclusively, yet. There’s not enough data in the FanGraphs post archive. But I can give you at least a little bit.

We’ve run Positional Power Rankings for three years now. They’ve been done in March, and they provide projected WAR by position, and then if you add those positional WARs up, you get a WAR projection for a team. It’s easy enough, then, to compare projected team WAR to actual team WAR. So that’s what I’ve done in the graph below. Note that, however, I’m only including data from the last two years, because three years ago the Positional Power Rankings were based only on ZiPS, and the depth charts were different, and it seems like the projections were estimated. In 2013, we started using the current method, so it seems natural for the sake of consistency to only consider information generated by the current method, blending ZiPS and Steamer and relying on our internal depth charts.

Of course, team projections have been done for years, based on different methods. They stretch back at least to the early 2000s. Again, I just want to examine results from the way we do things here now. Which gives us only 60 team data points, but, it’s something. I presume that projections have gotten better every year, so I’m not even that interested in how well we could predict baseball in, say, 2006.

Last note: to remind you, this is based on stuff generated in March, blending two projection systems. On FanGraphs right now, you’ll only see Steamer projections, because we don’t have full ZiPS data yet. And, of course, rosters are still changing, and Max Scherzer is still a free agent. More uncertainty! But anyway, here’s the damn image.

If you eyeball it, there’s a clear, linear relationship. Worse projected teams have generally been worse actual teams. Good projected teams have generally been good actual teams. That right there is enough to say, yeah, there’s value in what’s provided. The projections aren’t telling you nothing. But you’ll notice, also, that there are some sizable gaps between the data points and the line. That’s to be expected. Certain things are unpredictable, like injuries or trades. And we don’t want to nail this 100%, not that that would even be possible, because then, what’s the point?

A brief, easily-consumable table:

Averaged out, things look good. Teams have stayed around their own groups. But deviations are real and sometimes big. Out of the 60-team sample, about half have ended the year within 5 WAR of the pre-season projection. A full 85% have ended the year within 10 WAR of the pre-season projection. But, that represents a span of 20 wins. It’s pretty obvious that a team can stray wildly off course, in a good way or in a bad way.

Over the two seasons, 20 teams have made the playoffs. I’m counting the wild-card games as the playoffs. They’ve averaged 37 projected WAR. Of the 20, 14 had at least 35 projected WAR. A total of 19 had at least 30 projected WAR, where the actual minimum in there was 32.7. But, the 2013 Indians played an extra game after entering the year with 27.6 projected WAR. So there’s the floor, so far. The Indians have had the lowest projected WAR of a playoff team. Then there’s a gap of more than five wins until the next-lowest projection for a playoff team, but this just proves you don’t have to project that well to actually do something. This is why the White Sox have a real shot. Maybe not so much the Padres, but, who knows? They’re not done.

I’m sure you’re curious about the biggest misses. The biggest whiff was the 2013 Red Sox, for whom just about everything went right. The Sox were projected for the 11th-best WAR, but then they beat it by almost 22 wins. That’s double the next-biggest miss, in the positive direction. Now that I think about it, “whiff” doesn’t feel right — this doesn’t demonstrate the projections were wrong. They were just exceeded. The Red Sox deserved to win that World Series.

And the 2013 Orioles beat their projected WAR by just over 11 wins. This one’s fairly simple to explain — Manny Machado projected well, but he didn’t project as one of the most valuable players in baseball. And Chris Davis projected a lot worse than Manny Machado, and yet he wound up even more valuable, at least by wins above replacement. Sometimes, players break out. Sometimes, they make immediate impressions.

At the other end, the biggest negative miss was the 2013 Phillies. That team fell almost 19 wins short of its projected WAR. Most importantly, that’s the year Roy Halladay went from Roy Halladay to basically retired. Jimmy Rollins had a down year, and Carlos Ruiz had a down year with a suspension mixed in. Delmon Young played a lot. Outside of Chase Utley, Cliff Lee, Cole Hamels, Domonic Brown, and sort of Jonathan Papelbon, no one really held up their end of the bargain. I guess Jake Diekman was good. Whatever. That turned out to be a bad team.

And the 2014 Rangers fell 17 wins short of their projected WAR. Simple explanation. This wasn’t really the fault of the projection systems — they couldn’t have known every single player on the roster would get tuberculosis. Injuries slaughtered the Rangers like no other team in recent memory, at least that I recall, and that’s an unfortunate break. Several unfortunate breaks, really. The team’s a useful reminder, though. We can’t really project injury problems, and certain injuries can muck everything up. Injuries can dramatically swing team and division outlooks, and while individual injuries are seldom that important, boy can they ever pile up quick.

And oh, by the way, WAR, of course, isn’t a perfect predictor of actual wins. I made this a few weeks ago:

Teams are always trying to maximize their WAR, even if they don’t think of it as WAR. But all WAR does is serve as true talent. On the way from talent to record, you run into monsters like random sequencing. Sometimes you have runners-in-scoring-position luck. Sometimes a closer picks the absolute worst times to hang a few sliders. Some recent editions of the Orioles have been super clutch. Last year’s Royals were super clutch. Maybe some of that has been by design. Not all of it has been by design. Projected WAR + breaks = Actual WAR. Actual WAR + luck = Actual Record. Did you know that you can’t predict baseball?

Ultimately this teaches you nothing you couldn’t have already guessed: the projections we have are fine, and they can generally identify good and bad players. Teams with more good players project as better teams. Teams with more good players end up as better teams. There’s also a lot of noise, such that we don’t actually ever know who’s going to have the best record when we’re looking at things in December or March. Yet people have been requesting something like this, so I think it’s worth the occasional reminder that what we have is functional, as a starting point if nothing else. Look at the projections, and go from there. They’re not trying to mislead you. But there are reasons they play the actual games, and it’s not just to make people money. Although it is in large part to make people money. But that’s a different conversation.