How Should You Interpret Our Projected Win Totals?

Last week, we published our playoff odds for the 2023 season. Those odds contain a ton of interesting bells and whistles, from win distributions to chances of receiving a playoff bye. At their core, however, they’re based on one number: win totals. Win totals determine who makes the playoffs, so our projections, at their core, are a machine for spitting out win totals and then assigning playoff spots from there.

We’ve been making these projections since 2014, so I thought it would be interesting to see how our win total projections have matched up with reality. After all, win total projections are only useful if they do an acceptable job of anticipating what happens during the season. If we simply projected 113 wins for the Royals every year, to pick a random example, the model wouldn’t be very useful. The Royals have won anywhere from 58 to 95 games in that span.

I’m not exactly sure what data is most useful about our projections, so I decided to run a bunch of different tests. That way, whatever description of them best helps you understand their volatility, you can simply listen to that one and ignore everything else I presented. Or, you know, consider a bunch of them. It’s your brain, after all.

Before I get started on these, I’d like to point out that I’ve already given our playoff odds estimates a similar test in these two articles. If you’re looking for a tl;dr summary of it, I’d go with this: our odds are pretty good, largely because they converge on which teams are either very likely or very unlikely to make the playoffs quickly. The odds are probably a touch too pessimistic on teams at the 5–10% playoff odds part of the distribution, though that’s more observational than provable through data. For the most part, what you see is what you get: projections do a good job of separating the wheat from the chaff.

With that out of the way, let’s get back to projected win totals. Here’s the base level: the average error of our win total projections is 7.5 wins, and the median error is 6.5 wins. In other words, if we say that we think your team is going to win 85.5 games, that means that half the time, they’ll win between 79 and 92 games. Past performance is not a guarantee of future results, but for what it’s worth, that error has been consistent over time. In standard deviation terms, that’s around 9.5 wins.

It’s an unsurprising error, because baseball is a game of uncertainty and probability. This isn’t basketball, where outcomes feel preordained; sometimes your bad hitter pops a homer, and sometimes your ace gets shelled. Take a look at the distribution of our odds themselves for a demonstration of this. In those simulations, talent level is fixed, and yet teams’ success rates vary markedly from one run to the next. The median absolute error from one run to the next is about four wins; in other words, even if you knew exactly how good every team was and they played at exactly that level, there’s a limit to how precise your predictions could get.

One useful takeaway from this: the number we’re reporting is more of a probability cluster than a point estimate. When we project a team for 85 wins, we’re saying that they’re an 85-ish win kind of team. You probably have a rough idea of what that means in your head. You probably also have a rough feeling that sometimes teams that look like 85-win teams win 90 games, or 80 games. Heck, sometimes they win 95 or 75 games. Our odds have even a bit of additional variance around that feeling, because they’re estimating how good a team will be before the fact, but the general concept applies.

More specifically, here’s a look at the distribution of misses:

The difference between our projected win totals and actual team win totals has a slight rightward skew; the most frequent outcome is that our model predicts two to four more wins than a team actually achieves, and the median is ever so slightly negative (-.25, to be precise). I don’t see an obvious explanation for that, but it’s not a huge effect in any case. For the most part, our projections on the whole have normally distributed misses.

With that out of the way, let’s get into specifics. I broke projections out into win buckets to see if they have any obvious bias based on team talent level. I started out with two-win buckets and looked for our average miss in each bucket:

As far as I can tell, there’s not much of a pattern here. Plenty of buckets where we missed low sit right next to buckets where we missed high. Teams we projected for 80–82 wins performed nearly two wins per year worse than that, and teams we projected for 82–84 wins outperformed their projections by more than a win. It’s all a big zig-zag.

To zoom out slightly, I bucketed wins in tens instead of twos. There, a clearer pattern emerges, and it’s a logical one:

Teams that we think will be awful are indeed awful, and they’re even a little worse than we think they’ll be. Likewise, teams that we think will be very good are indeed very good — better than we projected on average. There’s a clear reason for this: trades. Bad teams tend to trade their good players. Good teams tend to trade for good players. We can’t account for those trades in preseason projections, so that natural drift makes sense to me. In fact, I’d be surprised if it weren’t there.

That’s the extent of the serious look I took at our data, but I did parse the data up one more way just for fun. You know how FanGraphs always hates your team, regardless of which team that is? Well, if you root for the Astros, you might just be right. We’ve missed on our Astros win projections by a lot: 6.4 wins low on average, to be precise. We’ve also been low on the Brewers, Dodgers, and Cardinals by roughly five wins each. A lot of that comes down to what I was talking about above: good teams tend to add during the season, and the four teams we’ve been lowest on have been good for most of the window for which we’ve had projections.

You might think we’re always low on the Rays, what with their front office made up 2/3rds brain surgeons, 1/3rd rocket scientists, and a bonus 1/3rd former FanGraphs employees. Not so much: we’ve been low by slightly more than two wins on average, which is middle of the pack in terms of absolute error. The A’s are another team that people frequently mention as smarter than the projections — but they’re the team we’ve projected best in our data set, at an average of 79.95 wins, and they’ve won an average of 80 games per season.

On the other side of the coin, Tigers fans might be angry with FanGraphs for giving them too much hope. We’ve missed by 6.6 wins per season, and not in the good way; they’ve averaged only 71.2 wins over the eight seasons I considered, and we’ve projected them for 77.8. We’ve also been far too high on the Padres, Nationals, and Reds (and yes, there’s some midseason trade action in here too).

So what do FanGraphs projected win totals mean? I’d treat them as a rough measure of the major league franchise’s prospects in the coming year. Angry about your team being projected for 86 wins instead of 88? I don’t think our projections are amazing at doing that kind of fine parsing, and I think the architects of the projections that feed into our model would agree. Angry that we projected your team for 72 wins when you think they’re a playoff contender? Well, that’s not the kind of thing we miss very much.

More specifically, 88% of our projections get within 15 wins of a team’s actual total. Only 7% of teams in our entire sample outperformed by 15 or more wins. That’s not to say it’s impossible — 7% is more than 0%, obviously — but it’s a reminder of gravity. If our playoff odds and projected win totals think your team is bad, it doesn’t mean they 100% are. But it does mean that most teams who project similarly to them have been bad.