Yes, March Projections Matter in September

Back in March, all we had to look at for the 2021 season was projection; we’re now on the cusp of September, with just over 80% of the games already played. To the consternation of a percentage of fans, projections on sites like this one look unreasonably grumpy to teams like the Giants, who have played above their projections in 2021. It’s undeniable that they have been off for teams, which is something you should expect. But are these computers actually wrong when predicting middling play from these high-achieving surprises going forward?

The Giants, in particular, have already outperformed their original preseason projections by somewhere in the neighborhood of 10 wins. With a month to go in the season, it wouldn’t be a total surprise if San Francisco ended up with 25 more wins than the predictions. Yet our assembled projections only list the Giants with an expected roster strength of .512 over the rest of the season. Using ZiPS alone is sunnier but also well off the team’s seasonal pace through the end of August.

Yes, ZiPS projects the Giants as the slight favorite to win the NL West now, but that doesn’t take a lot of mathematical courage given that they’re already in first place with most of the season done. With the help of a little math, you can see that the computer projects San Francisco to go just 16–16 the rest of the season; to be more precise, ZiPS thinks the Giants have a roster strength that indicates a .521 winning percentage with their average opponent for the rest of the season having a .520 roster strength, resulting in a mean projection around the .500 mark. That’s up quite a bit from the preseason, when ZiPS thought the Giants were a .467 team with a .502 schedule, ending up with a 75–87 preseason estimate. But it almost feels a bit disrespectful to the team leading the league in wins for most of the season.

So why are projections so grumpy? In a nutshell, it’s because they’re assembled based on empirical looks at what baseball’s history tells us, not (hopefully) the aesthetic valuations of the developer. And in the baseball history we have, seasonal projections do not become this all-powerful crystal ball when it comes to projecting how the rest of the season goes.

ZiPS has generated win projections for teams since 2005; not including 2020, we have 15 years of projected standings versus reality, making up 450 teams. To demonstrate the point, I went back and got the preseason projections for every team, the actual records of each team through the end of August in the season in question, and the September records of these teams. One-month projections are going to be volatile, but if preseason projections have limited predictive value over the actual performance, the April-to-August performance should still crush them in terms of predicting the future.

In reality, they don’t. Of the 450 teams, seasonal performance only was the better projection of September performance by a measure of 229 to 221. Using a more robust method than a simple head-to-head matchup, seasonal performance had an RMSE (root mean square error) of 0.107, compared to 0.112 for the original projections. A model of September winning percentage using just these two numbers would consist of a mix of 56% actual and 44% preseason. Using only teams from 2010 to ’19 — 2010 is when I started using a probabilistic playing time estimate, which I feel is superior — ZiPS beats actual in predicting September, 153 to 147, with the RMSE gap shrinking to 0.106 versus 0.108.

Nor did the preseason projections have trouble with the outliers, the teams with the best and worst winning percentages through the end of August:

I highlighted whether the ZiPS preseason projections or the season-to-date winning percentage was closer to the September results. As you can see, ZiPS did just fine with the teams that were absolutely crushing it or absolutely flailing and does a bit worse when we’re talking about the teams most defying their expectations, losing 8–12 against overachievers and 7–13 against underachievers.

The reason this happens is that the biggest piece of information that the seasonal stats have “access to” that ZiPS does not isn’t how the players played, but who the players are. If the projected Braves rotation in March suddenly decides after Opening Day to give up the whole baseball thing and travel from the country in a giant bus, solving mysteries and setting right what once went wrong, the seasonal performance “knows” that the Braves no longer had those pitchers, but the projections do not. The teams that miss their expectations by the most are far more likely to have had significant differences in projected playing time versus actual playing time.

Once you tell ZiPS who actually plays with no update based on seasonal performance of individual players, it turns the table on the April-August results. Preseason ZiPS with known playing time through August comes closer to September performance from 2010 to ’19 by a margin of 166–134, and the RMSE drops from 0.108 to 0.097. The most accurate mix for projecting September over that period would then be 62% ZiPS and 38% April-to-August record.

The advantage of preseason projections even continues into the following season. Taking away again the knowledge of who actually played, ZiPS still did a better job of predicting next season’s winning percentage for teams than the most recent season’s winning percentage, 157–143, with an RMSE of 0.071 compared to 0.078 for the previous year. To sum this up without any math talk: Based on projections and results from 2010 to ’20, you would expect a 2021 preseason team projection to do a better job projecting 2022 than the team’s actual 2021 performance.

(Let me take a bit of an intermezzo here to note that I tested ZiPS because I have access to ZiPS; it isn’t a magical unicorn, and I expect that projections based off of Steamer, THEBAT, etc. will also display this behavior. I just couldn’t test that.)

So why is this true? What you see here is the fundamental difference between Bayesian inference and frequentist inference. Bayesian inference means that we treat the probabilities themselves as unknowns; the frequentist approach would use the actual results as the underlying probability. The latter, using April-to-August as our expectation moving forward, assumes that the Giants have an underlying 64.6% chance of winning games because we observed them winning 64.6% of games. That approach fails to work historically because one of the first things you learn when projecting anything in baseball is that you never actually get to know what the right answer was.

To see this, look at the hypergeometric distribution (not binomial, since there are exactly 2,430 wins in a full season) of wins for a league in which every game is a coin flip.

In other words, with perfect knowledge that every team is precisely a .500 team in an abstract sense, you’d still expect, on average, about seven teams a season to win 89 or more games or 73 or fewer through nothing but sheer luck. That the season itself is a small sample is one of the frustrating things about making projections; you’re guessing at an answer to which you don’t get to find out the answer. If Joe Schmoe hit .300 after getting a mean projection of .270, there’s no divine intervention that tells you if he was “actually” a .300 hitter, a lucky .270 hitter, a really lucky .250 hitter, or even an unlucky .330 hitter. An updated projection will move toward .300 from .270 (ignoring all other factors), simply because it’s more likely that a .300 hitter hits .300 than a .250 hitter does, but not all the way.

One of the fundamentally wrong beliefs about projections is that certain teams or types of players regularly outperform or underperform their projections. But issues of bias are generally one of the first things to wring out in modeling baseball, and one of the places you get the most improvement. Instead, the issue for ZiPS — and, I believe, all other prominent projection systems — is accuracy rather than skew. For the 30 teams in baseball from 2005 to ’20, the year-to-year r^2 of one season’s winning percentage error to the next is 0.000215, meaning that the errors themselves are distributed just about randomly. There are going to be a lot of projections that miss by a mile because projecting the future is hard, but there’s no reason to believe that projection systems are biased against certain types of teams.

(Yes, from a certain point of view, me developing a projection system that has been too positive about the Rockies is a monumental, glorious self-own.)

ZiPS doesn’t hate the Giants or the Red Sox, and neither does their creator. It just has the advantage of knowing that, in baseball history, projections are better when you don’t weigh too heavily what came recently. If San Francisco’s season ends with a championship, the players are just as entitled to have beer-and-champagne showers as the Dodgers were last season. But that doesn’t mean that any of this should have been expected or that the team’s winning ways will continue at the same clip when we look at future accomplishments rather than current ones.

All models are flawed, but some are useful. Projection systems are handy and remain so, even this late in the season, with no updates included whatsoever. Ignore them at your peril.