Looking Back at the 2022 ZiPS Projections

Before we get to the 2023 ZiPS projections, there’s still some unfinished work from 2022 to do. Looking at which projections went the most wayward is definitely an exercise that makes me soul cringe a bit, but in any model, being wrong is an important component of eventually being right. Calibration is a long-term project, and while chasing greater accuracy in mean projections isn’t likely to result in any huge bounty — there’s a reason projection systems are so tightly clustered — there’s still improvement to be had in things like calibrating uncertainty and long-term data.

Let’s start with how teams performed versus their projections:

Teams have gotten a bit more polarized in how they’re run in-season. Looking at the in-season ZiPS projections, roster strength has varied much more in recent years than when I started doing this. It wouldn’t be surprising to see the mean absolute error — for an exercise like this, I want to use the simplest tool that gets the point across — creep up over time. That is the case here, as the MAE of 8.3 wins is above the ZiPS historical average of 7.5 (not including 2020). ZiPS underperformed its usual matchup vs. Vegas, only going 17-13 in over/unders as of the date of release (April 6); historically, ZiPS has averaged 19-11.

What would a perfect projection look like? If you knew the win probability of every game played before it was played, the MAE of your seasonal projections, based on the hypergeometric distribution, would be around five wins.

Thirty-six percent of the estimated error in ZiPS was simply guessing wrong on roster construction. With perfect knowledge of who actually played, the MAE drops to 7.1 wins. The biggest gains in accuracy would have been the Boston Red Sox (88 wins projected, 78 knowing plate appearances/total batters faced), the New York Mets (88, 96), and the Cleveland Guardians (78, 84). Amusingly, the Orioles were one of the four teams ZiPS would have actually missed on by a larger amount if the system had known playing time information ahead of time. Simply put, ZiPS would have been even more negative about the O’s if it had known what their rotation would look like!

With 10 deciles and 30 teams, you ideally want three teams to finish with a percentile projection in each decile. In other words, you want three teams to fail to meet their 10th-percentile projection, three teams to at least hit their 90th-percentile projections, and so on. So how did ZiPS do?

Amusingly, ZiPS got four of the six divisions in the right order, with the AL Central error being Cleveland actually finishing first instead of third, and a real mess in the AL East (four teams projected at 88 wins and the O’s were the biggest out-performers in baseball). There was no change in the year-to-year correlation between errors; there’s no significant relationship (an r-squared of -0.0013) over the entirety of ZiPS’ history. This is also true on a per-franchise level. What this means is that there’s no tendency for errors one year to predict errors the following year. This holds true even on the team level, with the Mariners having the highest r-squared in season-to-season errors at 0.02. So, no, there’s no reason to think the 2021 Giants uncovered a special ability to outperform projections; they merely outperformed projections in 2021.

For hitters and pitchers with at least 300 plate appearances or batters faced, the MAE for OPS+ was 16.3 points, while for ERA+, it was 20.9 points. Both of these are consistent with the results over the last decade (16.4 and 21.1, respectively). I actually expected to do a little worse than normal.

Here are the hitters the system most underestimated:

I believe that the miss for Albert Pujols is the largest ever in ZiPS for a player 35 years or older. And I am very happy to have missed on that one! I think it’s unfortunate that we have a player who is a slam-dunk, first-ballot Hall of Famer, but who basically spent the last decade of his career hitting like Darin Ruf instead of Jimmie Foxx. There are a lot of younger baseball fans whose only experience of Pujols has been that of a mediocre 1B/DH. While 2022 wouldn’t challenge any of the classic seasons from Pujols’ first decade of play, getting one last burst of glory was really cool to see and lifted his farewell season above the sad trudge of Derek Jeter’s endless goodbye in his final campaign.

The unexpectedly excellent performance of Michael Harris II had a greater impact on his future projections than any of the other top performers here. Given his young age, there was a lot of uncertainty about his upside. He was actually considerably more likely to hit his actual OPS+ (135) than any of the other 14 players listed. Now that he’s apparently hitting that upside, his 2023 ZiPS projection has seen a massive bump. With only 154 plate appearances, Matt Carpenter did not make this list, but he was the most unexpected player overall in 2022, with the projections only giving him a one-in-2,300 chance of hitting as well as he did, even in a relatively small sample.

Now here are hitters who most underperformed their projections:

Three White Sox made this list, with two of those three expected to be among the team’s most valuable players heading into the season (I’ll let you guess the odd man out). It’s not surprising Chicago’s offense struggle to score runs in 2022. One interesting thing from the larger dataset, which you can see in some of the names above, is that being a Three True Outcomes player had a slight effect on projection accuracy in 2022, one that hadn’t been present at any time in the past. I’m definitely curious to see if this development persists, assuming we have the same deadened baseballs we appeared to have this season. Now let’s turn to the pitchers:

I think half of the Dodgers starting rotation made this list, which is hardly surprising given that the Dodgers outperformed their already excellent projection by only one fewer win than the Orioles did. Andrew Heaney wasn’t far off from making this group, either, and that’s with ZiPS already being more optimistic about the left-hander than most. If you bet on Matt Moore being a highly valuable relief pitcher, pat yourself on the back; ZiPS sure didn’t! Now for the under-performers:

Hey look, more White Sox! Given how poorly so much of their team did relative to the projections, I’m actually a little surprised that ZiPS only over-projected them by seven wins (88 vs. 81). I agreed with ZiPS that Dallas Keuchel would at least be a serviceable fifth starter this season, which absolutely did not happen! That said, I didn’t agree with ZiPS’ relatively friendly projection for Patrick Corbin, so take that, set of algorithms I developed!

Let’s finish up with the calibration data for pitchers and hitters. This is actually something I focus on a lot more than the mean projections. Models aren’t right, they’re useful, and a properly calibrated projection system should know how often it is expected to be wrong.

Next up: the comically preliminary ZiPS projected team standings for 2023!