10 Years of Team Performance, 10 Years of Team Projections

I’ve got a post going up at JABO today. It might’ve already gone up, I don’t know — I’m writing this in advance, and I wrote that post in advance, and I’m not in charge of when they publish the content that’s been sent their way. But today, there is this post and that JABO post, and within the latter, you will see the following image:

actualprojectedwins_2005_2014

You’re smart enough to get this. That’s a whole decade of information, showing single-season projected team win totals, and single-season actual team win totals. You observe both a relationship, and a fair amount of noise. I suppose I don’t need to spoil that other post. Read that other post, whenever it’s published. Read that other post twice! Click click click!

Yet seldom do we dig into historical team projections. There are reasons for this, but now that I’ve gone to the trouble of gathering all the necessary data, there’s some other stuff we can do. There’s a lot of other stuff we can do; I have done a small amount of it.

As in the JABO post, I need to acknowledge the issues, here. The bulk of the information was supplied by Replacement Level Yankees Weblog, which has a history of preseason team-projection blowouts. They date back to 2005, and they blend multiple projection systems, through a simulator. ZiPS has been included, and PECOTA has been included, and so on and so forth. But between some years, some different systems have been selected. The systems themselves have changed as we’ve learned more and more about how baseball and forecasting work. It stands to reason that the projections will be worse the further back you go.

But I think it’s still OK to proceed. You see the relationship in the first image. And, really, projections haven’t changed that much over the years. They’re still pretty good, and they’re still all imperfect. Systems haven’t improved by leaps and bounds. One system will mostly agree with another system, such that you don’t usually see one team being either good or bad based on two different forecasts. There are a lot of projection systems. They do basically the same thing, using basically the same numbers.

We can accept there are some issues and still play with the data. There are some issues with WAR, but that doesn’t stop us from playing with WAR. So, acknowledging the issues, let’s get to the fun stuff. The first thing that occurred to me: compare team performance to projected team performance over the decade. So, over a combined 10 years, which teams exceeded their projections the most, and which teams most dramatically fell short?

actualprojected20052014

Leading the way are the Angels, who, since 2005, have topped their projected win total by 47, or nearly five a year. Right behind the Angels are the White Sox, at 43, and Dave actually wrote about this two years ago. He gave a lot of credit to Don Cooper helping to keep starting pitchers healthy. There’s a pretty substantial gap between the second-place White Sox and the third-place Tigers.

One standard deviation here is 21. There are six teams who fall outside of that range, and the most extreme data point is that of the Cubs. The Cubs show up at -55; next-worst are the Mariners, who are better by 19. I don’t know exactly how to explain this, but what’s happened has happened, so if you’ve felt like the Cubs have mostly been underachievers, this is support for that. The Mariners, incidentally, are mostly where they are because of the crippling disappointments of 2008 and 2010. That the Red Sox and Rockies also show up pretty strongly negative makes you wonder about potential park effects. I don’t know, it’s complicated. I’m just trying to show you stuff, not necessarily explain it.

That looked at the last 10 years. What if we just looked at the last five years?

actualprojected20102014

Now the Orioles show up. And while the Orioles are at +26 wins over five years, they’re actually at +44 wins over three years. There’s been nothing quite like their 2012. Of some interest, look at the Angels and White Sox. They were the leaders in the previous graph, but here they fall way back to the pack. The White Sox are actually at -1. They did their projection-beating years and years ago, for the most part.

At the bad end, some familiar names. The Rockies have badly underachieved. The Cubs’ pattern hasn’t disappeared. The Red Sox, of course, have been somewhat bipolar, and as for the Marlins, they were a mess, relative to expectations, in 2011 and 2012. The Marlins had a particularly fun five-year stretch:

  • 2008: +16 wins
  • 2009: +15
  • 2010: 0
  • 2011: -10
  • 2012: -15

Looking at just the last three years — I didn’t prepare an image, but, the leaders are the Orioles, A’s, and Pirates. Bringing up the rear: the Rockies, Red Sox, and Blue Jays. You can definitely see the effect health has, and over smaller samples, that doesn’t get much of an opportunity to balance out.

Real quick, before I go, let’s take a look at sustainability. If you beat a projection in Year X, does that mean anything for Year X+1? If you fall short of a projection in Year X, does that mean anything for Year X+1? Spoiler alert: no, basically. On the x-axis: actual wins – projected wins, Year X. On the y-axis: actual wins – projected wins, Year X+1. I don’t think I need to show you the r-squared value.

projectedactualwins_yearx_yearx1

Take all the teams that beat their projections by at least 10 wins. In Year X, they averaged a difference of +14 wins. In Year X+1, they averaged a difference of 0 wins. Now take all the teams that fell short of their projections by at least 10 wins. In Year X, they averaged a difference of -15 wins. In Year X+1, they averaged a difference of 0 wins. It’s not conclusive, but I don’t know what else to tell you. The evidence suggests this is magic, and baseball teams aren’t magical.

Of course, this probably comes off like a shot at the Orioles. It’s not intended that way; I’m just following the numbers. Because the Orioles have been better than expected for three years in a row, for the most part, consensus is mounting that there’s just something special about them. And I’m totally willing to concede that it’s possible, but it doesn’t have to be 0% signal or 100% signal. It can be signal and noise, and I don’t think the Orioles should bet on consistently overachieving by several wins. In recent history, it’s been a difficult thing to pull off.

The Angels did it, right until they stopped. The Giants have done it, but not in the odd years. The Marlins had those back-to-back years I showed you above. Take the 2008 – 2010 Twins: they overachieved by a combined 31 wins, then they underachieved by 21 in 2011 alone. I’ll remind you of the earlier White Sox example. This isn’t me being stubborn; this is me just not finding enough precedent. It’s difficult to believe the Orioles have just figured something amazing out. It’s not impossible, but they might well be the first.

Anyway, I think I’ve written enough. That’s 10 years of team performance, and 10 years of team projections. I’m not sure how much we’ve learned, but we’ve at least validated the feelings of Cubs fans. Sorry, or, you’re welcome.





Jeff made Lookout Landing a thing, but he does not still write there about the Mariners. He does write here, sometimes about the Mariners, but usually not.

52 Comments
Oldest
Newest Most Voted
Jaack
11 years ago

But I thought that fangraphs hated the Orioles irrationally. Have the comment sections been lying to me?

Pale Hose
11 years ago

I am not sure how hard this would be, but I am curious if year to year roster continuity (or lack of it) plays a role in projection accuracy.

Joe Butler
11 years ago
Reply to  Pale Hose

That’s a good idea. I wonder if they can more accurately project teams like Cardinals or the recent Giants, as opposed to teams like this year’s Padres.

Orsulakfan
11 years ago

Is there one kind of team that outperforms or underperforms projections? Are teams with WAR predominantly based in offense vs. defense vs. pitching more volatile? Or have you already written this article? Still trying to figure out that Orioles problem.

chris
11 years ago

Theories besides just randomness
– Young, unknown players exceed their projections causing the team to exceed its overall projection. As they become more accurately projectable the team expected win number climbs with their projections.
– Broken down older players still retain decent projections, until they prove for another year that their skills have diminished, resulting in the team underachieving against the projections. Future projections understand their drop off, and now the team is more accurately projected.
– Young prospects with high projections (Betts, Heyward, Uptons, etc) don’t live up to the hype.

The luck part is when you have a cluster of these events happen in the same year, such as the Rangers experienced last season.

ps – I told the captcha I’m not a robot, but are we not simply highly advanced biochemical robots? 🙂

Matt P
11 years ago

I’m confused. Suppose the Rockies won 25 more games than expected in 2010, 2012 and 2014 (75 total). Suppose they lost 25 more games than expected in 2009, 2011 and 2013 (75 total). According to your method of determining accuracy, wouldn’t that mean you were 100% right?

If so, why does that make sense?

Doug Lampert
11 years ago
Reply to  Matt P

If I understand correctly, the first chart shows all 30 teams over all 10 years individually, 300 data points.

Over and under by equal amounts would show as lots of noise there.

That chart establishes that there is some relation between projections and performance in a given year. He then asks the related questions “how much is this variation over more than one year” and “is variation from this expected performance a skill”, and comes up with “probably not”.

All the 5 and 10 year combined numbers are working on looking for possible trends over time.

I’d be happier if he gave r^2 values for both charts. But the first shows a clear (but noisy) trend, showing that projections do something meaningful, and the second chart of year to year variation shows that the noise probably isn’t the result of a consistent skill by anyone in the organization.

Matt P
11 years ago
Reply to  Doug Lampert

I agree with you for the first and fourth chart. That first chart does what you say it does and I agree about its utility.

The second and third chart are where I have a problem. It’s unreasonable to expect the 2010 Rockies to be similar to the 2014 Rockies. Even if they are all the players would be four years older. That means he’s measuring whether any organization has magical powers. I’m glad to see that he finds that they don’t but I question the sense of it. On the other hand, if any organization can figure out a way to build a model that’s consistent for four or five years then his test wouldn’t actually test for it (because other teams have changed their form).

That’s why I’d prefer to see standard deviations which are in his JABO article. I’d also like r^2 but whatever…

topiaryMember since 2016
11 years ago

i wonder how much deviation from projections is the result of in-season moves. an underperforming team might sell off their best players to reload for a future run, and likewise a team with a better record than they anticipated might go all-in and trade for a star or two. without any research it feels like the further a team deviates from expectation the more they may compound that trend with roaster moves.

Thanks, Comcast
11 years ago
Reply to  topiary

How is this handled? Are these fixed pre-season projections, or are they adjusted based on in-season transactions? I.e., if a team trades away a 4-win player midway through the season, are they adjusted down to an x-2 win projection?

WiIl
11 years ago
Reply to  Jeff Sullivan

Might explain the Cubs, who annually from 2012-14 moved major league caliber players at the deadline. Though they’re still -22 from 05-11, so maybe it is just that stupid goat.

Dusty Baker
11 years ago
Reply to  topiary

Has anyone studied the impact of cooking methods on the correlation between projections and performance? Steamer projections, roaster moves, Jason Grilli and the Friars of San Diego should figure prominently in this discussion.

Sorry. That setup was irresistible. Let’s go back to talking about R^2.

FormerlySufferingBucsFan
11 years ago

My quick guess would be teams that have bad records through 2/3 of the season are “out of it” and therefore play suboptimally by playing younger, worse players for the back third (esp. after roster expansion). On the other side, teams doing well add via trade AND get to play some of these teams with nothing left to play for.

I’d be interested to see a wins per month for each month vs each projection with the expectation of teams over/underperforming to a much larger degree near the end of the season. Obviously, this will be noisy as heck and may be inconclusive (or just a bad premise in general).

FormerlySufferingBucsFan
11 years ago

Woooo…I have a popular opinion!

Phillies113
11 years ago

I wonder how much injuries factor into how much a team underachieves or overachieves (think a Wally Pipp-type player being injured and being replaced by Lou Gehrig) in a given year.

Some Dude
11 years ago
Reply to  Phillies113

Yes, I wonder if there’s any correlation with DL days.

Catoblepas
11 years ago

Really curious what the team was that was projected to win more than 100 games, at the top right of the first graph.

Anon
11 years ago
Reply to  Catoblepas

Yes, and also the team projected at 100+ losses on the bottom left.

Anon
11 years ago
Reply to  Jeff Sullivan

The 2004 Cardinals did win 105 games, and the 2005 team was very similar players. Even at the beginning of the sample, it is difficult to say this projection had more noise than others.

Mike
11 years ago
Reply to  Anon

2004: 105 Wins, Swept in World Series
2005: 100 Wins, Lose NLCS
2006: 83 Wins, Win World Series
2008: 86 Wins, 4th in NL Central

Baseball!

Catoblepas
11 years ago
Reply to  Jeff Sullivan

Yeah, it was also very interesting to me that the projections nailed the most extreme team on the high end. Probably a coincidence, but a cool one.

everdiso
11 years ago

you guys know what I’m gonna say.

Eric R
11 years ago

What was the R2 on the first chart? I charted pythag against actual wins for the same group and got a 0.86 R2

You have a 21 win standard deviation; If +/- 3 SD *should* cover nearly all teams, then the projections have a range of +/- 63 wins, +/- 6.3 wins per year.

The SD I got using pythag was 14.7 wins, so doing the same comes up with +/- 4.4 wins per year.

That the gap between the two is less than two wins a year is pretty amazing given that the latter you are given a big piece of actual future information.

Blue
11 years ago
Reply to  Eric R

Yep, we need to know that.

Eyeballing it, I’m thinking in the .5 to .6 range.

everdiso
11 years ago

Jeff, I think an AWESOME use of this data would be an analysis of which TYPE of team may or may not get consistently over or underrated.

I.e. do teams with projected elite performers but little depth over or underachieve more or less than teams with little projected elite performance but great depth?

and maybe there are other ways to define “type” of team that would be interesting. your idea of looking at teams with extreme park effects is a great one too.

everdiso
11 years ago
Reply to  everdiso

and there’s a whole bunch of even better ideas in this vein in the comments above. nice.

everdiso
11 years ago
Reply to  everdiso

and do teams that overachieve do so based more on individual breakout performance or on general teamwide overachievement?

davels
11 years ago

I wonder if the number of replies to all articles has gone down since Fangraphs started using the captcha “I’m not a robot” thingie?

If it has gone down, is that a good thing?

Phillies113
11 years ago
Reply to  davels

It’s not a good thing if you’re a very honest robot.

Statistics don't lie
11 years ago

Captain Obvious says that something about Duquette and Showalter’s team building strategy is not measured well by preseason prognosticators. The major discrepancies began when these two came aboard, and I don’t hesitate to say, will continue this season. (They won 98 last year and estimates have them below 80 wins this year?)

1) The plan is to catch lightning in a bottle in an educated way, by increasing your odds. Look at 2B and OF this spring. Cheap breakout candidates are there, probably 6-8 of them. Preseason experts rate them low, because individually their odds of breaking out are poor. But if you have multiple breakout candidates for a position, you don’t need to know which one breaks out in April. Play the hot hand until the best one becomes apparent. (Roll 8 dice at the same time and the odds of getting a 6 are better than if you are rolling only 1-2 dice.)
2) Plan A, Plan B, Plan C, and Plan D are of similar quality. There isn’t too much fall off if someone gets injured or underperforms. Plug in the next guy and the team moves on in the same trajectory. Deep depth.
3) Consistent starting pitching, EXCELLENT DEFENSE, EXCELLENT BULLPEN DEPTH thru AAA. Ability to have interchangeable AAAA relievers available for long relief.

Just a few ideas.

anon
11 years ago

2nd paragraph describes a lot of teams in this or recent years

everdiso
11 years ago

red sox not surprisingly among the worst underachievers, not a surprise when you consider their 2 world series victories in that time frame

los
11 years ago
Reply to  everdiso

The Red Sox had two disaster years and one year that they were much better than projected. I don’t really see this as them being consistently over projected. Projections were mostly in line for them for 7 out of 10 years.

K
11 years ago

This thread is going to suck.

hookstrapped
11 years ago

There are lots of interesting ideas in the comments as to what might be some common chacteristics of overachieving and underachieving teams. it would be a very cool analysis to see if those ideas do help explain divergences from projections. Makes sense to look at teams that closely matched their projections, too.

Can’t wait!

Blue
11 years ago

Interesting, but single IV analysis doesn’t get that far. You have enough degrees of freedom to put in some control variables (e.g., league, year).

Sir
11 years ago

One part of the Cubs’ poor performance relative to projection is likely in season lineup turnover. Being a bad team that is selling off their good players (eg, Garza, Dempster, Samardzija, Hammel, et al.) every year at the trade deadline is going to lead to losing some wins. Not surprising at all too see the Cubs at the top of the list here, as they’ve been one of the most active trade deadline sellers for the past several years.

RMD
11 years ago
Reply to  Sir

Nah, it’s day games. It wears the players out that causes their underachievement.

Hee-Seop Choi
11 years ago

Is there any correlation between having a good/great bullpen and outperforming projections? We know that bullpen performance can vary greatly from year to year, and that teams with good bullpens can outperform projections because of how they can use their bullpens to perform better in higher-leverage situations (such as the 2014 Royals).

Would regression in a team’s bullpen cause them to go from outperforming projections one year to meeting/underperforming the next year?

Dave
11 years ago
Reply to  Hee-Seop Choi

Maybe. Yet, 2009-2014 Tigers.

Mike Green
11 years ago

Thanks, Jeff.

What always needs to be borne in mind about pre-season projections is how wide the error bar is. You’d think that if a team is projected to win 80 games that it would fall within the 70-90 range, 90 or 95% of the time. That isn’t so. There are substantial clusters in both the 60-70 and 90-100 range. It should be noted that, in recent years, with the increased amount of intra-divisional play, you would expect there to be clusters in a particular division in a particular year. Let’s say two teams in a division are projected to win about 80 games. One is struck with a rash of injuries early on and the other is uncommonly healthy. The impact of luck is magnified by the number of games the teams play each other.

studes
11 years ago

Here’s a question: are projections better than just using the previous year’s record? If you combined the two, could you improve your projections? How about factoring previous year’s record with other variables, such as pythagorean record?

Has someone already done this? If not, I could take a crack, Jeff, if you don’t have the time.

studes
11 years ago
Reply to  studes

Sorry. I missed that you had included your R2 (0.36) in the comments. When I correlate previous year’s wins vs. current year wins for the same group of teams. I get an R2 of .25. So that’s the simple gain given by projection systems.

Blue
11 years ago
Reply to  studes

Not exactly earth-shaking.

GBORG
11 years ago

I think it would be interesting to see the same chart, but with Pythagorean wins and losses. To a large extent, what projections attempt to do is indicate the offensive and defensive production — and that can be measured more precisely than wins and losses.

That might explain some of the extreme results. The Orioles, for instance, outperformed their Pythagorean record by a total of 25 games over the last five years.

Matt
11 years ago

I have a feeling that if you reduced the records to only pre All Star game (presumably before the sellers start selling and the buyers start buying) then everyone will be much closer to projection and only ‘home ballpark’ and ‘managerial decisions’ along with ‘injuries’ and a few other factors will be at play here.

Seeing the Cubs at the bottom, when they have consistently been sellers, makes it seem like the buyer/seller issue makes up most of these differences.

Dave StudemanMember since 2017
11 years ago

Jeff, can I ask a dumb question? I can’t find the answer anywhere. Are Fangraphs’ team projections based on a simulator? Or are they a sum of each player’s projected WAR and playing time? Or something else?

section 34
11 years ago

Wow, I thought I could read this without another slap at the 2012 Orioles. Jesus, Jeff, can’t you just admit you were wrong and let it go?