We Were Bad at Predicting (Some) Things
Before Opening Day, 40 staff writers and contributors from across our family of blogs made predictions about which teams would make the 2018 playoffs. I compiled the results, which can be found here. Some predictions were bold; others were fantastical. Many exhibited a perhaps unsurprising deference to our preseason projections. Affecting an air of clairvoyance is a rite of spring; realizing we’re a bunch of goofs is fall business. So before the playoffs begin this evening, let’s spend a moment reckoning with the fact that we’re bad at predicting things.
First, though, we should acknowledge the winners. I will not name the loser here; the world has enough cruelty in it. No one got every playoff team right — nor did anyone’s ballot perfectly mirror the order in which the field was eventually seeded — but David Laurila and Neil Weinberg each correctly predicted eight of the 10 playoff teams. David had the Twins in Oakland’s place, while Neil thought Boston would be traveling to Los Angeles for the American League Wild Card, but the rest of their AL ballots were correct. They also predicted four of the five National League teams. They both thought the Nationals would win the NL East, and they weren’t alone: 39 of 40 ballots agreed, with the final vote going to the Mets. We messed that one up, though not as badly as the Nationals did. Our full results are below.
| Correctly Predicted Playoff Teams | Number of Writers |
|---|---|
| 8 | 2 |
| 7 | 11 |
| 6 | 19 |
| 5 | 7 |
| 4 | 1 |
Jeff Sullivan was the only person to correctly predict all five AL playoff teams. He had New York and Boston swapped for the AL East crown, but rightly thought Oakland would be the second Wild Card Team. Paul Sporer and Ryan Pollack also believed in the A’s, though the remainder of their ballots did less well. The AL results are below.
| Team | Number of Votes |
|---|---|
| Boston | 33 |
| Cleveland | 37 |
| Houston | 37 |
| New York | 38 |
| Oakland | 3 |
| Correctly Predicted Teams | Number of Writers |
|---|---|
| 5 | 1 |
| 4 | 28 |
| 3 | 9 |
| 2 | 2 |
The National League stymied us much worse. The Braves not only didn’t appear as a division champion on anyone’s ballot, they weren’t included on anyone’s ballot anywhere. Only four of us believed in the Rockies: Laurila, Sporer, Weinberg, and erstwhile contributor Travis Sawchik.
| Team | Number of Ballots |
|---|---|
| Atlanta | 0 |
| Milwaukee | 19 |
| Los Angeles | 39 |
| Chicago | 36 |
| Colorado | 4 |
| Correctly Predicted Teams | Number of Writers |
|---|---|
| 4 | 2 |
| 3 | 14 |
| 2 | 24 |
One could interpret these results as a reminder of our limitations as human beings or perhaps as a testament to how our affection for the things and teams we like best can cloud our judgement. But I prefer to read it a bit differently. I’d hate baseball that was entirely predictable. My pride might wish I had seen Oakland coming and could say so, but my heart? My heart delights in the surprise. Happy playoffs, everyone.
Meg is the editor-in-chief of FanGraphs and the co-host of Effectively Wild. Prior to joining FanGraphs, her work appeared at Baseball Prospectus, Lookout Landing, and Just A Bit Outside. You can follow her on Bluesky @megrowler.fangraphs.com.
So in isolation the biggest surprises would be Braves up and Nationals down, but they kind of share. I feel like if we plot wins versus some kind of playoff pick points, Oakland might win.
Early congrats to the one person who picked Yelich for MVP.
That was Josh Shepardson! Nice call!
A good reminder that predictive stats are exactly that, predictive. They do not pre-ordain any set of outcomes or define who a player is and forever will be in the future. It’s just the most likely outcome. Unlikely things happen all the time!
Just to nitpick, the most likely war up to a decimal point for most players in a season would be 0.0.
Predictions we see show the median, not the mode.
If we’re gonna nitpick, don’t they show the mean, not the median.
It seems to me like it should have been easier to get the AL correct. If you just wrote down last year’s AL playoff teams you would have 4 out of 5. And three of those are slam dunks (NY, Bos, Hou).
I’d say Cleveland was a slam dunk too, especially with how weak the rest of its division is and was.
What surprised me most was that there were no unanimous picks in the preseason despite 7 supposed superteams. While obviously there are no such things as sure things in baseball, I would figure the odds of any of the superteams missing the playoffs would be way lower than 50% , and it did play out that way since of the 7 pre-season prohibitive favorites, 6 made the playoffs and 4 weren’t even close to missing it.
time and time again the A’s exceed their projection. It’s likely because they roll out young teams with hardly any data consistently. Anyway, Go A’s, now and the future when they actually have a starting rotation
The A’s have a tradition of doing more with less.
Any chance we’ll get a similar postmortem for the Playoff Odds page? Two teams in a single year with <3% odds to win the division and damn near a 4th. Not to mention the Athletics with ~10% postseason odds. I understand there's a tremendous amount of uncertainty built into baseball, but I don't think these numbers reflect that.
Actually, I’d rather have a post-mortem on the several ‘this season is boringly predictable and super teams are ruining baseball’ posts we got in the spring. There was a lot of doom and gloom going into this season about how many teams were ‘locked in’ for playoff spots, but in the end we had a pretty fun September.
Yes. That misperception went down in flames.
not in the AL.
Even the AL was much more interesting than advertised. You had pre season picks fall flat (Twins, Angels), collapses (Mariners), surprise contenders or quality teams (A’s, Rays). The As actually managed to turn the AL West into a race that went into September. I’ve got no complaints – if the issue is that for the most part really good teams turned out to be really good then I suspect that happens pretty frequently…
I would figure out the ROC curve for past and current predictions and plot the distributions to start. That is, figuring out true positives, false positives, true negatives, and false negatives based on prior probabilities. From there you could see if the model is not just over or underplaced but if it over and under precise (which is my concern)
It was Sean Dolinar for anyone else as curious (and awful) as I am.
This actually seems like it ought to be titled “We Were Pretty Good at Predicting (Most) Things.” Apart from the Braves there’s really nothing here that seems like a big whiff, and I’m pretty sure nobody in the world (including the quantitative models as well as the humans) predicted the Braves.
I feel like there’s often a “trendy” pick, usually a rebuilding team with a good farm and an interesting front office. often this team disappoints, but this year it was the Brewers, which makes the predictions look extra good.
In retrospect, no one can really be all that surprised at the Nats underperforming again, but I think even someone who predicted that would have thought it would finally be the year the Mets kept more than three players healthy.
“Mets” and “healthy” in the same sentence?
I would have taken the under on number of MLB players contracting hand, foot and mouth disease – and lost.
The A’s would like a word with you.
The whiff on the Braves at 3% speaks to structural issues of the model. Given the number of free parameters. it’s the kind of prediction that tells you you are using the wrong kind of underlying distribution to calculate the probabilities. (my suspicion is it’s at least an extreme value distribution of one sort or another)
Given this is the second year in a row this has happened, and the third year in four…. might be a time to take a deep look under the hood.
I wonder if it was difficult to predict the immense defensive contributions of Swanson, Albies, and Camargo (along with continued excellence from Inciarte)?
I think it’s as simple as the model(s) all, quite predictably, whiffing when a bunch of very young rookie/near-rookie players simultaneously have star-caliber seasons. I bet Fangraphs wouldn’t have picked the ’86 Mets either.
My MVP picks were so bad I went back to down vote them
Shall I assume that your unwillingness to name the loser indicates that it was not Carson Cistulli?
Good point: Carson would relish being called out as the worst at predicting stuff, as well as the rest of the FG staff.
I want to know who didn’t pick the Astros or Indians to win their divisions? I could ALMOST see picking against Houston if you’re expecting a WS hangover, but Cleveland? In that poo-poo platter of a division, half the teams in the other divisions would have won it if they switched with Cleveland.
Pu-pu platter. Please.
I wouldn’t be so quick to pooh-pooh that spelling.
Agreed. It almost makes me wonder if at least a couple of people threw in one or two contrarian picks for the sake of being contrarian.
You guys are really bad at predicting the Orioles record. I think 5 years straight, they performed waaaay over the projections, and this year they performed waaaay under them. I think in 2017 you guys pinned the tail on the donkey, though.
Meh – anyone who comes out here and grouses about projections/predictions being incorrect is kind of like the dummy on 538 going on and on about how they ‘got it wrong’ by not predicting a Trump win.
A good reminder that this site employs 40 FREAKING PEOPLE who provide excellent baseball content. I had no idea the footprint was that big. Keep up the great work, FG.
The A’s and their $66 million payroll just eternally refuted three very stupid but commonly held misperceptions:
1. “Only rich teams make the playoffs!” NOT
2. “Only super teams make the playoffs!” NOT
3. “A team can’t win without a big payroll!” NOPE
Hopefully we will not see this nonsense repeated next year.
Almost nobody here holds any on those misconceptions. At the same time pretending it isnt a factor is equally as silly.
BTW, the super teams went 7 for 8 making the playoffs, our ability to recognize good teams in the winter isnt really broken.
Exactly. Being a top 5 payroll isn’t necessary for regular season success, but it is sufficient.
Tell that to the Giants
No, it’s not, and this season was fairly consistent in that respect. Also tell this year’s Nationals, the 2015-17 Tigers, 2012-14 Phillies, 2013-14 Yankees, etc.
Being able to run a high payroll obviously helps to overcome free agent signings that don’t work out for teams or to supplement a roster around cost-controlled players, but it’s really tough to win a lot – even with a high payroll – without major contributions from minimum salary and early arb year players.
“Almost nobody here holds any on those misconceptions.”
LOL, where were you last pre-season? There was a chorus about how only rich superteams with huge payrolls could make it to the playoffs. People were claiming there wasn’t even going to be a competition in 2018; division races were going to be runaways. And in only one division of the six did that come to pass.
At the end of the season, with the A’s and Braves making the playoffs and with the incredible excitement of two divisions being settled by tiebreaker games, people are now looking at their shoes and saying that nobody here really believed that. Sorry, but people did, and some of them were really loud about it. And the A’s, with their $66 million payroll, proved them dead wrong: there’s still excitement in MLB, there’s still uncertainty, and there’s still room for Cinderella—and while poor Cinderella may not get to marry the Prince she has proven that she can fight her way to the ball and at least get to dance there.
So next pre-season, when one of you out there is thinking about posting about how unfair MLB is and how only rich superteams with huge payrolls have a chance of making the playoffs, remember the A’s and think again. And think differently that next time.
Shocker: it looks like we are going to see the exact same misconceptions repeated next year. Some people just won’t let go of their pet wrong ideas, even when reality has proven them to be wrong.
RE: #2 If there are 10 playoff spots and 7-8 “super-teams”, by default, 2-3 playoffs spots are automatically up for grabs between the rest of MLB. Braves getting in based on the failure that was the Nationals season is the only surprise. Brewers, Rockies and A’s took the three spots that HAD to be available.
re: 1 and 3. Large payroll obviously has an impact in keeping the “super-team” window open a lot longer than small market teams. The Dodgers can keep/pay Kershaw and other, the Royals cannot. So Dodgers now have 6 straight NL West titles, and three years after winning the WS, the Royals lose 104 games.