A Playoff Odds Check Supplement
Yesterday, I tested how well our playoff odds have predicted eventual playoff teams. Today, I’m going to slice the data a few more ways to get a more robust look at what our odds do well, and where they have fewer advantages over other models. It will be number- and picture-heavy, word-light. Without further ado, let’s get started.
A discussion with Tom Tango got me wondering about why our Depth Charts-based odds do so well early in the season relative to other systems. Their advantage is particularly strong at the beginning of the season and fades as the year goes on. For all charts in the article that are based on days into a season, I’ve excluded the 2020 season for obvious reasons. Here are the mean average errors for each of the three systems over the first 60 days of the season:
What’s driving that early outperformance? In essence, it comes down to one thing: the projection-based model is willing to give teams high or low probabilities of making the postseason right away. Our season-to-date stats mode is hesitant to do that, and the coin flip mode obviously can’t do it. Take a look at the percentage of teams that each system moves to the extremes of the distribution — either less than 5% or more than 95% to make the playoffs — by day of season:
Why does this matter? If you’re judging based on mean absolute error, making extreme predictions that turn out to be right is a huge tailwind. If you predict something as 50% likely, you’ll have an error of 0.5 no matter what. The further you predict from the center of the distribution, the more chance you have to reduce your error.
Of course, that only works if you get it right. If you simply randomly predicted either 5% or 95% chances without any information about the teams involved, you’d do just as poorly as predicting 50% for everything. Making extreme picks when you have information that suggests they’re likely to be right is the name of the game. Read the rest of this entry »