Here’s How I’m Planning on Evaluating Free Agency Predictions

Every year, FanGraphs (in this case, I am FanGraph) releases contract predictions for our top 50 free agents. We also run a contract crowdsourcing project for those players, and I have to say, the crowd is spectacularly good at this. Last year, for example, I looked through all of the various predictions across the internet and awarded the crowd the title of best overall prognosticator.
But honestly, the winner of that award was hard to determine because I didn’t have a great way to evaluate the various predictions. Why so difficult? Because not every deal ended up being for the length we all predicted. As an example, I predicted 12 years and $48 million per year for Juan Soto, while the crowd predicted 13 years and $45 million per year. Soto signed a deal that was for 15 years and $51 million per year. Who was closest to the mark? It’s not immediately clear. I did better on the AAV, but the crowd did better on the number of years. There’s no obvious determining factor to use when comparing the two. Even worse, the two are inversely correlated; more years generally means a lower AAV. The two predictions seem pretty similar to me, but I had to grade AAVs and total guarantees separately, and that just felt clunky and confusing.
After some time bouncing ideas off my friends and colleagues, and plenty of time in the FanGraphs Idea Generation Lab (not real, but man, it should be), I think I have a solution. It’s simple, really. Evaluating contract predictions would be much easier if the predictions and the actual contract were for the same length, so I made them all the same length. Read the rest of this entry »










