Evaluating Our Free Agent Contract Predictions

As I write this, I’m watching a spring training game on my other monitor, which is a good reminder that another season of baseball will soon begin. Forty-eight of the Top 50 free agents of the winter have signed, with Zack Littell and Lucas Giolito the lone holdouts. That means it’s time for my annual review of contract predictions, mostly mine and the crowd’s.

I like to evaluate my own predictions so that I can get better at making them in the future. I like to evaluate your crowdsourced predictions because it’s fun, and because everyone likes hearing how smart they are. Our crowdsourced predictions have been consistently excellent, arguably better than any industry expert, and that makes displaying them particularly enjoyable.

To evaluate our accuracy, I broke the signings down into three categories: hitters, starting pitchers, and relievers. I also examined the entire Top 50, without positional separation. I used a formula that I discussed earlier this winter as my chief metric of accuracy, but I also checked how close we came on average annual value, total guarantee, and number of years. I looked at how the predictions matched the overall amount of money spent in the market, and also considered how close each individual prediction came. That way, I was able to evaluate two things: Who did the best job predicting the broad market, and who predicted what each free agent would get with the greatest accuracy.

I borrowed the format for my review from last year’s edition. I gathered all of my predictions, as well as those of the crowd and several of my colleagues from across the industry. I compared each actual contract in the Top 50 to what was predicted for that player. If one of my colleagues didn’t predict a contract value for a particular player, I simply excluded that player from my analysis of that writer’s set of predictions. Below, you’ll find some broad analysis of how we all did this year. In the appendix, you’ll also see a complete record of all the categories I measured, as well as some notable performances from non-FanGraphs prognosticators. A quick note before we begin: For the entirety of this article, a positive number means an overestimate, while a negative number means an underestimate.

Let’s start with the predictions that came in too hot.

Too Hot

I did a poor job of predicting the market for NPB stars this winter. There are two possibilities here. The first is that there’s something specific about Munetaka Murakami and Tatsuya Imai that my style of projections just misses. I don’t think it’s that, though, because every other prognosticator was stymied too. The average prediction on Imai across the group I surveyed was a whopping $75 million higher than the deal he signed; the average miss on Murakami was $85 million higher. I think that there might be something structural going on here; outside guys at the very top of the market, the posting fee system encourages smaller, shorter contracts. Those allow the player to hit free agency sooner and also don’t pay as much in fees back to the posting team. Aside from that, we’ve got the normal mix of relatively small misses on stars, guys on pillow contracts, and my irrational love of Josh Naylor. Oh, and just for posterity’s sake, the Miss column uses the formula I laid out here to account for contract length mismatches.

The crowd didn’t escape Murakami madness, but great work on the rest of it. Your misses were basically the pillow contract guys – two players who accepted qualifying offers and two former Diamondbacks stars who signed late after their markets didn’t develop as expected. This is a really good showing, and as you’ll see, it’s part of a pattern: The crowd had lower estimates pretty much across the board compared to the “pros.”

For the record, I was too hot on the overall market, and by around the same amount that the crowd was too cold. We both did a pretty good job in the aggregate, though. There was one systematic error – my overestimation of NPB contracts – but the rest of our misses high were scattered and don’t suggest any biases to me.

Too Cold

Boy, is there egg on my face for predicting that King would accept a qualifying offer; he got three times that amount. Other than that, though, I feel pretty good about these. They boil down to slightly higher demand – a few million dollars here, an extra year there – for good hitters. There’s no equivalent to the NPB problem here; the way that free agency works implies some misses like this, and I don’t see much bias in the types of players I missed low on.

You have to love seeing Kyle Tucker on my too hot list and the crowd’s too cold list. His market was indeed polarizing, and the Dodgers gave him a deal that makes me happy I have a method for comparing contracts of differing lengths. Other than that, your misses are pretty much like mine – sometimes estimates are a little low. There’s one structural bias, which I’ll get into below, but it doesn’t involve a particular type of player. The misses are spread across hitters, starters, and relievers. Again, great work. For the most part, this was a pretty straightforward year for free agency, and everyone handled that pretty well.

Just Right

I removed one-year contracts in this section, which I largely posted to pat myself on the back. I had a pretty good handle on the top of the pitching market this winter, and some reliever contracts as well. As I mentioned, a lot of the players this offseason had a clear market, and the top group of pitchers in particular had a lot of suitors and clear differentiation from the next tier down, which helped me estimate their contracts pretty well.

Meanwhile, maybe I should just ask the crowd what my NPB contract estimates should be going forward. This was incredible work! As I mentioned up above, the group of predictions from professional baseball analysts was far too high, but you all got Imai’s deal to within $2 million. Give yourself a pat on the back for this great work. As an added note, the fact that you have two starters, two hitters, and a reliever suggests that there are no holes in your estimating skills.

Overall Market and Final Thoughts
The market for pitchers ran hot early and trailed off. The market for hitters, in contrast, never really got going, with Murakami a key part of that. I tracked five professional prognosticators and the crowd, and your predictions were the only ones that didn’t overshoot. In fact, you and Kiley McDaniel of ESPN were the only groups not to overshoot overall; everyone aside from you, me, and Dupree Kiley was over by $5 million per player or more. In other words, the three prognosticators here correctly judged that the market wouldn’t continue to increase at the rate that it had in recent years.

I moved the future-year-multiplier factor, which I use to equate contracts of unlike years, and found that this was robust for multiples between 0.5 and 0.75 (I used two-thirds for the study, as per my earlier article, a regression based on old contracts suggests that 0.63 might be a slightly better estimate). For the entirety of that range, the crowdsourced predictions, my predictions, and Kiley’s predictions were the best in some order.

For what it’s worth, the drivers of error among the three top sets of predictions were quite different. Remove the three NPB players from the dataset, and my predictions were almost dead on – around a third of a million low per player. Removing those three also helped Kiley’s predictions out meaningfully. It feels like he and I might have fallen victim to some groupthink here, which is something I’m going to keep an eye on so that I can attempt to avoid it in future years.

On the other hand, the crowd predictions were low on each of the top nine players. It might be time to update your internal model of the cost of a win in free agency – perhaps in keeping with some recent research on the topic. Remove just the top five players, and these predictions were almost perfect. On the other hand, the top five players are often the most interesting part of the market. Great work in total, but we all have areas for improvement.

As I seem to conclude every year, you readers did incredibly well at making contract predictions. Sure, the top of the market came in hotter than you expected, but this was an excellent set of predictions overall. If I were a team-side analyst looking for a good baseline estimate of contracts, I’d call up the FanGraphs crowd first thing.

I’m quite happy with my estimates, as well. Setting aside my much-discussed misses on the players posted from NPB, I nailed the top of the market. My average absolute error on the top 10 players was half of the next-closest group, which was, believe it or not, the crowd. That’s because almost every other analyst missed very high on Tucker and very low on Alex Bregman and Cody Bellinger. If I were suggesting a hybrid approach to a team, I’d tell them to take my estimates at the top of the market, take the crowd’s for everyone else, and under no circumstances ask me what a player from a foreign professional league might sign for.

That’s not to say that no one else did a good job. Kiley came the closest to predicting the total amount of money committed after adjusting for contract length. The MLB Trade Rumors team had the best set of relief pitcher predictions that I’ve measured in four years of doing this exercise. Not to be overly congratulatory, but I think everyone should take a bow. The nerds and their spreadsheets did a pretty good job of guessing contracts this year.

Appendix
Here are a few tables showing error metrics for a variety of ways of scoring predictions. In each of these tables, I’ve included my predictions as well as the crowd’s, plus the best prediction from other industry prognosticators. If two industry prediction sets performed equally well on a given metric, I included both. First, the year-adjustment method I detailed earlier this winter, in both average and absolute average forms:

Next, AAV by the same average/absolute average method:

And finally total guarantee:

Here’s a non-monetary one, too. I noted how many years I predicted for each contract as well as how many years each contract actually included and displayed them as a grouped bar chart.

I did the same for the crowd:

My read of this data is that even with the slight biases in both my predictions (the NPB contracts) and your predictions (the top of the market), we outdid the rest of the prognosticators this year, and that it’s probably too close to call between our two sets of predictions. To be honest, that’s been my read in a lot of years. If I had to pick between me and the rest of the professional prognosticators, I’ll admit I’d take myself. I’m not certain that I’m better, but I’ve achieved better results of late. On the other hand, if I had to pick between using my predictions or the crowdsourced predictions, I’d call it a true coin flip. Perhaps I’m biased – you should always be skeptical when someone grades their own predictions and gives themself a good mark, and as I’ve said, my colleagues did good work – but for me, the results speak for themselves. It’s a golden era of contract predictions, and there’s no set of predictions I’d prefer over our crowdsourced ones. That’s pretty dang cool to me.