Comparing and Contrasting This Year’s Prospect Rankings

A few weeks ago, lead prospect analyst Eric Longenhagen published FanGraphs’ top-100 prospect list. Baseball America recently performed a similar exercise, as did Baseball Prospectus, Keith Law, MLB.com and John Sickels.

On the whole, there’s a lot of consensus among these rankings. Although the order varied, all eight of Andrew Benintendi, Dansby Swanson, Amed Rosario, Alex Reyes, Gleyber Torres, Eloy Jimenez, Yoan Moncada,  Brendan Rodgers and Austin Meadows ranked within each outlet’s top 20. Fifty-nine players made every single top 100. The point of this article, however, isn’t to celebrate those similarities, but to point out the differences. In what follows, I identify the prospects that each outlet ranks higher and lower than the “establishment,” and look at how the various outlets compare to each other. Brace yourselves for an onslaught of tables and plots.

*****

List of Lists

Firstly, here’s the consensus prospect list. I hard-coded all unranked players as having ranked 210th, since Sickels ranked just over 200 names. The “Avg. Rank” column below is a simple average of all the rankings.

That calculation is a bit flawed, however, as it assumes the correlation between rank and value to have a linear relationship. In reality, it’s far from linear. For example, every evaluator would agree that the gap in talent between the No. 1 and No. 20 prospect is much larger than the gap between the No. 20 and No. 40 prospect. Jeff Zimmerman quantified this phenomenon in last year’s Hardball Times Annual and came away with a formula for estimating a prospect’s value using his overall rank. By applying Zimmerman’s formula to this year’s lists, I created a consensus-top-prospect list sorted by expected WAR value.

*****

Players Liked and Disliked by Each Outlet

Next, let’s look at the prospects each outlet likes and dislikes relative to the industry consensus. I omitted players who were ranked within one spot of the average consensus rank. Although the difference between the No. 2 and No. 3 spot represents a large value gap, it doesn’t necessarily represent a large difference in evaluation if the evaluator regards the two as near equals. I threw my KATOH forecasts (the “stats-only” version) into the mix as well, just because.

*****

*****

*****
Note: Keith Law’s rankings are published behind a paywall and therefore not reproduced here; however, they can be found at ESPN Insider.

*****

*****

*****

*****

Comparison of Lists

Now that I’ve run through the specific players on whom the various outlets disagree, let’s take a step back and see just how much they differ on a more macro level. Below, you’ll find a correlation matrix that compares the prospect values implied by each outlet’s ranking.

Keep in mind that none of these correlations are necessarily good or bad. Going against the establishment is great if you’re high or low on the right players, but it could also be a bad thing if you stray from the mainstream on the wrong ones. The same goes for siding with the comparisons to KATOH. A player’s tools and performance are both critical pieces of information. Betting on the tools can absolutely be the best strategy in some cases, but at the same time, plenty of grinders have parlayed their minor-league successes into solid big-league careers. I don’t believe the above chart tells us anything about the quality of the rankings, but it’s interesting to look at nonetheless.

On the whole, there was relatively little dissonance between most of the rankings; however, the bright red squares indicate the areas of relative disagreement. The brightest red square is at the intersection of Baseball Prospectus and MLB. The biggest driver here is Alex Reyes, whom BP ranked first, compared to MLB’s 14th. Other sizable discrepancies include Andrew Benintendi (3rd vs 1st), Yoan Moncada (5th vs 2nd) and Cody Bellinger (26th vs 12th). Most of the other lists disagree similarly with Baseball Prospectus.

Unsurprisingly, KATOH is easily the most contrarian here. Its correlations with other lists fall between .40 and .51, while no other pairing dips below .77. A bit more surprising is that Baseball America’s rankings correlate best with KATOH’s, while Baseball Prospectus’ correlate most weakly. Like KATOH, Baseball America is relatively low on Lucas Giolito and Robert Gsellman, but loves Cody Bellinger and Vlad Guerrero Jr. Baseball Prospectus, on the other hand, is significantly higher than KATOH on Gsellman and Alex Reyes and lower on Bellinger and Ozzie Albies.

Although it may be a coincidence, I also find it interesting that the consensus column seems to correlate with the KATOH column. (Correlation of correlations? Whoa, meta.) In other words, the lists that agree with the consensus also agree with KATOH’s projections. I don’t know what to conclude from this other than it’s interesting.

I don’t believe any of these findings should be used to evaluate the prospect rankers, either now or in the future. The sample size is far too small, especially considering the inherent unpredictability of prospects. Every evaluator has hits and misses, so cherry-picking a few would prove almost nothing. Nonetheless, it’s interesting to objectively identify the players whom each ranker likes and dislikes relative to their peers. And it will be fun to look back in a few years to see which of the “going out on a limb” rankings correctly predicted one of tomorrow’s stars.