Balancing Information and Bias in Prospect Reporting

I’ve done plenty of rankings over the years for various outlets, but this new venture at FanGraphs is the first time I’ve gone through all 30 organizations to rank each team’s best talents. I’ve done some team prospect rankings in the past and the thing I experienced then — and heard about from people all over the industry — are the biases you have to wade through when club officials and scouts talk about their own players.

This is an obvious thing that even the most casual reader of prospect analysis would understand. The team wants to have the profile of their prospects boosted, both for their job security and for trade value. Nearly every source I talk to for these lists will reference a player from an earlier list in the series and ask a question about them, ask where I think their system stacks up so far in my research or ask my feelings about teams I haven’t written up yet. Most teams take the rankings from trusted publications and add them to their internal metrics, and they’re always checked for discrepancies or useful nuggets of information before trades are consummated. This isn’t just some fanboy nerd thing on the Internet. Well, it is that, but it isn’t only that.

There were instances in past years where a team executive would push a guy on me, but when I’d call a couple scouts from other organizations and I’d find “the truth”; normally between an overzealous exec and a pessimistic scout, landing around where the average scout would be. There was one instance this year when one scout, who was credited with signing the player in question, suggested I move a player (not at the top of a list) up one spot. Multiple rival execs said I had the player in the right spot, and I learned there were rumors the team was trying to trade the player in question.

That said, it was one spot and in the middle of the list; not exactly the hard sell. In every other instance of a surprising piece of info from a club source (more than 14 organizations so far), it’s been confirmed by multiple outside sources, to the point that if I wrote the lists purely from info from the team itself — which I’d never do — I’d have almost the same order as I would after talking to multiple rival scouts.

Why is this the case? I have a couple theories, but I think the larger phenomenon here is the commoditization of prospect rankings. Billy Beane essentially said as much a few months back: “I think you’re undervaluing the value of the present, particularly in sports.” Scouting reports on big league players essentially don’t exist on the Internet, thus there aren’t rankings and there isn’t hype created about which team has the highest-ranked major-league this or that. Instead, we focus on the teenager who hasn’t even played in single-A yet. We’re so focused on trying to peg the future that we’re overlooking the present, at least in the scouting wing of baseball media.

Multiple publications rank these players via talking to scouts and execs, along with numerous other outlets parroting these rankings when news warrants it. We’re all talking about mostly the same players in mostly the same areas on these lists.  There’s plenty of room to differentiate, innovate and become the industry standard, but the casual reader won’t necessarily pick up on these cues.

If a team coordinated a couple sources to spread propaganda, there’s a good shot I’d figure it out before I put up the list. If I didn’t, at least one of the other publications would. At some point, those execs would lose credibility — and while that isn’t the end of the world for them — they like having an open line of communication and don’t want to make enemies. In fact, ownership groups read these lists, and the public perception of a team’s farm system can have an impact on the careers of those involved in player development.

In one instance, I was talking to a general manager about one of his players who’d been known to have character issues. I wanted to hear how he would present this player to me. I mentioned the name, gave a sentence or two about the player and then said, “But some scouts aren’t really on board,” as a vague way of referencing the makeup concerns. The GM stepped in and volunteered to list some of the specific character issues in the past and said the organization hadn’t had any problems with the player this year and they think it’s a maturity issue that he’s mostly past. Still, the general manager conceded his player was still probably not quite up to a major league level in the mental department.

There was another instance with a player who had a lengthy suspension for off-the-field issues and I only referenced his on-field strengths when I brought him up in conversation. In response, every source I talked to with that club proceeded to give me additional information I didn’t have about why the player was suspended, what he needs to work on and that they hope he can get his situation straightened out. In both cases, I moved the player down on my list.

I also commonly hear references, even by scouting directors, about how that pick or that whole draft (which he presided over) was a dud, or execs talking about how their system is below average, etc. There’s an understanding that these comments are off-the-record, but it’s still more honesty than you’d expect to hear in many cases. There’s some rhetorical value to being honest. I could trust your evaluations more on other players since you were upfront when you didn’t have to be, and an executive could use that for some other self-serving purpose in the future. That said, I know which people are tied to which players and who stands to gain something from a player’s rating being bumped, so even that may not yield an advantage.

I used to assume, until proven otherwise, that every exec read The Art of War and treated these conversations as a battle. Very few seem to have taken that approach with me, though. It could be due to something specific to me. I have a past working for clubs, and many of these guys regularly see me at games or have known me for years prior to my work here at FanGraphs. Or maybe they want to be on my next future GM list, and are simply buttering me up for the cause.

In addition, I’ve shifted a bit from mostly amateur (draft and July 2nd) rankings to more minor league-focused rankings. The amateur markets are inherently secretive since misdirection or generic reports may make clubs more likely to get the player they want. With development of minor league prospects, being deceptive is just delaying the inevitable. An exec could tell me a player who hasn’t performed in years looked great in instructs when there weren’t many scouts, but if he’s struggling the same way halfway through the next season, I can essentially disregard that information.

Some combination of all these things likely factor into the changing way that I talk to sources, but I think there is a bigger truth driving it, something Beane hinted at. Prospect rankings have been commoditized and may even be overrated in regards to their real-world impact on the big league standings, so execs don’t see talking to people like me as a way to get an advantage, but rather a way to feed the beast of people online caring about what they do.

Indirectly, that could help send more people to the ballpark, which would give them the power to get more/better players, which would eventually be manifested on a prospect list and in their job security and pay. Of course they aren’t thinking all of these things consciously, but a conversation that used to be standoffish with people who used to be unavailable are now returning my calls and being more honest. Maybe I shouldn’t be jinxing it.