The Status of the Scouts vs. Stats Debate

“Scouts vs. stats” is an expression that boils a complex, gray issue into clear black-and-white sides,in a way that’s familiar to those who follow political media. In the reality of front-office decision-making, however, this “debate” has been settled for years and the obvious answer was always “both.”

In fact, the issue has moved past simply using both. Until recently, if one suggested that a club should move further toward one side at the expense of the other, anyone could shoot back with a counter example of recent success from the other end of the spectrum. That’s a bit harder do now: two years removed from the Royals’ latest World Series appearance and three years out from the 2010-2014 Giants run, there isn’t a current standard bearer for the traditional point of view, even if that’s just cyclical and I’m using a somewhat subjective label.

The final four clubs standing in each of 2016 and 2017 — the Astros, Blue Jays, Cubs, Dodgers, Indians, and Yankees — would all rank among the top 10 of any industry poll of the league’s most progressive clubs. If you want to argue that their success is the result of variance, a blip, or mere coincidence, this development isn’t just the product of randomness. There’s an actual explanation. In these last two seasons, we’ve seen a fundamental change in the style of play (a greater emphasis on the air ball, quick hooks on starters, more aggressive bullpen usage, etc.) — particularly in the postseason. A progressive club, by definition, will adapt more quickly to such changes.

It is possible, even likely, that this late-season domination by progressive clubs will tail off as the factors changing the game become more standard. Provided the meta factors causing these changes (velocity, the ball, Statcast/TrackMan data) stay consistent and MLB doesn’t intervene (changing the ball or mound height), then it would follow that clubs would converge on the best postseason and team-building strategies. Two areas that seem ripe for exploration are how best to allocate scouting resources (multiple clubs have told me they are kicking around similar ideas) and investment in infrastructure and brainpower to be the first club to truly weaponize Statcast data, now that we have a big enough sample of seasons to have some confidence in the conclusions.

Leaning Back to Scouting?

There’s a case to be made that, in the face of this shift, a club leaning on more traditional valuation techniques could come out ahead. In practice, traditional valuation means weighing scouting reports, makeup information, and trust in player-development staff to drive decisions. Not in place of analytics, mind you, but with a greater weight on these resources than most clubs are currently using.

There are pitfalls to this approach. Something you learn quickly in a front office is that everyone internally knows the strengths and weaknesses of your scouts. I’ll quote Ken Rosenthal here on the topic of Houston’s recent shift in scouting:

While clubs also must filter out noise in their data, some consider the information coming from scouts as nosier. “There is a big population [of scouts] working in this game who are lucky to have jobs. They don’t know anything. They’re professional BS artists,” one executive said.

Analytics are an equalizer, forcing scouts and major league staffers to offer sound rationales for opinions the numbers do not support. The data is more complete. A pro scout might get only five looks at a player over the course of a season, while analytics provide fresh information daily.

Most scouts work hard and do the best they can, but everything that is asked of them sounds ridiculous when you spell it all out. A club is asking a pro scout (watching the minor and major leagues) to have expertise in the collection of makeup information, in grading tools, and then valuing those tools on the open market, all after seeing that player and his 24 teammates for just five or six days — and then to do it 20 more times in a row for about six months, all while living out of a suitcase, writing hundreds of reports, filming some video from their seat, filing expenses every week, and averaging less than two days off a week.

Anyone good at all those things could also likely work in a front office and be closer to the decision-making, sleeping in their bed much more often. With TrackMan data from nearly every minor-league game holding all kinds of invisible facts about the game they just watched, could any single scout seem essential to a progressive front office?

One of the clubs with which I worked tried to “scout the scouts” and see if it was possible to identify empirically who the best scouts were, or even their relative strengths. With most scouts working on one-year deals, you’ll likely promote/demote them at least once and give them four or five new contracts before you even know if they were good in year one. If they were, then they may have changed as an evaluator since then. Or, by virtue of keeping them around for five years and promoting them — and thereby creating a narrative that they’re good — you’ve raised their price before knowing if they are worth that price. It’s very possible the evidence would be hazy enough to suggest the money would be better spent on two hungry entry-level scouts instead.

On the amateur side, you’d need something like a half-dozen years of a scouting director’s entire draft board before you had a statistically significant idea of how good he was. In some organizations, you’ve already fired his replacement by the time you find out how good the first guy was. It’s essentially impossible to prove how good a scout is in a timely enough fashion that it would constitute actionable information. Simply making pro, amateur, and international staffs entirely composed of above-average scouts (that’s upwards of 50 evaluators in total) is easier said than done. You’d only have this scouting-the-scouts data (that is of limited use) on the scouts who had already worked for you for years. You don’t know who made the final call on a high-profile decision or what their whole list was (i.e. how lucky they were to sign the right player) if you’re poaching a scout from another club.

In the very unlikely event that you accomplished the feat of finding the right 50 scouts, the players you sign would likely stand out after a few years and other clubs would just poach the scouts that couldn’t be promoted since only good scouts were ranked higher internally. A raise only goes so far, as most scouts want the chance to make the decisions. The strategy of emphasizing traditional methods isn’t a bad one, it’s just harder to properly execute: there’s more unknowns and it’s a department in which many GMs have never worked.

The Reality of Scouting

There is no replacement for having current tool grades on players. The further from the big leagues, the more important it becomes. One R&D analyst told me he thought scouts were essentially snake-oil salesmen until he did the research and found their reports were the single most important input in a draft model. He didn’t mean their individual rankings were critically important or even the difference between the scouts was enormous and measurable, so much as the tool grades give invaluable context to what the stats and TrackMan data mean. It helped this analyst create a draft board that’s been tested to have greater predictability than any one scout’s list.

You can imagine how some clubs see this landscape and decide to treat scouting like an assembly line: young, cheap, enthusiastic workers who can handle computer-admin work and a video camera are primarily tasked with sending in tool grades so the office can use them in a comprehensive analysis for the most accurate valuations. It’s not hard to convince owners that this is the best way to get more correct evaluations for lower prices than traditional methods would provide.

You can also see how a scout would quickly feel worthless working in such an environment, no matter how enthusiastic they were on day one.

The Conundrum

There’s a cost to drifting too far toward the analytical end of the spectrum, as well. Clubs that run their drafts primarily off of a model aren’t frustrating just to area scouts; some crosscheckers and special assistants (the highest rankings scouts in the draft room behind the scouting director) feel powerless as a group in their draft rooms relative to the influence exerted by the analytics department.

Imagine you’ve a national crosschecker with 25 years of experience in the game. You kill yourself with one day at home a week for four straight months trying to learn everything about the players in your region leading up to the draft — this, after seven months of preparation scouting these players in the preceding summer and fall. Then you get in the draft room and start stacking the board as a department when a couple guys from the office who saw zero players — who don’t know how to scout and first started working for a club a few years ago — have more weight in the room than you do.

Now imagine you’re the GM of that club and there’s hard evidence with which you’ve wrestled for five years essentially proving that the stats and exit velocities, etc., from a player’s entire career are more predictive for decision-making than what a scout saw for one or two days in the spring.

By virtue of the crosscheckers, analytics team, and GM all doing the best they can, working as hard as they can, and making the best possible decision for the franchise with no agenda, any one disagreement over any of the nearly 1,000 names on the board could split the room apart and turn it into two warring factions. Maybe your top scouts will leave and go to a more traditionally run club where their voice has more weight in the process. Maybe they’ll bad-mouth you enough on the way out that you have to overpay to get experienced scouts on staff in the future. It may just be easier to embrace the assembly line scouting approach and dive further into analytics.

Given this context, you could also understand why there are plenty of instances of shouting matches in draft rooms (or just outside them). You could also see why more progressive-minded clubs see low-level scouts as a more fungible asset and thus have trouble keeping them on staff as they get more experience, while the more traditionally minded clubs have generally better and happier scouts along with more and better traditional scouting information, as that makes up a larger percentage of how they stack their board.

A Common By-Product of a Progressive Approach

In the event that this embrace of progressive methods continues and intensifies, many clubs will be valuing players using similar methods. This is already happening, as sources have told me a major reason why this offseason is moving so slowly is many clubs value players similarly, making it hard to match up for trades. Clubs also have similar valuations or wait-and-see strategies for free agency, leading agents to wait for separation or activity in the market. This was echoed by Ken Rosenthal in a recent article about why the offseason is moving slowly:

At times, the entire sport seems to suffer from an advanced case of paralysis by analysis. It’s a culture of fear — many GMs are reluctant to trade prospects and sign free agents, terrified of facing criticism if they make a mistake. The GMs become too process-oriented, passing on opportunities to improve, protecting their long-term plans. There is no incentive for them to act with greater urgency; few face an immediate threat to their job security…In too many cases, the goal is to win tomorrow, not today.

I’ve also seen concrete examples of the limitations of this approach. I’m aware of of a high profile instance of a progressive club turning down an attractive deal for one clearly superior prospect, instead choosing multiple lesser prospects, an example that would’ve shocked readers at the time and still would today. When I asked a member of that front office to explain the thought process, he said that they saw minor leaguers as gambles. Getting only one player for a premium asset means that the one prospect could bust and give them nothing (read: bad PR on a high profile trade), so they opted to diversify with multiple assets. I’m almost certain this club would’ve traded three similar assets for a premium prospect in a vacuum, so the PR of a high profile trade made them take different players than they would normally. I’m also aware of enough trade talks over the years with this club and other clubs with a similar process to see multiple examples of this thinking.

There’s another progressive club that would shock you with how their draft decisions are made, essentially letting their draft model make the picks so that no one decision-maker is tied too strongly to a certain pick. You can spot clubs that operate this way because, regardless of outcome, their upper-level executives never get fired. Rosenthal is hitting on a topic that drives some people in baseball mad, but is increasingly becoming reality given the sophistication of analysis that fans and even the owners regularly read online: the risk-averse, performative rebuild. “Trust the process” is increasingly no longer just a plea for patience to see a sound long-term plan through; it’s a vehicle for job security and for the public to defer judgment.

A Caveat

I’m intentionally painting with a broad brush here, labeling clubs as progressive or traditional. In reality, different tendencies can exist within the same club, department to department, due to a variety of factors. Most clubs in most years don’t have shouting matches in the draft room. That’s not because there isn’t tension, though: it just takes one blowup to know what’s worth fighting over and where you stand. If you think of progressive/traditional as a left/right political spectrum, I think, across all departments, basically every club these days in aggregate is no more extreme than center-left or center-right. There will be outlier or impulsive decisions at times, and some clubs may ignore some data that others think is key, but everyone generally has the same information now and acts accordingly.

What’s the Answer?

I’ve tried to be even-handed in presenting these issues, and I’m honestly in the middle on most topics — in part because I have some experience in the trenches on both sides. I’ve seen and, in some cases, created the research that backs up a progressive, analytically driven process that coldly comes to the best possible conclusion via the scientific method, often prioritizing data over scouting reports.

I’ve also seen clubs that embrace this philosophy too much and keep making the same mistakes with valuation due to the blind spots this approach can create: ignoring makeup, for example. This alienates their staff and the industry at large, all to improve a process at the margin. That strategy with that implementation often isn’t worth it. I’ve personally seen disenfranchised scouts from these organizations essentially give any information that’s asked for to a traditional club in hopes of getting a job they deem better for them and their families. A poorly implemented progressive approach has lots of similar “leakage” that isn’t included in their presentations to ownership outlining why their approach is superior.

On the other side of the spectrum, there’s sometimes a laughable ignorance of how analytics can enhance scouting along with a lack of curiosity to investigate it or even listen to someone talk about it. You can have all the scouting reports and makeup information and intel you want, but if the decision-makers can’t weigh that information correctly, the decisions will be poor.

There isn’t an objectively correct answer, obviously. It comes down to the execution and flexibility of the analysis and the internal communication about it. The most successful organizations generally have decision-makers who understand both sides are equally important, then weigh the factors differently depending on the decision, honestly communicate that to their staff, and ensure (as much as possible) that the staff is generally happy with the outcomes, which tend to be above average. Being tied to that kind of organization makes you more attractive to other clubs for promotions, as well.

Given the uneasy relationship between many scouts and their front offices, it would seem to further underline the need for executives that are fluent in both progressive and traditional approaches to valuation, with perhaps the most important characteristic being the people skills to get employees on both ends of the spectrum to buy into the club’s vision.