Where the Fans and the Numbers Disagree

It’s pretty convenient and informative to have two different and legitimate versions of something. For example, there are a bunch of different baseball stat projection systems. Most of the time, they’re going to agree with one another, because they’re founded on more or less the same information and in the end all projection systems project based on recent performance. But it can be fascinating to identify players or teams where projection systems don’t see figurative eye to figurative eye. For example, for this coming season, Steamer doesn’t love the Reds’ starting rotation. ZiPS, meanwhile, is considerably more optimistic. It’s of interest to examine the reasons why that might be.

Along the same lines, we can turn our attention to player defense. FanGraphs keeps track of both UZR and Defensive Runs Saved, and those are two different methodologies, yielding two different sets of results. On several occasions, people have compared and contrasted the two. But FanGraphs also keeps track of the results of the annual Fan Scouting Reports. To my knowledge, less has been attempted with that data, and I thought it could be fun to see how fan opinion compares to stat opinion.

In case you don’t know, the idea behind the Fan Scouting Report is this: fans watch a lot of baseball. Especially baseball with their favorite teams. So they should be able to give some evaluation of the defensive abilities of their teams’ players, and this information becomes more valuable as the sample size grows. In the end, players can be given an overall defensive rating, and then players at different positions can be directly compared. In theory, it should match up well with FanGraphs’ Defense rating — that is, UZR plus position. They’re different approaches to try to get to the same destination.

Now, there are some potential issues. The Fan Scouting Reports can have pretty small sample sizes. The people filling them out are different for every team. And many of the people filling them out are likely to be familiar with advanced defensive metrics, which can bias them in certain directions, even if subconsciously. We don’t have perfect data, but we have data we can play with, and if we expect that the Fan Scouting Report and Defense rating will largely agree with one another, then, well, where don’t they agree?

I decided to look at the window from 2011 to 2013, and I identified all players who put in at least 1,000 innings in the field. I then removed the catchers from the sample, because our Defense ratings for catchers are probably incomplete. Catcher defense is a little mystifying. For each player, I calculated Defense per 1,000 innings. I made some necessary adjustments for part-time DHs. I then plotted that against their overall ratings from the Fan Scouting Reports. Not surprisingly, there’s a lot of agreement:

def_fsr_11_13

The linear relationship is unmistakable, and it would be weird if it weren’t there. Generally, a worse defender is a worse defender by both systems, and a better defender is a better defender by both systems. Using the best-fit equation, one’s able to calculate an “expected” Def/1000. This is the Defense rating, per 1,000 innings, we’d expect based just on the Fan Scouting Report results. You can then compare the actual Def/1000 to the expected Def/1000. The pool includes 318 players, and for 38 of them — 12% — the two systems differ by at least ten runs per 1,000 innings. The greatest individual difference is about 19 runs. The average difference is about five runs, and the median difference is about four runs.

Right here, you can look at a spreadsheet for all 318 players. That’ll show you both Def/1000 and expected Def/1000, so you can look up your favorite regular or semi-regular. But what about the extremes? There might as well be a couple tables in this post, since the point is to identify the players with the greatest disagreements. First, the ten players the fans have liked more than the numbers the most:

Player Innings Overall Def/1000 expDef/1000 Difference
Eric Hosmer 3807 57 -14.5 0.3 -14.8
Nick Markakis 3715 64 -10.9 3.6 -14.5
Carlos Gonzalez 3046 77 -4.4 9.6 -14.0
Carlos Beltran 3466 59 -11.7 1.2 -12.9
Eduardo Nunez 1709 41 -19.5 -7.1 -12.3
Kosuke Fukudome 1262 59 -11.1 1.2 -12.3
Matt Kemp 2867 63 -8.8 3.1 -11.9
Albert Pujols 2633 64 -8.3 3.6 -11.9
Domonic Brown 2039 48 -15.6 -3.9 -11.7
Shin-Soo Choo 3428 56 -11.3 -0.2 -11.1

Leading the way is Eric Hosmer. Hosmer, by DRS, has been a below-average defensive first baseman. UZR has thought even less of him over the same period of time. Yet the Fan Scouting Report has pegged him as being above-average, rubbing shoulders with Brandon Belt and Adrian Gonzalez. Hosmer’s defensive numbers picked up in 2013, but the fans thought he was equally good in 2011.

Right behind Hosmer is Nick Markakis. The fans have him slipping — he’s gone from a 79 overall rating to 71 to 64 to 58. Still, the advanced numbers suggest the trend began earlier, and they haven’t liked Markakis much since 2008.

Then there’s Carlos Gonzalez. The fans have loved him. The numbers have liked him all right in 2011 and 2013, but they didn’t like him in 2012, and the fans have rated him as highly as Carlos Gomez and Peter Bourjos. It’s possible that there’s some kind of Coors Field park effect that’s making Gonzalez’s advanced numbers look worse. Alternatively, he could just be a little overrated. We don’t actually know who’s right!

You can go over the rest of the table yourself. How about the other side? Here are the ten players the numbers have liked more than the fans the most:

Player Innings Overall Def/1000 expDef/1000 Difference
Juan Uribe 1904 61 20.7 2.2 18.6
Luis Valbuena 1552 50 14.7 -2.9 17.6
Craig Gentry 1665 69 22.0 5.9 16.1
Lorenzo Cain 1441 65 19.2 4.0 15.1
Nick Punto 1298 55 14.2 -0.6 14.8
Jarrod Dyson 1271 50 11.7 -2.9 14.7
Jhonny Peralta 3479 52 12.4 -2.0 14.4
Casey McGehee 1954 35 3.9 -9.9 13.9
Miguel Tejada 1083 37 4.2 -9.0 13.2
Pete Kozma 1254 56 13.0 -0.2 13.2

And there’s our headliner, in Juan Uribe. You look at Uribe, and you wouldn’t have any reason to believe he’d be an elite-level defender. The fans have seen him as being fine, even above-average, but the numbers have fallen head-over-heels in love with his performance. This is the greatest difference in baseball, and defense is a big reason why Uribe’s player page puts his 2013 season at 5.1 WAR. DRS and UZR are both rather fond of the guy.

Luis Valbuena shows up in second, and I don’t really have anything to say about him, except that, again, DRS and UZR both suggest good things of his work at third base. In a few ways, this could be sample-size noise. Or it could be a real thing, I don’t know. I can’t imagine having a strong opinion about Luis Valbuena.

Then there’s Craig Gentry. Fans have liked him a lot — they’ve ranked him around Darwin Barney, A.J. Pollock, and Jack Hannahan. The metrics put him in second in all of baseball, between Andrelton Simmons and Manny Machado. DRS and UZR say the exact same things. The stats think Gentry has been an elite-level sprinter in the outfield. The fans have thought, hey, that guy’s pretty good. Pollock, incidentally, would show up right after Kozma, if the table included an 11th player.

And again, you can review the rest. Jhonny Peralta remains an interesting case. UZR has liked him a lot. DRS has thought he’s been fine. The fans have thought he’s been mediocre. A lot of other people around the game would agree with the fans, and based on his free-agent contract, the Cardinals don’t agree with the fans. The quality of Peralta’s defense is controversial, which means Peralta’s controversial, but then if the Cardinals believe something, you might want to stop and think about that.

I realize I haven’t dug into why the fans and the numbers might disagree on certain players. I mean, disagreement in some places is inevitable, just because they’re different systems with different inputs. There are probably some sample-size issues. It’s important to recognize that we don’t know which side is right, in the case of a disagreement. The numbers are objective, but the fans might also see some things that the numbers just don’t. It’s interesting enough that there are disagreements, sometimes significant ones, and I would think one would have the greatest confidence in defensive numbers that match up across the board. So, when the fans and the numbers disagree, perhaps that’s cause for additional consideration. Maybe the fans are actually full of crap, but given the well-known issues with the statistical data, they probably shouldn’t be ignored.





Jeff made Lookout Landing a thing, but he does not still write there about the Mariners. He does write here, sometimes about the Mariners, but usually not.

42 Comments
Oldest
Newest Most Voted
triple_rMember since 2020
12 years ago

Where does Adam Jones rank?

triple_rMember since 2020
12 years ago
Reply to  Jeff Sullivan

Bah! Did not see that.

j6takish
12 years ago

I’m surprised Jhonny Peralta is that low on the list. His positive advanced fielding metrics are baffling to casual fans and even on stat heavy websites I see a collective “Wtf?” every time he is noted as a decent defender

Mr Punch
12 years ago

Interesting start. The why is the issue, though. How would the fans’s ratings correlate with fielding percentage? I’d guess that fans still don’t balance errors and range properly. Are some OFs they rate “too low” guys whose main asset is speed? Does Markakis’s arm give him an anecdotal edge? First base, I think, is a separate matter, as it has something in common with C: the main job is to catch thrown balls.

olethros
12 years ago
Reply to  Mr Punch

I’ve seen a lot of players/managers complement his pre-pitch positioning. Basically, the argument is that he gets to an above-average share of batted balls for his position despite below average mobility because he’s already where they’re going to hit it.

Whether there’s any validity to that, I have no idea, but it would explain the perceived difference.

JorgeFabregas
12 years ago
Reply to  Mr Punch

I think you’re right about the range bit. Difficult to determine visually, especially since television broadcasts often don’t show the initial jump.

Other than the rangy outfielders, another type in the underrated fielders group seems to be doughy infielders. And you could throw Nick Punto in with them to get an atypical body type category.

Sparkles Peterson
12 years ago
Reply to  Mr Punch

Peralta I know is the exact opposite of this phenomenon. A large part of his positive ratings in advanced defensive stats is how sure-handed he is with the balls he gets to, but fans just see a chubby guy playing shortstop, assume (Perhaps correctly, perhaps incorrectly) that he doesn’t have the range, and write him off as simply as that.

Minus the fat thing, the same thing may be at play for Kozma (Though it could simply be the FanGraphs set forcefully rejecting Nichols’ Law): The sure-handedness isn’t being weighed appropriately vs. the unexceptional range.

Dave
12 years ago

I wonder if the game announcers are influencing the fans, even subconsciously. I mean, not John Sterling, but other announcers.

SeaBass
12 years ago
Reply to  Dave

According to Hawk Harrelson, Gordon Beckham is the best defensive 2B in the league…lol

ReuschelCakes
12 years ago

funny that as a whole there seems to be some relationship between fans’ perception of fielding value and actual batting value.

maybe the fans are no better than the Gold Glove voters!

Derek Jeter
12 years ago
Reply to  ReuschelCakes

What’s wrong with the Gold Glove voters?

Miguel Cabrera
12 years ago
Reply to  Derek Jeter

Exactly. That is like saying there is something wrong with MVP voters! Ridiculous!

Rafael Palmeiro
12 years ago
Reply to  Derek Jeter

Absolutely nothing!

DBA
12 years ago
Reply to  ReuschelCakes

This.

jim S.
12 years ago

I watched about 2200 of MLBAM’s condensed games last season (I know, I must be nuts)– which means I saw nearly all the difficult plays which had to be made — and IMO the two most overrated defenders are Peralta and Nolan Arenado. Peralta just has no range and I have no idea how BIS figures Arenado saved all those runs.

Angelsjunky
12 years ago

The “why” is crucial. We could say that part of it is homerism, but the other part – and why I generally don’t like statistical projections – is that formulas don’t take into account the human being on the field, only as they are reflected in the numbers. In other words, there is a narrative, a story, behind each player that involves lifestyle, living situation, family, psychology, and other factors that can’t be quantified.

Take Albert Pujols, for instance. What the formulas don’t/can’t know is that for the first month or two of 2012, when he started terribly, he was separated from his family and quite unhappy about that. He turned his season around and, while he wasn’t quite Albert v. 2009, he was Albert v. 2010 for a few months. Then in 2013, he suffered with plantar fasciitis, which isn’t exactly a great thing for a power hitter and can only hurt speed and defense.

If Pujols is healthy in 2014–and that’s a big if–then there’s a good chance he has a bounce back to at least 2011-12 levels. Now let’s look at the different predictions:

Steamer: 138 games, .282/.357/.515, 3.8 WAR
Oliver: 143 games, .265/.332/.448, 1.5 WAR
Fans (61): 135 games, .287/.359/.499, 3.5 WAR

Steamer and the Fan prediction are almost exactly the same. So what’s up with Oliver? I can only assume that it is completely unaware Albert’s narrative. I’m not sure why Steamer is closer to the Fans prediction, but perhaps there is some element in its algorithm that takes into account some intangible factors.

Prince Fielder is another example – Steamer and the Fans are closely aligned (3.6 and 4.1 WAR, respectively) while Oliver seems to think 2013 was the beginning of the end and that he’ll actually decline rapidly (1.2 WAR). It seems quite logical that his divorce last year drastically impacted his numbers. Or 2011 Husband of the Year, John Lackey.

To put all that more succinctly, life actually impacts performance, and no statistical formula that I’m aware of takes into account life outside the ball-field, but fans can and do, I think.

Bill
12 years ago
Reply to  Angelsjunky

Life does impact performance without a doubt, but it is completely unpredictable and, I suspect, consists of distractions that good players have long ago learned to filter out. But, had Pujols started 2012 on a tear and then slowed down, someone would have pointed out how Albert was better when he didn’t have family distractions.

NS
12 years ago
Reply to  Angelsjunky

“life actually impacts performance”

That’s deep, bro.

The fact that fans “take into account life outside the ball field” is not necessarily a good thing, since they are likelier to do so badly or inappropriately to suit a variety of odd and oddly powerful cognitive biases like, say, I don’t know, homerism, angelsjunky.

ankle explosion hr celebration
12 years ago
Reply to  NS

“The fact that fans “take into account life outside the ball field” is not necessarily a good thing, since they are likelier to do so badly or inappropriately to suit a variety of odd and oddly powerful cognitive biases like, say, I don’t know, homerism, angelsjunky.”

exactly. It’s too complex and we don’t have the information. It will be different for different players. Some guys will take personal troubles and turn them into great performances, and others will turn them into stupid mistakes.

And we probably, for the foreseeable future, will not be able to estimate it, if for no other reason than we have no idea about all of the personal issues players face. Probably many times there are marital troubles we don’t hear about. Conversely, things which seem like they would be a big deal maybe sometimes aren’t.

I can understand the desire to take these factors into account, but there’s just not an unbiased way to do so at present.

Jon L.
12 years ago

The biggest difference that jumps out to me is all the stars in the first table, versus all the obscure names in the second. We know Pujols, Kemp, & Markakis are strong fielders, even if that information is wrong or outdated, and yes, no one feels strongly about Luis Valbuena (or Craig Gentry).

jwise224Member since 2024
12 years ago
Reply to  Jon L.

This was my thought, too. The list of “overrated” names are all established players that benefit from fans’ biases. The “underrated” players are mostly afterthoughts who are role players on their respective teams.

ReuschelCakes
12 years ago
Reply to  Jon L.

I feel strongly about Craig Gentry

E
12 years ago

to me it looks like the fans are sometimes basing their defensive rating off of who looks more athletic and in turn appears to be a better defender.

Guys like Kemp, Gonazalez, hosmer, brown, markakis all have what someone might consider the ‘look’ of a good defender.

Conversely, guys like peralta, mcgehee, tejada, uribe are a bit stockier and look like they should be slow. and guys like dyson, punto, valbuena may be perceived as having weak arms because of their size.

Not every person on those lists fits those profiles but I think a lot of the time people base their opinion on whether a player looks like he should be a good defender or not, regardless of the plays they make or do not make.

bobaloo
12 years ago

Given how poor our memories are, I wonder can we report the defensive measure in year t without being influenced by year t-1? There is evidence that humans improperly (in the Bayesian sense) put too much weight on more recent observations even ignoring the forgetting aspect of past observations. The defensive metrics are all based on the specific year. That is, there is no laggged variable for previous years. A simple first test would be if instead of averaging year (2011-3) defensive metric, would be to report for each year a weighted average of t and t-1 of the defensive metric and then see if that does better matching to fans’ report.

Xeifrank
12 years ago

This is one of your best posts. Keep up the good work! 🙂
vr/Xei

dirtbag
12 years ago

Shouldn’t your “Defense” rating be raw UZR, without the positional adjustment?

The Fan ratings are giving Goldschmidt/Gonzalez/Belt high 50 ratings to indicate that they’re better than the average 1B. But when you use UZR with the positional adjustment, there’s not a single qualified 1B in baseball last season with a positive “Defense” rating — they’re all below average. And that’s biasing your results.

Kris
12 years ago
Reply to  Jeff Sullivan

But if you ever pay attention to individual votes in the FSR, you will see that the fans blatantly ignore the instruction to ignore position. It’s the main reason why I have a hard time putting much of any stock into the FSR, when different fans are voting on different criteria.

tdi1985
12 years ago

I wonder if some of the difference, at least on the end where fan’s overstate a player’s ability, can be explained by a player being out of their natural/appropriate position. Think Choo. His defensive value is pretty low in Center, but fans still evaluate his overall ability as a defender higher than he can practically perform in Center.

bjsguess
12 years ago

Jeff – suggestion for a future Fans vs other projection systems – why the heck is Mike Trout hated so much by the Fans? I couldn’t find another player that has similar characteristics (young, improving year by year) where the Fans project 20%+ regression from 2012/2013 and are 10-20% lower than even the projections (which actually aren’t predicting much a decline when you factor in projected PA’s). Curious if other players also face this type of a bias.

gnomez
12 years ago
Reply to  bjsguess

Because we’re terrified Trout might actually be a Terminator.

Matthew
12 years ago

One thing that could affect the difference is fielder positioning. I know from watching a bunch of Rockies games that Dexter Fowler often looks like an exceptional center fielder, yet grades consistently below average statistically. He doesn’t always run the best routes but he has great closing speed. However, he was often set up too shallow which hurt his ability to get to deep left and right center. I think I also remember seeing an article or comment sometime about Eric Hosmer playing especially close to the line. He might have decent range but because he has poor pre-pitch positioning, he covers a smaller zone.

mcneo
12 years ago
Reply to  Matthew

I read the article about Horner as well. Hard to say who is right. I like that we have some objective defensive stats, but I still question the.

walt526
12 years ago

Yeah, having seen a good number of Tigers games over the past few years, I just don’t believe that Peralta is an above-average defensive shortstop.

I’ve wondered if part of Peralta’s strong defensive numbers the past few years were due to extra OOZ opportunities by virtue of playing next to a very (historically?) poor fielding 3B. Although he had a good UZR in 2011 (when Inge got a small majority of the playing time), so perhaps not.

pft
12 years ago
Reply to  walt526

I think positioning and pitching plays a huge role in these stats. Positioning is largely determined by scouting reports, coaches and the teams approach to the hitter. The effectiveness of the positioning is determined by the pitchers ability to locate his pitches.

zkello
12 years ago

The reason Peralta is rated high by the numbers is because he is very sure handed. He makes all the plays he can get too. What the fans see is his terrible range; he is very slow and it’s painful to watch a ball roll by just out of his reach. It would be interesting to see which is the more important defense trait: sure-handedness or range?

ReuschelCakes
12 years ago
Reply to  zkello

“hmmm, I wish they had some stat which accounted for this factor of “range” you speak” – Bill James

pft
12 years ago
Reply to  zkello

Peralta is not very flashy and makes plays look easy. Players like Pedroia dive for every GB and make routine balls look difficult.

brendan
12 years ago

Jeff mentions the possibility of a zips/steamer comparison for reds pitchers, or in general to highlight the difference between them. Yes please! If there is already such an article, please link to it.

pft
12 years ago

“The numbers are objective, ”

Not 100% so. You have interns who must make subjective determinations on how hard a ball is hit and classifying it as a LD, GB, fliner, FB which play a big part in the assigned runs saved. We can’t check for bias at home fields since we still don’t have H-A splits for defensive stats, which I find mind boggling

Seattleslew
12 years ago

This is a bit off topic but this is the most appropriate place I could find to post.

How was Randy Johnson’s body able to hold up for that long? How is it possible that he was able to pitch more effectively in his late 30’s than any other time in his life?

I know the simple answer would be, “he was just that talented” but is there any other explanation for this phenomenon?