How Fans and UZR Disagree: the Giants

When attempting to judge a player’s defensive skills, I prefer to look at a couple different metrics. Chief among them is the Fans’ Scouting Report (FSR) and Ultimate Zone Rating (UZR). Both are on the same scale, runs relative to average, but come to their conclusions in disparate ways. The FSR ratings come from surveys filled out by fans while UZR is entirely algorithm-driven. They take such different paths, but how close do they come to arriving in the same place?

One hundred and thirty-one non-catcher* hitters qualified for the batting title in 2009 and 127 such players in 2010. The correlation between the FSR and UZR was 0.56 in 2009. That rose to 0.60 in 2010. That is not a perfect match up, but a perfect match up would render them redundant anyways. For two systems that rely on such completely different inputs, I find that an agreeable level of correlation.

*UZR ignores catcher’s defense.

Just as agreement on a macro level is not interesting, so is general agreement on a micro scale. Precise agreement is worth a minor note. Hunter Pence is such a case as both FSR and UZR put him at three runs above average in 2010. Overall, however, I find it more interesting to look at the Andrew McCutchens, the Yuniesky Betancourts and the Matt Kemps of baseball. All three of whom had a greater than 15 run difference between FSR and UZR this last season.

Before looking at individual players however, I wanted to call attention to team-level ratings. While figuring out an individual player’s contributions are fraught with error and sampling issues, team defenses are much easier to gauge. Two teams stick out, and for opposing reasons, when looking at FSR and UZR: the Giants and Rockies.

In 2009, fans thought that the Giants defense was bad at -26 runs. They warmed to them in 2010, rating them as average, +1 run, as a unit. UZR has a very different interpretation, interestingly giving the Giants the same +56 runs in 2009 and 2010. In fact, the only Giant players to record a negative UZR with at least 100 innings of play at a position in 2010 were Aubrey Huff’s -1.3 runs in right field and Buster Posey’s -1 run at first base. The 82 and 55-run spreads between FSR and UZR for the Giants were the highest in the league each year.

Ignoring catchers makes this even a slightly bigger gap in San Francisco last season as Buster Posey was second on the Giants in FSR with nine runs while Bengie Molina was second worst with -5 runs and nobody had an opinion about Eli Whiteside other than that he has an awesome name.

The two biggest differences in FSR and UZR for the Giants come from Andres Torres and Edgar Renteria. Renteria is the biggest offender of the two. Fans rated Renteria well below average, a combined -21 runs for 2009 and 2010, but UZR saw the short stop as reasonably above average, a combined 2.7 runs. Andres Torres’ play is another example, garnering 29.4 UZR runs across the entire outfield but only 11 runs via FSR.

As big as the disagreement is on those two, the spread for the Giants does not come from a divergence of opinion on just those few players, but rather appears to be more of a systematic difference. Renteria and Torres totaled about 42 runs of difference between themselves, but the gap between FSR and UZR for the entire team is 137 runs.