The Observer Effect and Fan Projections

In high school, I — like many — was taught that the Heisenberg uncertainty principle refers to changes that the act of observing can have on the phenomenon being observed. So, for example, if I (for some unknown reason) were attempting to observe a particle and was using some sort of device that would bounce a photon off of said particle, I would inevitably change the particle’s position or momentum. Thus, in attempting to measure it, I would’ve have actually changed it. To measure it au naturale would, in this incredibly hypothetical scenario, be impossible.

Despite the fact that my high school was impossibly prestigious and reputable, it turns out my teacher only had it sorta right. The actual definition of Heisenberg’s uncertainty principle (per Wikipedia, which is always right) is that “certain pairs of physical properties, like position and momentum, cannot both be known to arbitrary precision. That is, the more precisely one property is known, the less precisely the other can be known.”

What my physics teacher was describing is actually known as the Observer Effect. And while (again, per Wikipedia) the two are related — Heisenberg was integral in defining the Observer Effect, too — it’s usually the Observer Effect that people are meaning when they say Heisenberg uncertainty principle.

And it was the possible influence of the Observer Effect that I considered when FanGraphs Overlord David Appelman announced the arrival of Fan Projections this past weekend.

As as been noted in these electronic pages — most recently by Dave Cameron — the idea of Fan Projections/Community Forecasting relies on the Wisdom of the Crowd, a concept ably explored by James Surowiecki in a book of the same name. That book begins with an anecdote about Statistical Wunderkind and Pretty Racist Francis Galton. The story (dramatized by Radio Lab here) goes like this: Galton visits a country fair. At the fair, he comes across a weight-judging competition, where fair-goers are encouraged to guess the weight of an ox. No one guesses the weight of the ox exactly, but Galton, gaining permission from the organizers to do so, collects all 787 entries and finds that the “average” guess comes out to 1,197 pounds. The actual weight of the ox? 1,198 pounds — i.e. closer than every single one of the entries.

That’s an exciting discovery. But the difference between Galton’s experiment and the one we’re conducting here is that the Crowd (read: you and you and you) can see the results in real time. If each of the fair-goers from the Galton anecdote had been able to see the guesses that had preceded their own, would that have affected the results? If you and you and you can see Derek Jeter’s current projection before making your own, how will that affect your own assessment of Jeter?

In other words: Will the Observer Effect come into play here?

The answer is: I don’t know. Of course, there’s a big emphasis on the “I” there. Here are some subjects in which I have little in the way of expertise: physics, statistics, personal hygiene. (The last of these, I recognize, isn’t wholly relevant to the present conversation; still, it’s the truth.) But you know who might know? People who are smarter than me. People like Tom Tango and Dave Cameron. So I asked them. Below are their respective responses.

Here’s Tom Tango:

No, I don’t think there will be a bias caused by seeing other fans’ forecasts. You will note that David shows the selections a bit differently from the standard line. Furthermore, fans are notoriously stubborn.

Having said that, it’s a simple matter to look at the forecasts early on and later on and seeing if the standard deviations of the selections are tighter the later the selections. I’d be shocked if you find anything.

Personally, I’d be shocked if I found anything, too — but only because that would mean that I’d checked.

And now Dave Cameron:

I think it’ a legitimate issue, but David has designed the inputs in such a way that limit the opportunity to just repeat what has already been done. Since most of the rate stats are calculations based on the raw inputs, the fans aren’t going in and just projecting players for the .360 wOBA that everyone else already concluded was likely. Instead, they are projecting the components, which are less likely to be observed. I know that I can tell you what the wOBA/WAR projections are currently at for a few players, but I have no idea how many doubles and triples anyone is projected to hit.

I think that will mitigate some of the problems the observer effect raises. We can’t eliminate it, of course, but it shouldn’t be a big enough problem to ruin the projections.

So the consensus here seems to be that no, mostly likely the Observer Effect will not influence (or noticeably influence) the Fan Projections. Consensus is different than fact, of course, but it’s good enough for now.