Tangotiger’s Projection Tests
Tangotiger posts the official results of his 2007-2010 projection tests. Specifically he tests CHONE, PECOTA, Oliver, ZiPS and the Marcel projection systems using wOBA.
It’s very in depth and there’s a lot of really great information here about how different projection systems fared for different “classes” of players.
David Appelman is the creator of FanGraphs.
Hopefully this will dispel the myth that PECOTA isnt a good projection system. Good work in there, though a lot of it over my head.
Well, aside from last year’s burp.
How so? PECOTA still does worse than a number of free alternatives. I also think Tango is being very generous by counting it as one continuous system, since there was clearly a drop off after Nate Silver left.
He does say that they are all about the same. Yes, you do have to pay for Pecota, but for me, I like that they factor in WAR(P).
To me, it appears as though its at worst only 8/1000ths away from being the best, which I would claim is statistically insignificant.
I thought the myth was “deadly accurate.” ?
PECOTA is in good hands now after being abandoned for a few years there. It’s back to being right there with other systems.
I was impressed with how well Oliver fared in these trials. Good to see.
-j
How does he calculate error?
I would copy and paste it, or try to explain it, but both would enable your extremely laziness.
I was wondering after looking at 3d. 0.04 seems way too big for an absolute defiance in that case.
so i guess we shouldn’t complain about losing chone for 2011
Why? CHONE did the best in most of the tests. But if your point is that they are all about the same usefulness, then point taken.
Great! We did an OK job predicting groups of players. So why do we even bother projecting individual players with these programs?
I have never seen this question answered. Every justification for the validity of these programs I have ever seen involves different sections that they use divvy up the player pool, and the accuracy of projecting any given one of those sections. Chone, say, only has to be ballpark with about 50% of the players to look good on paper–it could be completely missing the top 25% and bottom 25% in any given group, but if they all average out to a little bit above or below Chone’s aggregate projection, then Chone will look a lot better than he actually is. See what I’m saying?
Well I’m an idiot, Chone could be wrong 100% of the time individually and the group could still average out to just right.
Read what I showed in TEST #7 and tell me if that answers your question.
Well, to a degree. It tells me that they didn’t in fact do terribly 100% of time, but it doesn’t make me feel too much better about projecting individual players that only a fifth to a quarter were within 10 points of wOBA. But there doesn’t seem to be any better way to do this with a computer, so it’s tough to say.