Projection vs Projection

It’s almost opening day, and it seems like everyone is talking about projections.

When considering a projection, there are really two questions to be answered – what is the player’s “True Talent Level” right now, and how will he perform next year? Between now and the end of next year, his talent level very well might change, as he’s a year older and might recover from or succumb to injuries. Even then, there’s still the random variance of a single season performance. In this article I’d like to explore how some of the major projection systems work when predicting different subgroups of players.

I tested the following projections: PECOTA (2006-2009), ZiPS (2006-2009) CHONE (2007-2009) and my own Oliver (2006-2009).

By wOBA

The first test was to group the yearly projections to the nearest .010 of wOBA, and then see how that group of players actually performed. There were 468 players who had projections from all four systems, and had at least 350 plate appearances in the major leagues in the following season. As 2009 is yet to be played, and CHONE is not available for 2006, these projections to next year comparisons are for the 2007 and 2008 seasons. All four projections were tested on the same 468 players. The observed results were unadjusted major league stats, so that the results of the test would not be influenced by which park factors or MLE formulas I chose to normalize stats.

To read the results, CHONE of the players would have a wOBA between .375 and .385, averaging .380, 25 of them had 350 or more PAs in MLB in the following seasons, and those 25 players had an average wOBA of .363, so at that level CHONE was .017 high. Oliver was .008 high on 21 projections, PECOTA .027 high on 26, and ZiPS .014 on 26. The last line of the table shows the root mean square error (weighted by number of players). Oliver had the lowest mean error at .006, followed by CHONE .011 and PECOTA and ZiPS at .012 each.

wOBA	CHONE		Oliver		PECOTA		ZiPS
Obs	Players	Error	Players	Error	Players	Error	Players	Error
0.250	0	0.000	0	0.000	0	0.000	1	-0.067
0.260	0	0.000	0	0.000	1	-0.041	1	-0.018
0.270	2	-0.057	1	0.001	3	-0.013	1	-0.043
0.280	2	-0.018	4	-0.036	2	-0.045	4	-0.022
0.290	8	-0.033	9	-0.017	11	-0.030	13	-0.020
0.300	14	-0.005	23	-0.010	20	-0.013	20	-0.012
0.310	29	-0.006	33	-0.002	31	-0.007	19	0.003
0.320	44	-0.005	53	-0.005	37	0.002	51	0.000
0.330	74	0.004	81	-0.002	58	0.003	56	0.000
0.340	91	0.000	87	-0.003	66	0.004	66	0.002
0.350	57	0.004	68	0.001	80	-0.004	74	0.001
0.360	50	0.009	48	-0.003	56	0.011	55	0.012
0.370	34	0.011	21	-0.004	33	0.012	36	0.012
0.380	25	0.017	21	0.008	26	0.027	26	0.014
0.390	9	0.003	10	-0.002	17	0.014	19	0.020
0.400	13	0.019	5	0.020	15	0.019	7	0.017
0.410	7	0.017	2	0.011	5	0.017	4	0.019
0.420	4	0.037	1	-0.049	5	0.027	6	0.029
0.430	2	0.047	1	0.001	1	-0.035	5	0.041
0.440	2	-0.009	0	0.000	1	0.018	3	0.023
0.450	1	0.025	0	0.000	0	0.000	1	0.026
rms	468	0.011	468	0.006	468	0.012	468	0.012

By Age

The same 468 players, same rules, but now the players are grouped by age. The combined rms error is about the same for all, at .007 for Oliver and .008 for the other three. CHONE and ZiPS are a few points of wOBA high for most ages. Oliver under projects younger (pre-peak) players at .005-.010 points of wOBA, and over projects older players about the same amount. PECOTA is the opposite, being a little high for the younger players and a little low for the older ones. Oliver shows the lowest total error (bias) of -.002, but because of it’s error correlating with age, Oliver shows the highest r2 correlation factor of .206 (for ages 21-35, which have 12 or more players each).

Age	Players	PA	CHONE	Oliver	PECOTA	ZiPS
19	1	411	-0.004	-0.007	0.016	-0.022
20	3	1485	0.017	0.022	0.026	0.014
21	12	6587	0.003	-0.002	0.005	0.006
22	23	12205	0.002	-0.006	0.011	0.004
23	38	21423	0.001	-0.009	0.001	0.001
24	36	20677	-0.002	-0.009	0.002	0.002
25	37	20538	0.000	-0.006	0.001	0.002
26	39	21891	0.011	0.005	0.014	0.016
27	44	23580	0.003	-0.004	0.007	0.007
28	35	19038	-0.010	-0.011	-0.008	-0.005
29	34	17434	0.010	0.001	0.006	0.008
30	32	18491	0.007	-0.006	0.003	0.004
31	37	19013	0.020	0.008	0.011	0.015
32	24	13975	0.002	-0.004	-0.004	0.000
33	18	9702	0.003	0.004	-0.001	0.003
34	17	8545	0.005	0.004	0.012	0.014
35	13	7063	-0.001	-0.003	-0.003	-0.002
36	7	3714	0.000	0.001	-0.004	0.001
37	5	2295	0.007	0.010	-0.011	0.005
38	5	2580	-0.010	0.009	0.008	0.000
39	6	2699	0.009	0.008	0.023	0.007
40	1	548	0.026	0.031	0.037	0.036
41	1	434	0.016	-0.061	-0.005	0.011
rms	468	254328	0.008	0.007	0.008	0.008
bias	468	254328	0.004	-0.002	0.004	0.005
r2	468	254328	0.031	0.206	0.037	0.033

In the final part of this series, I’ll look at how minor league performances are evaluated.