Yes, September Stats Count

Call-up season is upon us. It’s a lovely time of year. Even guys like Canadian Legend Simon Pond can get a bit of playing time here and there as teams give their regulars a bit more rest. It also means that a greater proportion of league-wide playing time than usual is going to a group of players whose true talent is replacement level or below. This situation inspires an assertion that I occasionally see this time of year: September statistics should be ignored or weighted less heavily because of the influx of less-talented players. It is easy to understand the motivation behind such a claim, but I think it is problematic.

Let me begin with a few qualifications. People probably mean different things when they talk about discounting September performances. If they mean it in a scouting sense — something like, “yeah, he had a couple of home runs, but they were off of [insert AAA-lifer reliever here] who was a September call-up” — then that makes some sense. One would also assume that those same sorts of considerations are made for scouting observations throughout the season. For now, all I’m concerned with is the stronger, statistical interpretation: that all players’ statistics accumulated in September need to be heavily discounted or set aside when used to measure a player’s true talent because of the watered-down talent pool.

Perhaps some people mean that we shouldn’t judge the performances of the call-ups themselves by their September performances because of the reduced talent level. I agree that we should be careful in judging the performances of who only play in September, but the real problem with doing that is that one month of even full-time play is far too small a sample to make a judgment from the statistics alone, and that goes for any time of the season.

The most important qualification I want to make is that what we really need is an empirical study. I did some searching and asking around and was not able to locate any such study (and if anyone knows of one, please let me know). Perhaps I’ll do one in the future (and if anyone else wants to do so, go ahead). It would be nice to have some empirical support or refutation. At this point, I’m presenting a hypothesis based on the following thoughts.

Like those who “reject” September statistics due to call ups, I am concerned with getting a good measure of players’ true talent (i.e., projections). However, I think that rejection would do more harm that good. While it is probably true that there is some watering down of the talent pool, just how significant is it? A detailed study would need to measure how much playing time goes to AAA-call ups. While many of them are probably replacement level, there are “true talent” replacement level players getting playing time in the major leagues all season long (e.g. Willie Bloomquist). Probably not as big a proportion as in September, but it happens. Moreover, not all of the players who get called up are replacement-level players. Some of them are legitimate prospects whose true talent is above replacement level at the time.

We need to figure out if the increase in scrubs’ playing time has a significant impact relative to the rest of the season. It isn’t as obvious as it might seem. While some teams who are out of it are going to give them more playing time, teams that are still in playoff races probably aren’t going to give them much time. There are also non-contenders like the 2011 Royals that have so many young players on the roster to begin with that they don’t want to sacrifice that developmental playing time to replacement-level scrubs.

Perhaps the quickest way of expressing my position is that while there is probably something of a watering-down of talent level during September, any potential benefit of throwing out or heavily discounting September statistics wouldn’t be big enough to outweigh the cost of throwing out the one month sample for everyone. While a one month sample by itself tells us practically nothing significant on a statistical level for an individual player, my hypothesis is that throwing it out for all players would make our projections worse on balance.

One final practical note against discounting September statistics: even if the talent level of those getting regular playing time is significantly lower in September, September comes every year. So until major league baseball decides that games in September don’t count toward playoff eligibility, teams are probably want to going to want projections for how their players are likely to perform during the whole season.