Consistency Is Inconsistent
Few baseball terms are misused as frequently as ‘consistency’ and ‘volatility’. Much like some of the more arcane statistics that have fallen by the wayside — looking at you, W/L records and batting average — the terms are often conflated with overall performance. We thought they told us more about a player than they actually did. Pitchers perceived to be more consistent are often deemed to be more valuable than their volatile counterparts, as long as their numbers aren’t drastically different. While that perception might make sense from a logical standpoint it fails to hold up under the lens of quantification.
Consistency might keep fans and managers from reaching for the Tums jar, but it should not be used as anything other than anecdotal. It certainly should not be used as a performance marker. Why? Well, because consistency itself is inconsistent.
A pitcher consistent one year is not guaranteed to achieve a similar level of consistency the next year. My interest in consistency was rejuvenated this week when Buster Olney cited “enigmatic inconsistency” as a means of explaining why the Rockies might be willing to trade Ubaldo Jimenez. When Dave Cameron showed how elite pitchers experienced wild fluctuations in their game score, I hearkened to old studies of mine and felt it prudent to recap some of my findings on the matter.
For starters, allow me to backtrack to the assertion that consistently itself is inconsistent. That conclusion was drawn upon running an intra-class correlation on data similar to what Dave showed this week: a standard deviation based metric that measures performance variance on a game-to-game basis. An intra-class correlation is essentially a year-to-year correlation, but over a longer period of time. Instead of running four separate year-to-year correlations for a player over a five-year span, the ICC evaluates the relationship over the five years as a whole. In this case, I measured the per-pitcher relationship of the deviation metric over a five-year span, and found an ICC of just .05, well below the threshold signifying that consistency is a tangible and repeatable skill.
Further, the per-game deviation metric correlated at a coefficient no greater than .10 to any performance marker. In other words, consistency is inconsistent, and it doesn’t automatically lead to better performance. So why is the term used so much? The answer seems to lie in personal preferences. A pitcher like Jon Garland is thought of as more consistent than, say, Joel Pineiro, and that supposedly makes him more desirable to certain teams in specific positions. Understanding consistency is important since teams may make decisions based on the perception of consistency and volatility.
Those assessments, however, are based on past data that may not be on display in the current season. This speaks to the differences between game-to-game and year-to-year consistency. Some may value game-to-game consistency more than anything else, while others might not care how the end park-adjusted ERA is arrived at, as long as it is stable over a three-year span.
Think back to the 2009 offseason, when both Jon Garland and Joel Pineiro were available. A team with decent playoff odds might have gravitated toward Garland given his past consistency. A team with a high level of variance in its playoff odds, meanwhile, might find Pineiro attractive. If Pineiro performed at his lowest level of potential production, the team probably wouldn’t suffer all that much since it wasn’t expected to make the playoffs to begin with. But if his performance reaches its apex, the team might be able to sneak its way into the playoffs. The flaw in this operating mindset is that both types of pitchers might actually be equally projectable when discussing year-to-year consistency or volatility.
At the beginning of 2010, I ran a study to test that theory. The methodology involved culling together four-year spans with park-adjusted ERA for starting pitchers, and comparing the actual fourth-year mark to a Marcel-esque projection of the fourth year. The pitchers were broken up into five bins, based on the standard deviation in the first three years and the median: very volatile, volatile, volatile/consistent, consistent, very consistent. The “very” bins were compared to one another so that pitchers hovering close to the median weren’t compared as if they were innately different.
The results indicated that, whether the pitchers made 5, 10, or 20 starts, the very volatile pitchers all outperformed their projections in that fourth season.
The consistent pitchers, though posting better overall park-adjusted ERAs, either fell right in line with, or worse than, projections. Further, the very consistent pitchers ended up with a lower root mean square error: their performances were easier to project, and at a statistically significant level.
One such reason the volatile pitchers outperformed projections deals with the inherent cause-and-effect relationship. If a pitcher proved volatile in the wrong direction — really bad starts — he would be unlikely to make many more starts, and could end up missing the cutoff used in the study. On the flipside, a consistent pitcher who underperforms is likely to be given more slack based on his track record.
Overall, there is a difference in projectability amongst very consistent and very volatile pitchers, but it is far less evident in larger samples in the same season. Consistency is easier to project on a year-to-year basis than volatility, but, within the same season, it doesn’t correlate to overall performance. And the per-game consistency itself is inconsistent on a year-to-year basis. Consistency might be deemed important, but it really shouldn’t be the primary barometer to use when making decisions.
Eric is an accountant and statistical analyst from Philadelphia. He also covers the Phillies at Phillies Nation and can be found here on Twitter.
The middle paragraphs got pretty jargony with quite a bit of hand waving. Then shortly after you started using park adjusted ERA as a measure of consistency, and I stopped caring.
O for 2 on articles about inconsistency.
Ok, constructive criticism… I think writing an article about consistency without visuals (or rigorous computation) is a mistake. For it to mean anything to us, it needs one or the other. Even Dave showed his methodology, as bad as it was. 98% of us don’t know what an ICCSKMYDK value is. And while you proceeded to make a half assed attempt explain what is it, we still don’t have any frame of reference or connection to the numbers you then provide. Those paragraphs were unconvincing.
The notion of consistency is extremely tough to define for a pitcher, let alone calculate. Should we look at park adjusted ERA on a game by game basis? I’d say no, since we know that sequencing plays a huge role in short term ERA fluctuations. Why not use FIP or some other combination of metrics that are rooted in things the pitcher can control, or at least has more control over.
Throw all of the math and jargon out the window – show me a graph over the last two years of a pitcher’s K%, BB%, FIP, SWSTR%, average FB velocity – I don’t care what exactly – but pick a few things that they have direct control over, things that make sense (then you can add ERA to that and see just how much more volatile that number is) and let’s just LOOK at it. What do we see? Does everyone kind of look the same? Who looks different? Since we have such a tough time quantifying this, let’s throw it out the numbers. That would be a nice intro piece.
I think you got caught in no mans land between mathematically rigorous and accessible to the masses, which is exactly where you don’t want to be. One or the other, because the crappy explanations of your calculations with no real data or visuals give us very little to work with.
When they say someone needs to be more consistent, they mean consistently good, not giving their own average performance each time out.
Not that this is contrary to your article. I’m just saying they say consistent, but they mean he needs to get better.
Yep, that’s the big thing. When they say consistent they mean better. “A more consistent offense wouldn’t get shut out!” No a perfectly consistent offense could get shut out every game.
“Inconsistency” can also refer to the phenomenon of a guy being generally good, but then having one crap inning every game. I’d be interested in seeing this studied a bit more.
I don’t think I’ve ever heard sportswriters use “consistency” to describe what Seidman & Cameron are talking about (start-to-start compared to overall numbers). It means either the “big inning” or guys who seem to underperform their “stuff.”
Just want to underscore Joe’s et. al., point, namely, that these fangraphs articles are interpreting the word “consistency” in a statistical sense that bears little resemblance to its meaning in everyday baseball speech. This fact makes these articles strangely besides-the-point.
I think it’s a very good point, and an interesting discussion to have is this — would you rather have a guy guaranteed to give 6 IP and 3 R each start, a literal robo-pitcher, or a guy who might give you 8 IP of 0-1 R but could also go 4 IP and 6 R? People do tend to mean consistently good, because the whole reason they clamor for consistency in the first place is because they have seen a guy dominate before and can’t figure out how someone capable of shutting the opposition down can also look so terrible.
I strongly agree with Joe, Rob and bc. When a baseball manager, executive, scout, et. al. says that he wants a player to be more consistent what he means is that he want the player to always be good.
Whether consistency itself is consistent is a mildly interesting, but irrelevant topic.
I seem to recall an article – either on The Book Blog or P-Pro IIRC – that proved start-to-start inconsisency as an overall “good” thing. Since offenses are themselves inconsistent, it doesn’t make a whole lot of sense to value consistency from a starter that highly.
like what adam said. kudos.
I think that was on hardballtimes.com a few years back.
http://www.hardballtimes.com/main/article/same-old-same-old/
Without looking at that article, that would be hard to believe, since when you are beating your opponent, every subsequent run is worth less than the last. Therefore blowing your opponent out of the water is a horrible use of “resources”.
ian kennedy 2011 – is mr consistent since the birth of his baby girl
ubaldo, how can you be consistent when you play in coors? the problem w. ubaldo is he runs 5-6 miles the day after a start and long toss inbetween a start and sometimes in the rain, also he doesnt get good sleep sometimes before a start.
i think its hard to be mr 100% if you play in coors. no matter who you are. hes always had delivery malfunctions which can contribute to the consistency is inconsistent.
Does it matter why a player is inconsistent?
Would you agree there are some pitchers who perform consistently, but have inconsistent results because of the strength of their opponents?
Or does this type of thing just average out?
what Adam said…
Link to some analysis regarding batters’ inconsistency:
http://www.beyondtheboxscore.com/2011/7/20/2283275/what-determines-the-volatility-of-hitters-player-volatility-part-v
until we start looking at what HITTERS a pitcher faced, we’re never going to figure any of this stuff out.
A pitcher could be 100% consistent, with exactly the same “stuff” every day, and his results are going to vary widely because hes facing different lineups.
IE, if he faces the Yankees/Sox, and goes 6 innings and gives up 4 runs, that could be a perfectly consistent result with facing the Mariners and going 8 innings and giving up 2 runs. You’re essentially ignoring a variable that creates 50% of the data you’re sampling.
Its like looking at a golfer’s scores, and ignoring that hes playing on different courses (and ignoring PAR).
To further go on, we know that batters K% and BB%s are pretty consistent year to year, and that the batter has a large amount of control over it, so its ridiculous to assume that identical K% of two different pitchers mean the same thing.
I know someone is going to make the argument of “It should even out over a large sample size”, but I’m sure that if you look at the data, it doesn’t. The quality of teams/hitters/pitchers varies widely division to division.
Over a year it all balances out.
You got some evidence of that?
The chances of it actually evening out with an unbalanced schedule are miniscule.
Last I checked on BBRef, the OPS against for starters is pretty much +/- .150 from league average.
How has Joe Morgan’s name not come up yet and how do you write an entire post on consistency without bringing his name up?…
It’s a matter of inconsistency.
Ken Tremendous must be infuriated.
I believe the focus should be more about Inconsistency. In a head-to-head format that is killer. Point being Ricky Nolasco. 5 earned runs in his previous 5 starts (39 innings) and then 9 earned runs in 1.1 inning. At home. Against the Padres.
This isn’t rotographs.
Another crazily misused term: commodity. Typically, the writer means “asset.” All commodity is an asset, but not all assets are commodities.
People say “commodity” in a baseball context? Are they talking about Kevin Cash?
“For starters, allow me to backtrack to the assertion that _consistently_ itself is inconsistent. ”
you mean _consistency_, right??
I think you just reproved that mean reversion exists.
The classic mean reversion is that if you take the best and the worst from time t and then look at time not t, you find that they are more like other players.
Here is a link to a study I did on team consistency
http://cybermetric.blogspot.com/2010/04/how-much-does-team-consistency-matter.html
I found that it helps to be consistent, but not nearly as much simply scoring more or allowing fewer runs. And consistency did not matter the same way for both scoring and allowing runs: “The more consistent hitting teams win more for a given average runs per game while the less consistent pitching teams win more.”
Given the difficulty of the sport, inconsistency should be the expectation.
It’s one thing to excel at a very difficult task. It’s an entirely different thing to excel at a difficult task repeatedly, at the same level of performance … especially when some of your performance is out of your control.
“Consistency” in baseball should include a “range”. Sometimes it seems like we expect guys to put up incredibly similar seasons continually, even though we know fluctuating is the norm.