The Most and Least Confident of Projections

A few weeks ago, I wrote a piece for this site wherein I noted that the Chicago White Sox rotation was (a) projected to be very bad in 2018 and (b) composed to a great extent of starting pitchers of whom the following could be said: “That guy? Who knows what he’ll do this year.” My editor Carson Cistulli titled the piece “The White Sox’ Rotation Could Be Anything,” and he was right. The White Sox’ rotation could be anything, because it’s full of players whose track records cause most projection systems to raise their digital shoulders, put on their best Robert De Niro face, and shrug magnificently right in your face.

In the comments to that piece, some of you expressed an interest in reading more about variance in player projections. So, here are some words and tables on that subject.

First, courtesy of Jared Cross at Steamer, here are the 10 starting pitchers whose 90th-percentile projection for runs allowed per nine innings (RA9) differs most from their 10th-percentile one. I put a floor of 100 innings on the query to make sure we were getting pitchers in whom someone might be interested in reading. That said, because the figures I’m reporting here represent “true talent” projections (i.e. there is no added uncertainty based on the number of innings they are projected to pitch), the list should not be biased towards pitchers with fewer projected innings. The same general statements are true of all the tables you’ll see in this piece.

That is certainly a list of pitchers, isn’t it? There’s one White Sock on it (Carson Fulmer) and also an awful lot of pitchers who have, at one point or another, had arm injuries that kept them out for some period of time. By my count, five of the players on this list spent some significant portion of 2017 on the disabled list, and another handful have had persistent injury difficulties over the course of their careers. It turns out that injuries affect pitching performance (usually, for the worse). It further turns out that a pitcher who produces two wildly different seasons can cause a projection system to throw up its hands in exasperation.

Let’s move on to the low-variance starters, shall we?

You may recognize this as basically a list of the 10 best pitchers in baseball from the last half-decade. And it is, but not entirely by design — the system isn’t trying to tell you that better pitchers are less volatile in their performance year over year than worse pitchers (though whether this is true would be an interesting question to explore empirically in the future, and perhaps I shall). It is telling you instead merely that the pitchers about whose performance Steamer has the least uncertainty this year are in fact also, generally speaking, very good pitchers.

This turns out to be a feature of the way projection systems like Steamer work, which is that they take in as much data about performance from the past as they can and then use that data to make some statements about the future. When, for an individual player, the systems have more data with which to work, they can (generally speaking) be more confident in their projections about the future. And bad players generally don’t stick around long enough to build up a long track records in the major leagues. So it’s not so much that this is a list of very good pitchers — although it is that, as well — but rather that it is a list of players who have remained employed long enough to build up a body of work sufficiently large that Steamer can, with a high degree of confidence, say something about their future.

Edit: As mersilis and aweb noted below, part of the reason for this feature is that pitchers who are projected for low RA9s have less room with which to work, relatively speaking, than pitchers projected for high ones. They suggest calculating the ratio of the final column to the projected RA9 as a different and perhaps more useful way to look at this concept. The top five low-volatility starting pitchers by this measure — using the same cutoffs — are Julio Teheran, James Shields, Miguel Gonzalez, Jeremy Hellickson, and Ian Kennedy. The top five high-volatility starting pitchers by this measure are Noah Syndergaard, Shohei Ohtani, Luiz Gohara, Garrett Richards, and Mike Minor. Thank you to both for the quick suggestion.

The downside to this approach, I think, is sort of the inverse to the downside to my original measure — really bad pitchers could look relatively stable by the ratio measure, even if there’s a very meaningful difference from a performance perspective between a pitcher putting up a 4.00 RA9 and a 6.00 RA9; the former might be hanging on to a job while the latter would definitely be out of one. Not so with the difference between a 1.00 RA9 and 3.00 RA9 pitcher, so you might be willing to accept the high measure of volatility there. But of course the way I presented things creates a problem in the reverse direction — so YMMV, and you’ll be best served by picking the right measure for your particular purpose.

And then, of course, there’s Luis Severino. Steamer seems very confident in Luis Severino as a pitcher even though he’s recorded only 326.2 major-league innings and enters just his age-24 season. This is likely due to Severino’s underlying skills having remained relatively stable in his three big-league seasons. His case is certainly a unique one, though.

Now, on to the hitters.

“Ah, look,” said the author. “There, in the middle of the chart, is the name Shohei Ohtani. I have heard it told that if you fail to speak his name in a story published On The Internet in the Year of Our Lord 2018 someone from the web publishing company Tronc comes knocking on your door and asks you if you have ever heard of the term Search Engine Optimization.”

But you all know why Ohtani is on this list, and it’s because nobody has any idea what he’ll be like as a hitter, much less as a pitcher or (for that matter) as a human being. He is a cipher, as far as Steamer is concerned, until such time as he steps into a big-league batters’ box for the first time and tries not to look quite so silly staring down a Kershaw slider.

Also on this list are a bunch of players who’ve struggled with injuries, and some young players, too — Gleyber Torres is who I’m thinking of here — who have an awful lot of upside but haven’t really established a baseline level of big-league performance yet, either. Their presence on this list will not shock you, I think.

Nor, probably, will the presence of anyone on this next one, save perhaps one person:

This is a list of good hitters and also Alcides Escobar. The reason Escobar is on the list, as you may be able to guess from our discussion of the low-variance starters, above, is that he has found a way to play in the majors for a long time — in this case, less because of his offensive skills and more because of his shortstop defense (or, at least, his defensive reputation). The other hitters on this list are here for the same reason the Very Good Pitchers were on the other list: they’ve been around long enough to develop a clear baseline expectation for their performance, as far as Steamer is concerned.

I won’t dwell for long on the tables for relief pitchers, because you’re starting to get the idea and also I want to save some time at the end for team data, and what I think we can take away from all of this. Here’s the high-variance reliever chart:

Burch Smith!

Now here’s the low-variance group:

And what of the teams? I’m sorry to say the data there isn’t quite as interesting — there are enough data points here that the low-variance squads look not all that appreciably different from the high-variance squads. But, again via Jared, and looking between teams, there’s this: Steamer projects a standard deviation of about ~0.46 in team RA9 by the end of the year, whereas over the last eight years team-level standard deviation has averaged ~0.49. And for hitters, Steamer projects a team-level standard deviation of about 14 points in wOBA, whereas over the last eight years, the average standard deviation was 12.5 points. The pitcher difference can probably be explained by injuries that we don’t know about not yet having taken place, but the hitter data is more interesting and may suggest that there is more separation in team talent level this year than in years past.

Here’s what I gather from this, and forgive me if you came to this conclusion years ago. Projections remain a useful way to think about future performance — and are particularly instructive on the team level, when it’s possible to mash all the high-variance individual projections together with the low-variance ones. When it comes to all but the most established of players, though, it’s important to remember that the figures presented on this site represent the median point in a range of possibilities. (I’d argue it’s important to remember that even for the established players, but YMMV.)

Put another way, simply knowing that Gleyber Torres is projected for a .305 wOBA in 2018 isn’t enough. It’s also essential to recognize what type of player he is (inexperienced) and what that means for his forecast (that it features a remarkably wide possible range of possibilities). And it’s important to have some sense of where on the continuum that range falls. There are, for example, reasonably young players like Freddy Galvis and Whit Merrifield who have much narrower bands of projected performance. I think that’s interesting.

There would be advantages to knowing the probability distribution — and not simply point estimate — for every player. As my old colleague Jonathan Judge argues, however, there are some technical barriers to that. And, of course, adding greater accuracy would also reduce the simplicity of the projections. For the moment, it’s best simply to remember that variance exists and that it exists in great volume for certain players. Anyway, these were some words about variance in 2018 projections, written by partial request.