The Most and Least Confident of Projections

Chris Sale features the smallest gap among pitchers between his 10th- and 90th-percentile projections.
(Photo: Keith Allison)

A few weeks ago, I wrote a piece for this site wherein I noted that the Chicago White Sox rotation was (a) projected to be very bad in 2018 and (b) composed to a great extent of starting pitchers of whom the following could be said: “That guy? Who knows what he’ll do this year.” My editor Carson Cistulli titled the piece “The White Sox’ Rotation Could Be Anything,” and he was right. The White Sox’ rotation could be anything, because it’s full of players whose track records cause most projection systems to raise their digital shoulders, put on their best Robert De Niro face, and shrug magnificently right in your face.

In the comments to that piece, some of you expressed an interest in reading more about variance in player projections. So, here are some words and tables on that subject.

First, courtesy of Jared Cross at Steamer, here are the 10 starting pitchers whose 90th-percentile projection for runs allowed per nine innings (RA9) differs most from their 10th-percentile one. I put a floor of 100 innings on the query to make sure we were getting pitchers in whom someone might be interested in reading. That said, because the figures I’m reporting here represent “true talent” projections (i.e. there is no added uncertainty based on the number of innings they are projected to pitch), the list should not be biased towards pitchers with fewer projected innings. The same general statements are true of all the tables you’ll see in this piece.

High-Variance Starters, 2018
Player Proj. IP RA9 10th Percentile 90th Percentile 10th-90th Percentile
Henderson Alvarez 164.5 5.38 6.19 4.61 1.58
Sandy Alcantara 120.0 5.53 6.34 4.77 1.57
Carson Fulmer 115.4 5.92 6.71 5.17 1.54
Andrew Triggs 126.3 4.88 5.67 4.14 1.53
Brandon Finnegan 115.7 5.21 5.99 4.48 1.51
Steven Wright 112.7 5.10 5.87 4.38 1.49
Homer Bailey 138.0 5.37 6.13 4.64 1.49
Jacob Turner 132.4 5.61 6.38 4.89 1.49
Anthony DeSclafani 151.4 5.02 5.78 4.30 1.48
Zack Wheeler 116.1 4.88 5.64 4.16 1.48
SOURCE: Jared Cross, Steamer
Proj. IP > 100, all figures park-neutral.

That is certainly a list of pitchers, isn’t it? There’s one White Sock on it (Carson Fulmer) and also an awful lot of pitchers who have, at one point or another, had arm injuries that kept them out for some period of time. By my count, five of the players on this list spent some significant portion of 2017 on the disabled list, and another handful have had persistent injury difficulties over the course of their careers. It turns out that injuries affect pitching performance (usually, for the worse). It further turns out that a pitcher who produces two wildly different seasons can cause a projection system to throw up its hands in exasperation.

Let’s move on to the low-variance starters, shall we?

Low-Variance Starters, 2018
Player Proj. IP RA9 10th Percentile 90th Percentile 10th-90th Percentile
Chris Sale 192.4 3.26 3.74 2.81 0.93
Corey Kluber 212.9 3.47 3.96 3.00 0.96
Clayton Kershaw 190.3 3.31 3.82 2.83 0.99
Max Scherzer 207.5 3.82 4.33 3.33 1.00
Chris Archer 196.1 3.87 4.39 3.38 1.01
Justin Verlander 203.8 4.03 4.56 3.52 1.04
Jose Quintana 196.1 4.02 4.55 3.51 1.04
Carlos Carrasco 168.6 3.70 4.23 3.19 1.04
Jacob deGrom 203.3 3.99 4.54 3.47 1.07
Luis Severino 188.3 3.64 4.19 3.12 1.07
SOURCE: Jared Cross, Steamer
Proj. IP > 100, all figures park-neutral.

You may recognize this as basically a list of the 10 best pitchers in baseball from the last half-decade. And it is, but not entirely by design — the system isn’t trying to tell you that better pitchers are less volatile in their performance year over year than worse pitchers (though whether this is true would be an interesting question to explore empirically in the future, and perhaps I shall). It is telling you instead merely that the pitchers about whose performance Steamer has the least uncertainty this year are in fact also, generally speaking, very good pitchers.

This turns out to be a feature of the way projection systems like Steamer work, which is that they take in as much data about performance from the past as they can and then use that data to make some statements about the future. When, for an individual player, the systems have more data with which to work, they can (generally speaking) be more confident in their projections about the future. And bad players generally don’t stick around long enough to build up a long track records in the major leagues. So it’s not so much that this is a list of very good pitchers — although it is that, as well — but rather that it is a list of players who have remained employed long enough to build up a body of work sufficiently large that Steamer can, with a high degree of confidence, say something about their future.

Edit: As mersilis and aweb noted below, part of the reason for this feature is that pitchers who are projected for low RA9s have less room with which to work, relatively speaking, than pitchers projected for high ones. They suggest calculating the ratio of the final column to the projected RA9 as a different and perhaps more useful way to look at this concept. The top five low-volatility starting pitchers by this measure — using the same cutoffs — are Julio Teheran, James Shields, Miguel Gonzalez, Jeremy Hellickson, and Ian Kennedy. The top five high-volatility starting pitchers by this measure are Noah Syndergaard, Shohei Ohtani, Luiz Gohara, Garrett Richards, and Mike Minor. Thank you to both for the quick suggestion.

The downside to this approach, I think, is sort of the inverse to the downside to my original measure — really bad pitchers could look relatively stable by the ratio measure, even if there’s a very meaningful difference from a performance perspective between a pitcher putting up a 4.00 RA9 and a 6.00 RA9; the former might be hanging on to a job while the latter would definitely be out of one. Not so with the difference between a 1.00 RA9 and 3.00 RA9 pitcher, so you might be willing to accept the high measure of volatility there. But of course the way I presented things creates a problem in the reverse direction — so YMMV, and you’ll be best served by picking the right measure for your particular purpose.

And then, of course, there’s Luis Severino. Steamer seems very confident in Luis Severino as a pitcher even though he’s recorded only 326.2 major-league innings and enters just his age-24 season. This is likely due to Severino’s underlying skills having remained relatively stable in his three big-league seasons. His case is certainly a unique one, though.

Now, on to the hitters.

High-Variance Hitters, 2018
Player Proj. PA wOBA 10th Percentile 90th Percentile 90th-10th Percentile
Jonathan Herrera 164 0.266 0.233 0.300 0.067
Gleyber Torres 286 0.305 0.275 0.335 0.060
Steven Duggar 208 0.299 0.270 0.330 0.060
Andrew Toles 242 0.330 0.300 0.360 0.060
Shohei Ohtani 247 0.352 0.322 0.382 0.060
Jeff Mathis 205 0.267 0.238 0.297 0.059
Steve Clevenger 192 0.299 0.270 0.329 0.059
Devin Mesoraco 220 0.315 0.287 0.345 0.058
Tyler Ladendorf 379 0.261 0.233 0.291 0.058
Gregory Bird 485 0.349 0.321 0.378 0.057
SOURCE: Jared Cross, Steamer
Proj. PA > 150, all figures park-neutral.

“Ah, look,” said the author. “There, in the middle of the chart, is the name Shohei Ohtani. I have heard it told that if you fail to speak his name in a story published On The Internet in the Year of Our Lord 2018 someone from the web publishing company Tronc comes knocking on your door and asks you if you have ever heard of the term Search Engine Optimization.”

But you all know why Ohtani is on this list, and it’s because nobody has any idea what he’ll be like as a hitter, much less as a pitcher or (for that matter) as a human being. He is a cipher, as far as Steamer is concerned, until such time as he steps into a big-league batters’ box for the first time and tries not to look quite so silly staring down a Kershaw slider.

Also on this list are a bunch of players who’ve struggled with injuries, and some young players, too — Gleyber Torres is who I’m thinking of here — who have an awful lot of upside but haven’t really established a baseline level of big-league performance yet, either. Their presence on this list will not shock you, I think.

Nor, probably, will the presence of anyone on this next one, save perhaps one person:

Low-Variance Hitters, 2018
Player Proj. PA wOBA 10th Percentile 90th Percentile 90th-10th Percentile
Charlie Blackmon 663 0.340 0.322 0.359 0.037
Brian Dozier 652 0.339 0.321 0.358 0.037
Evan Longoria 611 0.342 0.324 0.361 0.037
Anthony Rizzo 633 0.394 0.376 0.413 0.037
Jose Altuve 654 0.366 0.347 0.384 0.037
Carlos Santana 596 0.362 0.344 0.381 0.037
Jose Abreu 616 0.365 0.347 0.384 0.037
Brett Gardner 574 0.328 0.309 0.347 0.038
Joey Votto 640 0.400 0.381 0.419 0.038
Alcides Escobar 467 0.271 0.252 0.290 0.038
SOURCE: Jared Cross, Steamer
Proj. PA > 150, all figures park-neutral.

This is a list of good hitters and also Alcides Escobar. The reason Escobar is on the list, as you may be able to guess from our discussion of the low-variance starters, above, is that he has found a way to play in the majors for a long time — in this case, less because of his offensive skills and more because of his shortstop defense (or, at least, his defensive reputation). The other hitters on this list are here for the same reason the Very Good Pitchers were on the other list: they’ve been around long enough to develop a clear baseline expectation for their performance, as far as Steamer is concerned.

I won’t dwell for long on the tables for relief pitchers, because you’re starting to get the idea and also I want to save some time at the end for team data, and what I think we can take away from all of this. Here’s the high-variance reliever chart:

High-Variance Relievers, 2018
Player Proj. IP RA9 10th Percentile 90th Percentile 10th-90th Percentile
Gerson Bautista 20 5.85 6.81 4.94 1.87
Burch Smith 40 5.08 6.03 4.21 1.82
Albert Abreu 30 5.54 6.45 4.68 1.77
Zac Reininger 35 5.10 6.00 4.26 1.74
Daniel Winkler 45 4.52 5.42 3.68 1.74
Huston Street 30 5.45 6.34 4.61 1.73
Carlos Ramirez 45 5.00 5.88 4.17 1.71
Tayron Guerrero 35 5.21 6.09 4.38 1.71
Chris Martin 35 4.54 5.42 3.73 1.69
Tyler Bashlor 25 4.71 5.59 3.90 1.69
SOURCE: Jared Cross, Steamer
Proj. IP > 20, all figures park-neutral.

Burch Smith!

Now here’s the low-variance group:

Low-Variance Relievers, 2018
Player Proj. IP RA9 10th Percentile 90th Percentile 10th-90th Percentile
Craig Kimbrel 65 2.70 3.28 2.16 1.12
Francisco Liriano 25 3.95 4.53 3.40 1.13
Collin McHugh 35 3.97 4.56 3.41 1.15
Chad Green 45 3.29 3.89 2.74 1.15
Dellin Betances 55 3.08 3.68 2.52 1.16
Andrew Miller 65 2.92 3.53 2.36 1.17
Brad Peacock 40 3.82 4.42 3.25 1.17
Aroldis Chapman 65 2.96 3.58 2.39 1.19
Matt Andriese 25 4.00 4.63 3.41 1.22
Jesse Chavez 30 4.29 4.92 3.70 1.22
SOURCE: Jared Cross, Steamer
Proj. IP > 20, all figures park-neutral.

And what of the teams? I’m sorry to say the data there isn’t quite as interesting — there are enough data points here that the low-variance squads look not all that appreciably different from the high-variance squads. But, again via Jared, and looking between teams, there’s this: Steamer projects a standard deviation of about ~0.46 in team RA9 by the end of the year, whereas over the last eight years team-level standard deviation has averaged ~0.49. And for hitters, Steamer projects a team-level standard deviation of about 14 points in wOBA, whereas over the last eight years, the average standard deviation was 12.5 points. The pitcher difference can probably be explained by injuries that we don’t know about not yet having taken place, but the hitter data is more interesting and may suggest that there is more separation in team talent level this year than in years past.

Here’s what I gather from this, and forgive me if you came to this conclusion years ago. Projections remain a useful way to think about future performance — and are particularly instructive on the team level, when it’s possible to mash all the high-variance individual projections together with the low-variance ones. When it comes to all but the most established of players, though, it’s important to remember that the figures presented on this site represent the median point in a range of possibilities. (I’d argue it’s important to remember that even for the established players, but YMMV.)

Put another way, simply knowing that Gleyber Torres is projected for a .305 wOBA in 2018 isn’t enough. It’s also essential to recognize what type of player he is (inexperienced) and what that means for his forecast (that it features a remarkably wide possible range of possibilities). And it’s important to have some sense of where on the continuum that range falls. There are, for example, reasonably young players like Freddy Galvis and Whit Merrifield who have much narrower bands of projected performance. I think that’s interesting.

There would be advantages to knowing the probability distribution — and not simply point estimate — for every player. As my old colleague Jonathan Judge argues, however, there are some technical barriers to that. And, of course, adding greater accuracy would also reduce the simplicity of the projections. For the moment, it’s best simply to remember that variance exists and that it exists in great volume for certain players. Anyway, these were some words about variance in 2018 projections, written by partial request.





Rian Watt is a contributor to FanGraphs based in Seattle. His work has appeared at Vice, Baseball Prospectus, The Athletic, FiveThirtyEight, and some other places too. By day, he works with communities around the world to end homelessness.

38 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
aweb
6 years ago

Since the pitchers basically break into “best and worst” groups, this doesn’t really tell us much. If the average RA9 projection is 5.2, of course there is more “raw” variance than someone with a RA9 projection of 3.2. What you want here is a measure of relative variance. Divide your variance measure by the RA9 projection to get the ratio instead.

HappyFunBallmember
6 years ago
Reply to  Rian Watt

My top-of-the-head analysis of those lists is:

1) The high-volatility guys each have a checkered injury history and/or a short track record, and there exists a plausible scenario for greatness.

2) The low-volatility guys are uniformly boring guys with long track records, relative health (at least for a pitcher) and almost no upside breakout scenario

Sometimes the obvious answer is obvious.

Jackofalltrades
6 years ago
Reply to  aweb

Keith Law said Roger Ebert got his cancer from eating pussy.