How Quickly Should You Change Your Mind About Elite Pitching Prospects?

As you might have heard, the Red Sox traded Rafael Devers to the Giants earlier this week. In my breakdown of the deal, I ranked the players headed to Boston in the order of my interest in them: James Tibbs III, Kyle Harrison, Jose Bello, and lastly Jordan Hicks, though that’s contract-related, as I think he’s probably the best current player of the four. The next day, someone in my chat asked me why I preferred Tibbs to Harrison – was I particularly high on Tibbs, or particularly low on Harrison? After all, Harrison was a consensus top 50 prospect only a year ago, while Tibbs took his first Double-A at-bats this week.

My initial answer was that I saw Harrison several times last year, and he didn’t really do it for me. Combine that with his uninspiring results and the fact that other prospects had squeezed him out of the Giants rotation, and I preferred Tibbs. Since neither guy is clearly ready to dominate the major leagues right now, give me the higher-variance unknown quantity.

When I stopped to think about it later, though, I decided that my answer wasn’t good enough. Right now, I’m knee-deep in spreadsheets, linear regressions, non-linear regressions, projections, scouting reports, basically every type of baseball data out there as I do some initial work on our annual Trade Value Series, which will run next month. I have tons of prospect data stored up. I even looked into how prospect grades translate into major league players earlier this year. Rather than try to re-evaluate Harrison based more or less on vibes and ERA, I decided to apply a bit of analytical rigor now that I wasn’t writing for a deadline.

First, I tried to find a cohort of prospects like Harrison. Initially, I wanted to use his Future Value grade – he merited a 55 FV, a mark that our prospect team gives out only sparingly to pitchers, on the 2024 Top 100. But a 55 FV hasn’t always meant the same thing at FanGraphs, as the team frequently mentions. On the 2025 Top 100, there were 32 players with a 55 FV or higher; on the 2017 Top 100, there were 72. I didn’t feel good about using those as a pool of comparables, because, well, they’re not all that comparable. Instead, I settled for a numeric cutoff. I limited my list to pitchers who were in the top half of the Top 100. I considered being even stricter (those in the top 25), but this study is going to be working with a small sample to begin with, so shrinking it even more felt counterproductive.

Next, I needed to define what I was searching for. Here’s how I had it in my head: How a pitcher performs in their rookie year is a huge data point, and how they’ve performed after two years is even more useful to know. If you re-ranked prospects after seeing all of them play against major league competition for a year, you’d do much better when assessing them. To give an extremely lazy example, most outlets (us included) had Dylan Crews and Paul Skenes, LSU teammates who went 1-2 in the draft in 2023, very close to each other coming into 2024. That’s not how it has worked out so far.

I grouped pitchers into three buckets based on rookie year performance. Given the small-ish samples, I thought it would be a nice way to get an overview without trying to imply there’s mathematical rigor that I simply can’t provide here. Next, I set a requirement that the pitcher exhausted rookie eligibility either the year a prospect list came out or the next year. That got rid of strange cases like Forrest Whitley, who ranked highly on several lists in the late 2010s before a long litany of injuries. He debuted in 2024, and I don’t think his performance tells us a lot about Kyle Harrison.

Next, I set a rookie season cutoff of 2022 or earlier. The reason for this is simple: I was trying to look at how valuable pitching prospects are during their team control years, which means that I needed some values for rest-of-season 2025, 2026, and so on. Those have to be projections, of course. A pitcher who exceeded rookie eligibility in 2022 will have three full years of major league stats (unless they got hurt or demoted, itself a bad sign) and another half-year in 2025, more or less. Their projections are going to be heavily based on their actual major league production.

Take MacKenzie Gore, for example. He put up 0.8 WAR in 2022, 1.2 WAR in 2023, and 3.2 WAR in 2024. He has another 2.7 WAR in 2025 so far, and he’s projected for 1.9 more this year. That’s a real track record, one that makes me feel a bit better about using projections for his final two years of team control (2026 and 2027). Compare him to Grayson Rodriguez, who debuted in 2023. He posted 1.8 WAR in 2023, 2.0 WAR in 2024, and hasn’t pitched yet in 2025. I didn’t feel great about calculating “team control WAR” for players like Rodriguez, where more than half of the sum is a projection.

With those constraints – former top 50 prospect, rookie season within a year of the time they were a top 50 prospect, and rookie season before 2023 – I then made one final pass to remove duplicates. Gore and Rodriguez are both good examples – they were top prospects for a while, but I don’t want to double-count anyone. That gave me a sample of 27 pitchers, which isn’t a lot. But the whole point of top pitching prospects is that there aren’t a lot of them, and I’m not trying to prove anything beyond a reasonable doubt here, just come up with a rule of thumb for how to treat early-career performance. Onward!

I decided on WAR as the method of evaluation. I did this not because I think it’s the perfect way to evaluate pitchers, but because it’s less bad than all the other ways I was considering. Volume matters – but only using innings pitched to evaluate success rates would make no sense. Run prevention matters – but it’s both noisy and incomplete. I don’t want to reward someone for throwing 50 unsustainably hot innings and then getting hurt. WAR ties those together. I settled on FIP-based WAR because FIP is already more regressed towards the mean than ERA, and I’m working with tiny samples to begin with, but I’m not strongly tied to that; again, this is just a framework.

With that in mind, I came up with some cutoffs in my head. What’s a good rookie year? Two or more WAR. There were only three pitchers who fit that bill, believe it or not: Logan Gilbert, Walker Buehler, and Michael Soroka. It’s possible that I missed someone because of improperly marking down when they exhausted rookie eligibility, but the point is, most rookie pitchers don’t get a full complement of starts, and this isn’t a list of all rookie pitchers, only top prospects, so you don’t get the surprise Mitchell Parkers and Tobias Myerses of the world (both exceeded 2.0 WAR in their rookie year).

What’s an acceptable rookie year? Between 1.0 and 2.0 WAR. Another eight pitchers fell into that camp. That left 16 top prospect debuts who finished the year with less than 1.0 WAR by our accounting. This group is all over the place. You’ve got players who got hurt, slow starters, and the odd reliever conversion, though since this group is all elite prospects, there are very few of those. You can think of Lucas Giolito (an aggregate -0.3 WAR in a brief 2016 cameo and then his 2017 rookie season) as an example of this group, or Gore. In other words, there are stars — Corbin Burnes started very slow, he’s in there — but there are also plenty of guys who scuffled and then kept scuffling.

I also did a second pass based on the first two years of performance. This feels closer to what I want: less focus on whether the player had his service time manipulated or got squeezed out of playing time in his rookie year, more time for talent to shine through. This time, I set the cutoffs at 3.5 WAR for “good” and 2.0 WAR for “acceptable.” The same three guys – Gilbert, Buehler, and Soroka – were the only ones in the good category, and somehow only five top pitching prospects were “acceptable.” The rest were in the muddled bottom – oft-injured, banished to the ‘pen, riding the minor league shuttle, or simply ineffective. For this cut, the “subsequent WAR” in question is all WAR accrued after those first two years, so it should be slightly lower on average, all else equal.

Here’s how those cohorts shook out over their years of team control (including projections for remaining team control where relevant):

I have to say, this data is noisy. Like, noisier than most baseball data, which is already quite noisy. Gore alone moves the bottom tier by more than 1.0 WAR. Soroka’s injury issues make the top bucket look meaningfully worse. But just because it’s noisy doesn’t mean it’s not true; part of the reason this data is so tough to work with is because baseball teams are in search of outliers. Prospects mostly don’t pan out. Do you see how many dudes we evaluate every year? Most of them don’t turn into star pitchers.

I came up with one additional method because I was quite frustrated with all the noise. Instead of looking at total team control WAR, I looked at peak single-season WAR. That mostly excludes projections, which I like. It also does a better job of controlling for injury – we all know that pitching prospects get injured sometimes, and that early-career playing time isn’t always a guarantee. I set a cutoff for pitchers who posted a peak WAR of 3.5 or higher. To me, that’s someone who at least had a star-ish season. That tracks, at least kind of, with observed results: There were 23 “star” pitchers last season by that definition, 18 in 2023, 29 in 2022, and 30 in 2021.

Here’s a slightly different table, then, looking at “star season percentage,” or the percent of starters who turned in at least one excellent season, based on early-career results:

What do these results tell us? One thing they tell us is that dealing with arbitrary endpoints sucks. I initially set the “star” cutoff at 3.0 WAR instead of 3.5 before deciding that was a bit too broad, and by weird random chance, there were zero players in that “bad first season” bucket with WAR between 3.0 and 3.4, so their odds of producing a star didn’t change. Meanwhile, the “acceptable first season” bucket surged, thanks to multiple players in the 3.0-3.4 WAR range.

More importantly, they say this to me: After one season of a top prospect’s major league career, there’s still plenty of noise. The more I messed around with cutoffs and eligibility criteria and the like, the more results I got. I could manipulate the endpoints and cutoffs, all of which seem reasonable on their face, and change the results meaningfully. One thing is clear: Pitchers with true standout first seasons are a cut above the rest. After that, it’s all kind of a muddle.

Two seasons starts to give you a better picture, though. If a pitcher hasn’t gotten things into gear after two years in the majors, they’re probably not going to. Maybe they’re oft-injured. Maybe they’re just not that good. Maybe they are good, but their playing time gets squeezed because of those two seasons of “meh” at the start of their career. I’m not here to tell you why, just that that effect looks fairly real to me.

Hoping for a larger sample, I re-ran the data with relaxed restrictions. I added players whose rookie season was two years after they were on in the top 50; that added exactly one player. That wasn’t enough, so I also added the 2023 debuts; that added an additional five players. The results aren’t particularly different, though. Here’s the rest-of-career WAR table for the larger group:

And the star season rate table for the larger group, as well:

I don’t think adding the extra data was worth it. The 2023 debuts just don’t have much major league data, particularly since we’re using 2023 to rank them; those guys have only had one full season since then, which means the vast majority of their “team control WAR” is made up of projections, and they haven’t had a ton of chances to spike a star season. I’m including it just to show you how difficult it is to bulk up the sample; “incredibly highly rated overall prospects who are starting pitchers” just isn’t a big cohort these days.

I also considered trying to account for pitchers whose first few seasons were interrupted by injury. The problem with that is that pitchers get hurt a lot. I’d lose more than half the sample if I excluded guys who have had multiple meaningful IL stints. In the end, I just left them in; we’re talking about pitchers, so you have to account for injury. That’s just how it is.

Where does that leave me vis-à-vis Kyle Harrison? He’s squarely in the lowest tier of performance so far, of course, but he’s only 182 innings into his career. The aggregate data are so mixed after one year that I’m willing to give guys some more time to click without too much consternation. It’s hard to be a superstar if you don’t break out early – a different way to present this data would be that players who exceed 1.0 WAR in their first year have much better careers than players who don’t, but again, that’s just an arbitrary cutoff. Early breakouts are good, but the rest of the group feels fairly undifferentiated even after a year.

After two years, things start to tilt. If you’re looking for a transcendent talent, well, they’ve probably transcended, at least a little bit, after two full years of major league play. More and more things start to stack up against the types of pitchers who debut, kick around for two seasons, and haven’t produced much at the end of that. Maybe they’re so inconsistent that they drift on and off the big league roster. Maybe they’re dealing with injury issues that keep them from handling a full workload. Maybe they’ve ended up in the bullpen thanks to their team’s roster composition. Maybe they just weren’t as good as we thought.

Whatever the reason, those things matter. If you’re in the bullpen after two years, you’re probably not going to turn into an elite starter, and if you do, it’ll probably still take you years to get ramped up to a full starter’s workload. If your team can’t find a spot for you in the rotation after two years, your leash will surely be short in the third year, too. Heck, if your team trusts you, but your results just aren’t there, that’s also a problem. In other words, by two years in, you should care more about major league results than pedigree. I don’t feel confident saying it with any more certainty than that – you’ve seen my reservations with sample size throughout the article – but if you’re looking for a broad rule of thumb, that’s a pretty good one.

For Harrison, that means that I’ll be watching the rest of his season very closely. His brief 2023 debut? Not great. His longer 2024 opportunity? Not much better. His 2025? Squeezed out of the rotation to the bullpen. The writing was on the wall, but now he’s in a new room, with a fresh coat of paint. There’s still time to turn it around, no doubt. Harrison is currently in Triple-A, but he’ll be back in the majors this year. A change of scenery is just what the doctor ordered. But daylight is fading fast – at least, probabilistically speaking. So, 2,500 or so words later, yes, I’d still rank Tibbs ahead of Harrison in terms of difference makers. Maybe I’ll be wrong. Maybe Harrison is the next Corbin Burnes or Lucas Giolito. But I’m an odds guy, as you almost have to be when you think about prospects, and I don’t like his odds.