How Much Did the Astros Really Benefit from Sign-Stealing?

With the offseason moving at it’s now customarily glacial pace, the story that’s dominated the baseball headlines for the past week has been the Astros sign-stealing scandal. Since the first story was published by Ken Rosenthal and Evan Drellich at The Athletic last Tuesday, we’ve seen bits of evidence confirming the Astros strategy: a crude system of live monitors and a trash can to relay the incoming pitch type to the batter. On Friday, Rob Arthur of Baseball Prospectus provided audio evidence from the TV broadcasts, on which the whacking of the trash can can be identified and analyzed. Rosenthal and Drellich followed up their post with another revelation that the Astros were exploring different ways to electronically steal signs. And then there’s the video evidence pulled from the 2017 Astros World Series documentary showing a computer — sometimes a monitor, sometimes a laptop — in the tunnel leading to the dugout with a trash can nearby.

MLB is still conducting its own investigation, but the evidence that’s been dug up publicly is pretty damning. Sign-stealing has always been part of the game, but the Astros crossed the line when they began using electronic means to relay information in real-time. And this isn’t the first time they’ve been accused of baseball skullduggery of some sort. There’s little doubt that some form of punishment will be handed down by the commissioner’s office once MLB wraps up its investigation. The only question is how harsh will the penalties be; will it be a slap on the wrist, or will the hammer come down hard to deter other teams who are likely exploring other ways to use technology to steal signs.

Gamesmanship and various forms of cheating are as much a part of the fabric of baseball as the unwritten rulebook, all in an effort to gain an upper hand against your opponent. But how much of a benefit does sign-stealing give to a batter? Knowing what type of pitch is incoming gives the batter a distinct advantage — more information is far better than just guessing — but they still need to translate that information into action, whether its a swing or a take. Since we have plenty of data to analyze, can we measure the effect of sign-stealing on the outcome of a pitch, play, or game?

In 2016, the Astros’ home strikeout rate was 24.5%, the third-worst in baseball. The next year — the year they supposedly implemented their sign-stealing system — it plummeted to 16.7%, the best in baseball. While it may look related, it’s probably more coincidental. Their strikeout rate away from home also dropped by four and a half points and they’ve managed to keep their overall strikeout rate below 20% over the last three years. Besides, sign-stealing should have far broader effects than simply reducing a team’s strikeout rate. Knowing what type of pitch is coming should also help contact rates and, perhaps, produce more high-quality contact.

Digging into the Astros plate discipline stats helps us get a little more granular. Here’s a comparison of some key metrics at the team level in Minute Maid Park.

The Astros improved in all three of these plate discipline metrics from 2016 to 2017. They chased pitches out of the zone less often, they made contact on pitches in the zone more frequently, and swung and missed less often. Then again, Houston posted better marks in each of these three metrics on the road than at home in 2017.

The Astros roster had some pretty significant turnover between these two years. Could the difference in talent level explain their huge improvements in plate discipline? Five of the nine batters who accumulated the most plate appearances in 2016 carried over to the next season: George Springer, José Altuve, Carlos Correa, Marwin Gonzalez, and Evan Gattis. If we limit our data query to those five batters, we see similar results.

These five batters improved their plate discipline across the board. And when we compare their plate discipline on the road in 2017, they posted a better chase rate and swinging strike rate away from home. Only their zone contact rate was better at home in 2017 (86.5% at home, 85.7% on the road).

At a broad level, it’s hard to make any conclusive statements about the possible effects of the Astros sign-stealing. Travis Sawchik came to a similar conclusion in an article posted at FiveThirtyEight yesterday. But what these plate discipline metrics fail to capture is the true impact on a per-pitch level. The additional information provided by the thumping trash can definitely helps a batter make a decision about whether or not to swing. Would it result in higher quality contact and better takes on borderline pitches?

You’re probably already familiar with RE24 — the run expectancy based on the 24 base-out states. As part of his work on Baseball Savant, Tom Tango recently introduced RE288; that is, the run expectancy based on the base-out state and the plate count state. That gives us an easy way to calculate run values for any pitch thrown and then aggregate those run values to begin to analyze how a team, say, benefited from specific knowledge about the pitch being thrown. Knowing what is coming on the first pitch of an at-bat isn’t nearly as important as knowing what pitch is coming in a 3-2 count with runners on base.

I pulled pitch data from Baseball Savant for 2016 and 2017 and calculated run values for swings and takes against the three major pitch types: fastball, breaking ball, and offspeed. At a team level, here’s what we see:

In 2017, the Astros massively improved their performance against both fastballs and breaking balls when playing at home, particularly when they swung at those pitch types. They also got better at watching breaking balls pass by. Their performance against offspeed pitches didn’t really budge, but I don’t think that tells us anything. Knowing whether a pitch is a fastball or a breaking ball is far more important because of the way those types of pitches move. If a fastball is signaled, the batter can gear up for heat and make sure they put a good swing on a relatively straight pitch. If a breaking ball is signaled, they can focus on the movement of the pitch to remove the deception it would otherwise create. If an offspeed pitch is signaled, the batter still has to recognize the speed of the pitch, battling their eyes and instincts that tell them a pitch that looks like a fastball was thrown.

I also calculated run values for swings and takes on pitches in and out of the zone. When paired with the plate discipline metrics above, their improvement in one single area comes into focus.

The Astros improved by nearly 75 runs when swinging at pitches in the zone in their home ballpark. We’d expect to see a big swing in run values given the Astros’ improved zone contact rate discussed above. The other three facets above also show improvements too.

What do these run values look like when we compare their 2017 home performance to their away performance?

Well that certainly clouds our data. In almost every respect, the Astros performed better on the road than at home in 2017. The only aspects where they saw better results at home were when taking breaking balls and taking pitches out of the zone. They were around 20 runs better at home than on the road when they spit on a breaking ball or if the pitch was out of the zone. Considering all the data we’ve seen so far, that might be the one area where the Astros truly benefited from their sign-stealing system, and it earned them maybe a couple of extra wins in 2017.

There are so many factors at play, it’s nearly impossible to isolate the effects the Astros’ sign-stealing had on their performance. The fact that they continued to use this scheme throughout the year tells me they found some benefit, even if it’s hard to determine exactly what it was from the data. In the end, the Astros improved across the board in 2017, leading to their first World Series in franchise history. That championship was fueled by new talent, player development, and possibly cheating. It’s likely some combination of the three, and the data is murky enough to leave it at that for now.