The Most Important Bangs of the Astros’ Scheme

Just when you thought every angle of the Astros sign-stealing scandal had been covered, a new wrinkle emerges. Previous attempts to discern the true impact of the banging scheme were always limited by the lack of granular data. Without going through the effort of scanning all of the available audio and video, we had very little idea how often the Astros banged on their trash can. Were they using their scheme in every plate appearance? Only in high-leverage situations? Without answers to these questions, we were using incomplete data that included a ton of noise.

Enter Tony Adams. He scanned through 58 Astros home games with available video and audio and logged every single instance of a bang using a custom application he built. He ended up logging over 8,200 pitches and over 1,100 bangs. Last Wednesday, he made his data public on his site.

With Adams’ data in hand, we have answers to some of the questions that thwarted earlier attempts at analysis. The data still isn’t comprehensive. The 23 home games with missing video are still shrouded in mist, and we can’t assume the banging scheme was in effect during plate appearances where the batter only saw fastballs. We also don’t have insight into other possible methods of communicating stolen signs, like whistling, buzzers, or the like. Still, the new data provides us with a thorough look at those 58 games, helping us draw some more definitive conclusions.

Rob Arthur of Baseball Prospectus had this to say after his initial look at Adams’ data:

“By and large, the Astros tended to get the signals right, but it was hardly perfect. They were most accurate when they whacked the can: When they did so, a non-fastball was on the way 93 percent of the time and they were wrong seven percent of the time. … Based on Adams’ data, the Astros used the trash can frequently and they were relatively accurate, moreso when banging than when silent.”

A 93% accuracy rate seems high, but when the banging misidentified pitch, it had a significant negative effect on the overall value of the scheme — at least according to Arthur’s methods:

“When the fix was in, the Astros received a substantial advantage. But when the players in the tunnel thought they had cracked the code and it turned out they hadn’t, it harmed the batters at the plate more than knowing the incoming pitch helped them. … The net effect of the banging comes astonishingly close to being zero. Nothing. Statistically, for all the work and effort that went into the cheating scheme, the grand result of it, at least as measured in this way, turned out to be no runs at all.”

Using Adams’ data, I re-ran my analysis using pitch-by-pitch run expectancy (RE288) and found that the Astros likely benefited from their scheme but to a lesser degree than I had previously estimated. I’ve scaled the run values below to standardize the values per 100 pitches.

The most surprising result is in that first column. On fastballs — the pitches that were supposedly misidentified in the banging scheme — the Astros actually performed better when a bang occurred than without. (I am including cutters in my categorization of a fastball.) It’s possible the Astros considered cutters breaking balls, and when I remove them from the fastball data, the banging impact is greatly reduced. The Astros also saw a benefit when swinging at breaking balls and offspeed pitches. That tracks with my previous analysis, though the cumulative effect was much less than I thought.

But beyond trying to discern the team-wide impact of the banging scheme, Adams’ data presents us with a few new avenues of analysis. Now that we have individual bangs identified and associated with individual pitches, we can look at specific instances where the banging scheme had a significant impact on the outcome of a game. Using Adams’ raw data, I mapped the leverage index for every at-bat where a bang occurred. That helps us answer the question of whether or not the banging scheme was used more often in high-leverage situations. The answer is resoundingly no.

The average leverage index of the at-bats where a bang occurred was .99, almost exactly average. The median leverage index for those at-bats was .85. That tells us that the average was skewed a bit by the high-leverage situations, and that there were more low leverage situations where the banging scheme was on.

That’s exactly what the data shows us. Just 9% of the at-bats where the trash can was used occurred in high-leverage situations. That’s around what we’d expect based on the normal distribution of high-leverage situations throughout the season. And in those high-leverage situations, the banging scheme didn’t seem to provide much of a benefit. Our run values show that in those 53 high-leverage at-bats, the Astros cumulatively lost around two runs. It’s not much of an effect overall, but it’s still surprising to see a negative impact.

In medium- and low-leverage situations, the Astros banging scheme definitely had a positive impact. Intuitively, this makes sense. The banging scheme is optimally used against starting pitchers, as cracking their signs will have a more significant effect over many innings. And if you were wondering if the Astros continued to use their banging scheme in situations where the game was already decided, 14% of their at-bats where the trash can was used came in extremely low-leverage situations where the leverage index was less than 0.2.

So what was the most important bang of the Astros’ 2017 season? It’s fitting that it came with Carlos Beltrán at the plate. (Watch the clips below with headphones or with the volume turned up to experience the full banging effect. The videos begin at the appropriate timestamp.)

Trailing by three runs in the bottom of the ninth inning against the Blue Jays, the Astros loaded the bases against Roberto Osuna. After seeing three straight cutters, two bangs rang out from the Astros dugout prior to the fourth pitch. Osuna threw another cutter. Beltrán swung and grounded into a fielder’s choice, dropping the Astros’ win expectancy by 10%. The next two at-bats would win the Astros the game — with the help of the banging scheme in both of them.

The highest-leverage at-bat where the banging scheme was used but the at-bat ended on a pitch with no bangs occurred on July 17.

With the game against the Mariners tied at seven in the bottom of the ninth, the Astros loaded the bases against Yovani Gallardo. Alex Bregman stepped to the plate with two outs. The first two pitches whizzed by without a bang from the dugout. On the third pitch, a slider, a single bang rang out with a few more sounding immediately after the pitch. On the next pitch, a fastball, Bregman grounded into a fielder’s choice (with some nice defense by Jean Segura to hold the game level).

Here’s a table of the five most important at-bats where the banging scheme was used, by leverage index:

The eventual game-winning hit in that game against the Blue Jays shows up as the third-most-important at-bat, though no bangs occurred on the pitch that Juan Centeno hit, a fastball. Another at-bat from that game against the Mariners shows up as well. Norichika Aoki was up earlier in the inning; he grounded into a fielder’s choice like Bregman did a few at-bats later.

We can also use this data to determine how each player on the Astros fared in different leverage situations with the help of the banging scheme. Below is a table of the cumulative run values for each player who saw more than 200 pitches in 2017, and their performance in high, medium, and low leverage situations in at-bats with at least one bang in them (the columns are sortable):

Only a few players — Alex Bregman and Carlos Correa most notably — were able to take the information provided by the banging scheme and put it to good use in high leverage situations. Yuli Gurriel benefited from the stolen signs no matter what the situation was, but particularly in medium leverage situations. George Springer was unable to take advantage of the trash can system when the game was on the line or when the game was all but decided, but he really thrived in medium leverage situations. Marwin Gonzalez, the player who heard of the greatest number of bangs during the season, simply couldn’t put that information to good use, no matter what the situation was.

While there isn’t much more that can be gleaned from this leverage data, the fact that we know the Astros frequently used their banging scheme even when the game had already been decided answers a longstanding question hovering over this scandal. If the Astros used their sign-stealing sparingly or only in particular situations, that might color our perception of how they cheated. We now know they used it often and in all situations, from low leverage to high.