What Do We See When We Watch Baseball?

We’re going to start with a little quiz. Here’s how it works. I’ll show you a short video clip. There’s something weird about the clip. Don’t make it full screen, at least on your first viewing. I just want you to see whether you can spot what exactly that weird thing is. Maybe you’ll catch it the first time you watch. Maybe it’ll take a few more views. Don’t scroll down too far or you’ll see the answer in the paragraph after the video, and that would defeat the point of our little exercise.

Ready? Here we go.

Did you see it? Did you not see it? Am I just vamping for two more paragraphs in order to give you a better chance of watching the video without spoiling the surprise?

Maaaaaybeeeee.

OK, here’s the answer: There’s no baseball in that clip. You can pause it at any point to check. I removed the ball, frame by frame. I took it out of Charlie Morton’s hand during his windup, I erased it from the air on its way to the plate, and I plucked it from the sky as it descended into Yankee Stadium’s right field bleachers. I didn’t manipulate this video because I was planning on writing about it. I was just fooling around in Photoshop. I thought it would be funny. But then I showed the clip to someone, and they didn’t notice anything remarkable about it. So I sent it to another person, then a third, and then a fourth. I edited another clip and sent it along too. This one wasn’t a home run, but a double play.

Only one of those four people, Daniel R. Epstein of Baseball Prospectus, noticed that the ball was missing, and even he wasn’t positive of what he’d seen. “This is going to sound weird,” he texted back as I kept pressing him to watch again and look for the anomaly, “but I can’t see the ball during the pitch.” When I told him that was the answer, he wrote back, “Wait seriously??”

To be certain, video quality played a role here. I was mostly texting the videos to people at 540p, and they were mostly watching on their phones. I’m sure this would’ve been much easier to spot at full resolution on a bigger screen. And later on, a few people did catch the manipulation on the first or second viewing. Still, the result of this impromptu experiment left me staggered. You can watch baseball without the baseball and not only is it possible that you won’t mind, it’s possible that you might not even notice the difference. What are we looking at when we watch baseball, and what are we looking for?

Maybe this shouldn’t be surprising. Eyes are slow, and humans are hard-wired to hunt for patterns and rely on shortcuts to fill in the gaps. It’s an evolutionary trait that helped our ancestors survive in the wild for millennia. These days it helps us enjoy cinematic masterpieces like Marcel the Shell With Shoes On, and I imagine that in a decade or two it will help us avoid marauding bands of water thieves as we scavenge our way across the barren wasteland that once was America. But there’s more to this than a simple optical illusion.

When I was growing up, my family had an enormous, extremely 1980s, wood-paneled console television. It wasn’t a big screen TV or anything; it was just an old, almost cubic behemoth that dominated the family room. At some point, the screen acquired a small black spot that couldn’t be cleaned off. It was located about an inch to the right of the center. It wasn’t a big deal. In fact, it didn’t affect your viewing experience at all — unless you tried to watch hockey. If you tried to watch hockey, you’d inevitably find yourself staring at the spot rather than the puck, and the game would no longer make any sense. All of a sudden, the action would be revolving around an axis that meant nothing to you. This would happen over and over again until you got frustrated and turned off the game.

I remember trying to explain this to my hockey-crazy cousins from Buffalo, who were aghast that I didn’t share their love for the sport. My excuse didn’t wash with them. They said I didn’t need to see the puck in order to watch the game; the players would tell me what was going on and where to look. Maybe if I already knew the game that would have worked for me, but I was a child and there was no way I could learn from watching it on that television. I still have trouble watching hockey, but decades later, I see (and don’t see) what they meant.

Even if you remove the ball from the frame, the architecture of the game remains in place. You can still tell the shape and location of the pitch from the way the catcher sets and adjusts his target. Daniel Epstein didn’t need to see the pitch Aaron Judge hit out to know that it was a four-seamer. You can tell how well the ball was hit by the swing and the reaction of the batter (as well as the reactions of the pitcher, the catcher, umpire, and the fans behind home plate).

Besides, sometimes the ball can lie to you. How many times have you seen a ball leave Bryce Harper’s bat and travel straight upward, looking for all the world like a harmless popup, only to somehow land in the 20th row of the bleachers? Carlos Correa hit this ball last August. It looked for all the world like it was going to be a popup, and not just to the television audience. Second baseman Nick Maton was fooled enough to point up to the sky. But if you ignore the path of the ball and focus on Correa, the way he finishes his backswing, the way follows the ball with his eyes, holds onto the bat, and eases into an ever-so-subtle strut, you’ll see that he thinks it has a chance to get out.

Once the ball is in play, it exercises a gravitational pull on everything around it. In the same way that astronomers don’t need to be able to see a black hole to know where it is, it’s easy to see the influence that the ball has on the fielders, the baserunners, and even the umpires. On a bouncer up the middle with runners on first and second, the catcher ventures a small, triumphant fist pump, the runner on second takes off for third, the pitcher leaps to make a play but then realizes it’s wiser to let the ball through to the second baseman, and the batter busts it out of the box in a futile effort to avoid the double play. The second base umpire scoots into position in front of the base, the second baseman hangs back for a juicy hop, the shortstop jogs toward the bag in order to receive the feed and throw to first in rhythm, and the runner on first sprints for second but has to slide early in order to avoid the throw.

Bob Carpenter is in his 41st year calling major league games and his 19th year with the Nationals. He told me about a game when he was forced to rely on the movements of the fielders to intuit the location of the ball. It was getaway day in Atlanta on September 21, 2022. The combination of a 12:20 p.m. start, an extremely bright day, and the fact that the right field line in Truist Park points nearly due South meant that the sun was shining directly into the eyes of everyone in the press box. “It was extremely hard to see the ball,” said Carpenter. “And I couldn’t rely on my monitor a lot either because it was so bright in the booth that we were putting cardboard shades over the monitor and going MacGyver on the thing with duct tape and all that, trying to shade the monitors so we could watch them. And that was nearly impossible. Probably for the first two or three innings of the game, if there was a ball that was swung on, I was watching the fielders to figure out where the heck the ball was. It was impossible… [Color commentator Kevin Frandsen] was having the same problem, and later we laughed about it, but it wasn’t a whole lot of fun while we were trying to do it. That day, my mind’s eye really had to picture where the ball was going.”

If you think about it, you might be surprised by how little we actually see the ball anyway. In every play, the ball starts in the pitcher’s glove and ends either in another glove or in the stands. When it’s in a glove or a hand, it’s usually not visible at all. Often, the ball moves too fast for either the eye or the camera to keep up with it. When it’s hit high in the air, it can be hard to see at all. When it’s hit or thrown hard, it’s a blur. As in the clip of the double play above, when the ball is hit hard at an infielder, he often secures it in his glove before the broadcast has time to cut to a shot of him. If you’re in the nosebleed sections or you’re watching on your phone, you really might not miss the ball much.

I asked Carpenter whether he thinks about the difference between what he can see up in the press box and what actually gets broadcast. “All the time,” he replied. “I’ll look at the monitor because I want to see what the fans at home are seeing.”

Carpenter also noted that the people bringing you the action might not always have the best view. “Sometimes our monitors in the booth are a lot smaller than the ones people have at home. And we were hearing from people on Twitter asking why didn’t we see this because they saw it… So they got us bigger monitors and to this day we have probably 40- to 48-inch monitors in the booth. But when we go on the road, we might get a monitor that’s like 12 inches diagonal. They’re tiny. And in that case, it’s hard for me to use that, being as lame as it is, to try to experience what the viewer is seeing at home.”

I sent the video of Judge to John DeMarsico, the director of SNY’s Mets broadcasts, who instantly clocked the missing ball. As it turns out, he was uniquely prepared to spot it. DeMarsico explained that the feed for the center field camera is actually routed to Major League Baseball, which superimposes the PitchCast strike zone on it, then routes that augmented feed back to the network (causing a delay that leaves it roughly eight frames behind all the other cameras). Because the system has occasional hiccups, he’s always on the lookout for problems with the center field camera. “So when I first saw it,” he said, “my initial reaction was, ‘Oh, it must be a PitchCast issue, that the ball got lost in some sort of glitch.’”

Once I explained what I’d done, DeMarsico had his own interpretation of why nobody missed the ball on its way to the plate: The PitchCast strike zone relieves us of the burden of paying attention to the flight of the ball. “You’re blocking the view of the flight path by putting a circle up across the screen. And honestly, on balls in play, it even happens to me. I lose the point of contact… It’s the most important thing in the game, and we’re putting a graphic over it. I think it’s become important for people to see the result of the pitch rather than engaging with the pitch, engaging with the game.” Where I saw the fact that people didn’t miss the ball as evidence that we’re using our imagination to fill in the gap, DeMarsico saw it as evidence that we’ve been trained to turn our imagination off entirely and just wait for a little graphic to tell us what happened.

As the foremost advocate of the idea that baseball is and should be cinema, DeMarsico worries about how this emphasis on the results rather than the journey — the proliferation of gambling, homogenization due to advanced analytics, the expanded postseason, PitchCast, and the impending ABS system — affects our ability to appreciate the narrative of the game. He described the three things that used to happen when the batter took a pitch: The catcher would flash signs to the pitcher, the pitcher would throw the pitch to the catcher who’d receive it, and the umpire would make a call. “So you have these three small little dramas that happen every single pitch. And over the course of a three-hour game, those little dramas add up to something. They force the viewers at home not to be told whether it’s a ball or a strike. You have to engage with the game and judge for yourself: Was that a ball? Was that a strike? The umpire said it was a strike. I thought it may have been a ball. Now, we’re being told: No, that was a ball. No, that was a strike, and now the umpire is wrong. And so now we have no engagement with the actual pitch; we’re being told whether it’s a ball or a strike, and over the course of a game, we detach a little bit.”

Once the ball is in play, DeMarsico agrees that seeing the ball does not necessarily equate to understanding what’s actually going on during the play. “We’re all there for the game. And you’re obviously trying to document the game. You don’t want to lose the ball; you want to keep the action in frame. But what really tells the story of the game is not the ball. It’s the human beings playing the game, the fans in the ballpark, all the things surrounding the event.” He told me that he enjoys cutting away from foul balls much later than other directors. He likes to see the person who ends up with the ball because it means so much to them. “My dad caught a foul ball, a Todd Zeile foul ball, in the late ’90s, early 2000s. And you would think that it was the Bill Buckner Game 6 ball. It’s so interesting, such a small moment as a foul ball can become such an important story within the grand scheme of the game. I really try to approach the job that way. That it is about the humans playing the game, less so than the mechanics of the game itself sometimes.”

Thomas Zinzarella, who calls games for the Single-A Bradenton Marauders, a Pirates affiliate, watched the video while he was on the phone with me. He caught onto the missing ball on his second time watching the video. Toward the end of our conversation, I asked him whether he gave much thought to exactly what he looks for when he watches a game, and how that might be different from what an average fan looks for. “That’s definitely something I’ll think about now for sure,” he said.

As he was describing what he looks for when he calls a game, Zinzarella picked out one play in particular. It was a ball that outfielder Shalin Polanco hit just a couple weeks ago. He knew Polanco had gotten hold of it, but he wasn’t sure whether it had enough juice to leave the ballpark. The Marauders were down, 5-2, with one out and the bases loaded in the bottom of the 10th inning. “The wind was blowing in from right field at like 25 mph, but he hit a line drive, and it’s like, ‘Alright, we’re down three runs. Is this one going to get out?’” From the press box, he had a perfect view of both the ball and the right fielder who was tracking it. But sometimes you just have to wait and see what happens.