How Should We Evaluate a Manager?

I’ve got a vote for American League Manager of the Year this season and I’m terrified. My first vote as a member of the Baseball Writer’s Association, and it’s the impossible one.

Maybe impossible is too tough a word. I’m sure I’ll figure something out in time to submit a vote. But evaluating the productivity of a manager just seems so difficult. We’ve seen efforts that use the difference between projected and actual wins, or between “true talent” estimations for the team and their actual outcomes. But those attribute all sorts of random chance to the manager’s machinations.

I’d like to instead identify measurable moments where a manager exerts a direct influence on his team, assign those values or ranks, and see where each current manager sits. So what are those measurable moments?

As far as I, noted idiot, can figure, it’s these variables that define a manager:

• When he uses his best relievers.
• How rigid his approach to the bullpen is.
• Where he puts his best hitters in the lineup.
• How often he bunts with non-pitchers.

What follows is a difficult, probably error-laden, and almost certainly futile effort to score managers on how they’ve done in these four categories. And the worst part is that, when it’s done, not only can I not reveal my vote, but the work won’t really be done — because there’s no doubt that managers are also in charge of personalities and shepherding a clubhouse full of athletes into a cohesive unit. We won’t find a number for that one, and it seems a necessary moment of chaos that threatens to upend our best efforts.

For the first of the above variables, however, what we’d like to do is have a measure of the reliever’s true talent, and then we can match it up with leverage index, which tells us how important the moment is in the context of the game. We’ve got two ways to do this.

The first, called Bullpen Management Above Random by creator Tim Kniker, uses weighted on base average allowed to lefties and righties to judge their true talent. Kniker was kind enough to run this year’s BMAR for us.

Many Cards fans don’t respect Mike Matheny’s bullpen management, so perhaps this passes the sniff test. For the purposes of my later vote, I find it interesting that John Farrell is second in the league in the percent of optimal usage, and that Buck Showalter has a really good bullpen and uses it well. Terry Francona is above average by percent of optimal, but third in this trio.

However, wOBA allowed may not be the best way to judge a player’s true talent. Rob Arthur and Rian Watt developed Weighted Reliever Management score to judge the same thing, and he used Baseball Prospectus’ Deserved Runs Allowed as the measure of talent. He’s a generous person, so he ran a one-year version of wRM for this year for our benefit.

Arthur wanted to point out that these are not the same as the wRM+ scores in the article, because they aren’t based on multiple years and regressed in the same way. Instead, shown here is the raw correlation between DRA and the leverage index of the relievers on the team, weighted by innings pitched. More negative means that the better relievers are being used with the most leverage, which means the manager is doing well.

By this measure, Farrell does well; Matheny, not so much.

Finally, there might be a third way, a better way to evaluate reliever usage — namely, by using projections instead of true-talent estimators. Because Arthur used full-season DRA and leverage numbers in his analysis, he was more answering the question “Looking back, did the manager make the right decisions?” It might make sense, however, to ask “Based on the info he had then, did the manager make the right decisions?” We have the benefit of looking at the results here, but since we also don’t have the full season or multiple years of regression, there’s more uncertainty with these rankings than the ones he published.

I’m a big fan of flexibility in approach. Flexibility, to me, is the sign of a supple mind, one that allows for considerations beyond one’s own worldview. So it’s fun that Ben Lindbergh and Dan Brooks developed a stat that measures the rigidity of reliever usage. He looked at the number of outs the team had recorded before a given reliever came entered, and the variation in that number, to see if some managers were less flexible in their usage. Lindbergh was kind enough to run this stat for 2016, leading up to September, which is fine for me because September baseball is played on the moon by aliens.

Here, a lower number means a more rigid approach to the bullpen. Brad Ausmus scores poorly even as he tried to find the right combination of relievers to lead up to closer Francisco Rodriguez. The Indians have been better at moving the pieces around once they got the right pieces — that was the reason Lindbergh brought out RRR this year, since Cody Allen has been moved around more than most closers.

For the next piece, Jonah Pemstein helped assess the ability of a manager to put the right players in the right spots in the lineup. What he did was look at the projected daily wOBA for each player in each lineup slot for each team, which was awesomely provided by Matt Hunter and SaberSim. Pemstein then ranked those wOBAs on the team, and then correlated their actual lineup slot to their ideal lineup slot, using Tom Tango’s The Book as a guide for the best way to put a club’s assorted players into the lineup.

What we have here, then, is a correlation between actual and ideal lineup slots. The closer a team is to one, the closer they were to putting out an ideal lineup every night.

To see what these distributions look like, Pemstein also created some visuals. Let’s look at the extremes and find the distribution of lineups for Pittsburgh and Kansas City. Clint Hurdle has his team’s lineups bunched up over by the right side, the ideal side, while Kansas City has a whole lot of Alcides Escobar at the top of the lineup.

For the purposes of my vote, Showalter gets some more love, while Farrell and Gibbons also do well. Francona, though, takes a little hit. Looks like SaberSim liked Tyler Naquin better than other projection systems, and he batted eighth for most of the year.

We’ve saved the easiest for last. Rather than going through each bunt situation in a granular way, I thought we could say something wide-ranging that should still ring true — namely, that it’s rarely a good idea to bunt with a non-pitcher. They should be good enough with the bat that it lowers win expectancy to trade their out for a base.

The American League plays the National League enough that a simple leaderboard tweak should give us what we want. Which teams bunted the least often with their non-pitchers? Answer: the Red Sox, with the Orioles in third, and the Tigers and Blue Jays in fifth and sixth.

In sum, we have some arguments for the managers who will be on the short list for American League Manager of the Year. John Farrell, John Gibbons, and Buck Showalter do well across the board, while Jeff Banister and Terry Francona have their strengths and weaknesses. Picking between them will be difficult. Almost impossible, even.

Thanks to Ben Lindbergh, Dan Brooks, Tim Kniker, Jeff Zimmerman, Jonah Pemstein, Rob Arthur, Rian Watt, and Matt Hunter for their help in putting this together.