MIT Sloan Analytics: Dean Oliver on WAR

The sixth annual MIT Sloan Sports Analytics Conference was held in Boston last weekend, and it was bigger and better than ever. Over 2,200 people were in attendance and the list of panelists included team owners, general managers, coaches, and more. According to conference organizers, 73 teams were represented among MLB, NBA, NFL, NHL, MLS and EPL. Notably, the Baseball Analytics Panel was moderated by Rob Neyer and consisted of Rocco Baldelli, Scott Boras, Bill James, Jeff Luhnow and Mark Shapiro.

This article won’t be a recap of the conference, but rather part one of a series of conversations I had over the two days. First up is Dean Oliver and in the weeks to come we’ll hear from Baldelli [hitting], John Dewan [defensive metrics], Dan Rosenheck and Adam Jonas [the ramifications of an international draft] and Jon “Boog” Sciambi [advanced stats and broadcasting].

——

During the conference’s Box Score Rebooted panel, Oliver suggested that he had a lot of opinions on Wins Above Replacement. He didn’t address them, so I caught up to him later in the day to hear his thoughts. A respected voice in basketball analytics, Oliver is currently the director of production analytics at ESPN.

Dean Oliver on defining replacement level: “I think we need some sort of consensus discussion on replacement level. Wins Above Replacement is very much defined by how you set replacement level.

“In one way, replacement level is related to how much you pay players, because minimum salary does define something about replacement quality. There are methods for setting replacement level based on who is getting that minimum, including rookie salary caps and restraints along those lines. That is valid.

“But there are so many ways to define what replacement level is, and many of them have validity. Some are very sport specific. For instance, how you set replacement level for quarterbacks is very tricky. You can have a Peyton Manning or a Tom Brady for many years, and maybe they have a good backup and maybe they don’t. Matt Cassel isn’t replacement level, but who would have ever known if he could really play had Brady not gotten hurt? Defining replacement level for a quarterback is very difficult.

“Defining replacement level for NBA players is different. No one player plays an entire game and keeps others completely off the court. Baseball is somewhat similar.

“Another relevant issue is that people often talk about replacement level as though it is one line. I think there are different lines for different teams — what one team’s replacement level is might be well below another team’s replacement level. It depends on what your assets are and what is available to you given your resources. Replacement implies that you can get them fairly easily, and it’s easier for some teams to get those players than it is for others. We need a better understanding of how to define one team’s line versus another team‘s line, versus an average line. You need to define the league overall.

“In basketball, you have players whose Wins Above Replacement are high, but they also use so many possessions that they can bring you down to average as a team because they’re not the type of player who will take you to an elite level. I’m not sure how much that exists in baseball, because the opportunities are more equal. You can’t have Barry Bonds hitting every time, where as in basketball, you can have Carmelo Anthony shooting every time.

“Allen Iverson and Maurice Cheeks can grade out fairly equally in a stat like Wins Above Replacement. Cheeks was a more efficient player and A.I. took a lot of shots, and while they may be the same in Wins Above Replacement, they clearly weren’t the same player. Style-wise, they weren’t even close. That’s an example of how Wins Above Replacement is just one metric, and another example of how baseball and basketball differ.

“In basketball, it’s about possessions. If A.I. doesn’t use them, someone else has to. If the players using them are replacement level, then A.I is better. If it’s good players, Mo Cheeks is better. So the stat isn’t telling the whole story. Wins Above Replacement is viewed as a good way to evaluate players, but it can’t tell you how you optimize the team.

“I have a lot of thoughts on Wins Above Replacement, but I don’t have the answers. I think we need to put together a panel and work toward a consensus, because it’s a meaningful stat.”

——

MIT Sloan tidbits:

* Speaking on the Baseball Analytics panel, Jeff Luhnow said, “The frontier, from my perspective, is turning player evaluation into player valuation. That’s what I care about. When Scott [Boras] and I have a conversation about a player, this player may be 12 runs above average, and another player may be 10 runs above average, but there are so many other factors that go into whether I’m going to be willing to pay more, or pay less, for that player. Here’s the most likely outcome for this player, plus 12 runs, but what does the distribution look like? Is there a 10 percent chance that he’s below replacement level? Is there a 50 percent chance that he’s well above superstar level? What does that distribution look like? Two players with the same average can be very, very different in terms of how I value him, and how Scott might value him.”

* Tom Kelly, a conference attendee, had an interesting idea. In an impromptu discussion with John Dewan and myself, Kelly, a V.P. at Lockheed Martin, suggested that the post-free-agency portion of a player’s salary not count against the payroll tax. This would give teams more incentive to keep players they signed and developed, as they wouldn’t risk being penalized for paying them fair-market value.

* Speaking on the Box Score Rebooted panel, Bill James said that the analytics community should work to “make rational adjustments to [statistics] that are wrong.” An example he gave was the blown save win. On the subject of analytical arguments in general, he uttered the line, “An unloaded gun always loses a gun battle.”





David Laurila grew up in Michigan's Upper Peninsula and now writes about baseball from his home in Cambridge, Mass. He authored the Prospectus Q&A series at Baseball Prospectus from December 2006-May 2011 before being claimed off waivers by FanGraphs. He can be followed on Twitter @DavidLaurilaQA.

21 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
James Lewis
12 years ago

“Kelly, a V.P. at Lockheed Martin, suggested that the post-free-agency portion of a player’s salary not count against the payroll tax.”

Doesn’t this sort of run counter the the very aim of the payroll tax? My understanding of this was that it was intended to improve competitive balance by penalizing big-market spenders. Since it’s generally only the big-market teams that have to worry about exceeding the payroll tax, such a change would likely make it easier for them to keep home-grown talent, but it would do little to encourage the likes of Minnesota to do the same.

If such a plan was in place, would it have made it an easier for Minnesota to resign Mauer a few years ago? How about Prince Fielder and the Brewers this year? Neither of these teams is even close to approaching the salary level where they might be forced to pay the tax, and so I doubt that this “incentive” would do anything except disproportionately favour teams with deep pockets who are pressing up against the tax threshold. The obvious question is then, in what way does this improve competitive balance?

me, Bob
12 years ago
Reply to  James Lewis

“Since it’s generally only the big-market teams that have to worry about exceeding the payroll tax”
I think that is the point Mr. Kelly was trying to make in regards to free agency. If a “hometown team’ and a “big-market team” are vying for the same potential free agent then the “big-market team’ would have to take into consideration the luxury tax hit while the “hometown team” could negotiate freely. I don’t think it would give hometown teams more incentive but it would give them more leverage.
“such a change would likely make it easier for them to keep home-grown talent”
I don’t think the luxury tax was put in place to punish teams for scouting, drafting/obtaining, and developing talent through their organizations. I believe it was put in to keep those “big-market teams” from raiding other teams who went to all the work to scout, draft/obtain, and develop talent. If a team like the Yankees knows it can have a larger payroll without taxation by keeping the homegrown players then the Yanks might be more inclined to keep developing internally rather than cornering the free agent market allowing the “hometown team” to negotiate fair market without having to worry about the Yanks skewering value by over negotiating. There is a reason most agents want the Yanks to “show interest” in their clients.

James Lewis
12 years ago
Reply to  me, Bob

“If a team like the Yankees knows it can have a larger payroll without taxation by keeping the homegrown players then the Yanks might be more inclined to keep developing internally rather than cornering the free agent market allowing the “hometown team” to negotiate fair market without having to worry about the Yanks skewering value by over negotiating.”

If this was the way that it worked I’d be in agreement with you, but its seems more likely to me that the Yankees would spend just as much on other team’s FAs, and would no longer be forced to pay the luxury tax since their home-grown talent would not count.

Just a quick glance at the Yankees situation:

2012 Luxury Tax Threshold- $178M
2012 Projected Taxable Payroll (Current System) – $210M
2012 Projected Taxable Payroll* (Kelly’s System) – $172M
*Removing Jeter, Rivera, and Cano’s combined $38.5M

So which system makes it more likely the Yankees go out and sign another team’s FA?

Take, for example, the impending Cano contract negotiations. If the Yankees can lock him up and not have his salary count against the luxury tax doesn’t this make it MORE likely they will go out and sign another team’s FA? If instead, his salary does count against the luxury tax, as it would now, it makes it far more likely that the Yankees shy away from raiding another team after they sign Cano.

Regardless, the author clearly states that Kelly feels this approach would provide an incentive for teams to keep their home-grown talent, but with so few teams subject to the luxury tax I fail to see how this is so.

Doug Lampert
12 years ago
Reply to  me, Bob

But as a practical matter the luxury tax cap is set based on typical payroll.

If you made Kelly’s change it seems likely that the player’s union would agree to drop the tax cap by 30 million or so and the Yankees would still be well above it.

me, Bob
12 years ago
Reply to  me, Bob

So which system gives the Yankees MORE incentive to sign their homegrown 2nd baseman?

Nivra
12 years ago
Reply to  James Lewis

I think you would also have to couple this with a mechanic that allowed the payroll tax threshold to grow more slowly than it has been.

The result, hopefully would be an increase in payrolls for the lower payroll teams, incentivizing them to keep their homegrown stars. You might have more teams butting up against the cap, but fewer teams going over the cap.