MIT Sloan Analytics: Dean Oliver on WAR
The sixth annual MIT Sloan Sports Analytics Conference was held in Boston last weekend, and it was bigger and better than ever. Over 2,200 people were in attendance and the list of panelists included team owners, general managers, coaches, and more. According to conference organizers, 73 teams were represented among MLB, NBA, NFL, NHL, MLS and EPL. Notably, the Baseball Analytics Panel was moderated by Rob Neyer and consisted of Rocco Baldelli, Scott Boras, Bill James, Jeff Luhnow and Mark Shapiro.
This article won’t be a recap of the conference, but rather part one of a series of conversations I had over the two days. First up is Dean Oliver and in the weeks to come we’ll hear from Baldelli [hitting], John Dewan [defensive metrics], Dan Rosenheck and Adam Jonas [the ramifications of an international draft] and Jon “Boog” Sciambi [advanced stats and broadcasting].
——
During the conference’s Box Score Rebooted panel, Oliver suggested that he had a lot of opinions on Wins Above Replacement. He didn’t address them, so I caught up to him later in the day to hear his thoughts. A respected voice in basketball analytics, Oliver is currently the director of production analytics at ESPN.
Dean Oliver on defining replacement level: “I think we need some sort of consensus discussion on replacement level. Wins Above Replacement is very much defined by how you set replacement level.
“In one way, replacement level is related to how much you pay players, because minimum salary does define something about replacement quality. There are methods for setting replacement level based on who is getting that minimum, including rookie salary caps and restraints along those lines. That is valid.
“But there are so many ways to define what replacement level is, and many of them have validity. Some are very sport specific. For instance, how you set replacement level for quarterbacks is very tricky. You can have a Peyton Manning or a Tom Brady for many years, and maybe they have a good backup and maybe they don’t. Matt Cassel isn’t replacement level, but who would have ever known if he could really play had Brady not gotten hurt? Defining replacement level for a quarterback is very difficult.
“Defining replacement level for NBA players is different. No one player plays an entire game and keeps others completely off the court. Baseball is somewhat similar.
“Another relevant issue is that people often talk about replacement level as though it is one line. I think there are different lines for different teams — what one team’s replacement level is might be well below another team’s replacement level. It depends on what your assets are and what is available to you given your resources. Replacement implies that you can get them fairly easily, and it’s easier for some teams to get those players than it is for others. We need a better understanding of how to define one team’s line versus another team‘s line, versus an average line. You need to define the league overall.
“In basketball, you have players whose Wins Above Replacement are high, but they also use so many possessions that they can bring you down to average as a team because they’re not the type of player who will take you to an elite level. I’m not sure how much that exists in baseball, because the opportunities are more equal. You can’t have Barry Bonds hitting every time, where as in basketball, you can have Carmelo Anthony shooting every time.
“Allen Iverson and Maurice Cheeks can grade out fairly equally in a stat like Wins Above Replacement. Cheeks was a more efficient player and A.I. took a lot of shots, and while they may be the same in Wins Above Replacement, they clearly weren’t the same player. Style-wise, they weren’t even close. That’s an example of how Wins Above Replacement is just one metric, and another example of how baseball and basketball differ.
“In basketball, it’s about possessions. If A.I. doesn’t use them, someone else has to. If the players using them are replacement level, then A.I is better. If it’s good players, Mo Cheeks is better. So the stat isn’t telling the whole story. Wins Above Replacement is viewed as a good way to evaluate players, but it can’t tell you how you optimize the team.
“I have a lot of thoughts on Wins Above Replacement, but I don’t have the answers. I think we need to put together a panel and work toward a consensus, because it’s a meaningful stat.”
——
MIT Sloan tidbits:
* Speaking on the Baseball Analytics panel, Jeff Luhnow said, “The frontier, from my perspective, is turning player evaluation into player valuation. That’s what I care about. When Scott [Boras] and I have a conversation about a player, this player may be 12 runs above average, and another player may be 10 runs above average, but there are so many other factors that go into whether I’m going to be willing to pay more, or pay less, for that player. Here’s the most likely outcome for this player, plus 12 runs, but what does the distribution look like? Is there a 10 percent chance that he’s below replacement level? Is there a 50 percent chance that he’s well above superstar level? What does that distribution look like? Two players with the same average can be very, very different in terms of how I value him, and how Scott might value him.”
* Tom Kelly, a conference attendee, had an interesting idea. In an impromptu discussion with John Dewan and myself, Kelly, a V.P. at Lockheed Martin, suggested that the post-free-agency portion of a player’s salary not count against the payroll tax. This would give teams more incentive to keep players they signed and developed, as they wouldn’t risk being penalized for paying them fair-market value.
* Speaking on the Box Score Rebooted panel, Bill James said that the analytics community should work to “make rational adjustments to [statistics] that are wrong.” An example he gave was the blown save win. On the subject of analytical arguments in general, he uttered the line, “An unloaded gun always loses a gun battle.”
David Laurila grew up in Michigan's Upper Peninsula and now writes about baseball from his home in Cambridge, Mass. He authored the Prospectus Q&A series at Baseball Prospectus from December 2006-May 2011 before being claimed off waivers by FanGraphs. He can be followed on Twitter @DavidLaurilaQA.
“Kelly, a V.P. at Lockheed Martin, suggested that the post-free-agency portion of a player’s salary not count against the payroll tax.”
Doesn’t this sort of run counter the the very aim of the payroll tax? My understanding of this was that it was intended to improve competitive balance by penalizing big-market spenders. Since it’s generally only the big-market teams that have to worry about exceeding the payroll tax, such a change would likely make it easier for them to keep home-grown talent, but it would do little to encourage the likes of Minnesota to do the same.
If such a plan was in place, would it have made it an easier for Minnesota to resign Mauer a few years ago? How about Prince Fielder and the Brewers this year? Neither of these teams is even close to approaching the salary level where they might be forced to pay the tax, and so I doubt that this “incentive” would do anything except disproportionately favour teams with deep pockets who are pressing up against the tax threshold. The obvious question is then, in what way does this improve competitive balance?
“Since it’s generally only the big-market teams that have to worry about exceeding the payroll tax”
I think that is the point Mr. Kelly was trying to make in regards to free agency. If a “hometown team’ and a “big-market team” are vying for the same potential free agent then the “big-market team’ would have to take into consideration the luxury tax hit while the “hometown team” could negotiate freely. I don’t think it would give hometown teams more incentive but it would give them more leverage.
“such a change would likely make it easier for them to keep home-grown talent”
I don’t think the luxury tax was put in place to punish teams for scouting, drafting/obtaining, and developing talent through their organizations. I believe it was put in to keep those “big-market teams” from raiding other teams who went to all the work to scout, draft/obtain, and develop talent. If a team like the Yankees knows it can have a larger payroll without taxation by keeping the homegrown players then the Yanks might be more inclined to keep developing internally rather than cornering the free agent market allowing the “hometown team” to negotiate fair market without having to worry about the Yanks skewering value by over negotiating. There is a reason most agents want the Yanks to “show interest” in their clients.
“If a team like the Yankees knows it can have a larger payroll without taxation by keeping the homegrown players then the Yanks might be more inclined to keep developing internally rather than cornering the free agent market allowing the “hometown team” to negotiate fair market without having to worry about the Yanks skewering value by over negotiating.”
If this was the way that it worked I’d be in agreement with you, but its seems more likely to me that the Yankees would spend just as much on other team’s FAs, and would no longer be forced to pay the luxury tax since their home-grown talent would not count.
Just a quick glance at the Yankees situation:
2012 Luxury Tax Threshold- $178M
2012 Projected Taxable Payroll (Current System) – $210M
2012 Projected Taxable Payroll* (Kelly’s System) – $172M
*Removing Jeter, Rivera, and Cano’s combined $38.5M
So which system makes it more likely the Yankees go out and sign another team’s FA?
Take, for example, the impending Cano contract negotiations. If the Yankees can lock him up and not have his salary count against the luxury tax doesn’t this make it MORE likely they will go out and sign another team’s FA? If instead, his salary does count against the luxury tax, as it would now, it makes it far more likely that the Yankees shy away from raiding another team after they sign Cano.
Regardless, the author clearly states that Kelly feels this approach would provide an incentive for teams to keep their home-grown talent, but with so few teams subject to the luxury tax I fail to see how this is so.
But as a practical matter the luxury tax cap is set based on typical payroll.
If you made Kelly’s change it seems likely that the player’s union would agree to drop the tax cap by 30 million or so and the Yankees would still be well above it.
So which system gives the Yankees MORE incentive to sign their homegrown 2nd baseman?
I think you would also have to couple this with a mechanic that allowed the payroll tax threshold to grow more slowly than it has been.
The result, hopefully would be an increase in payrolls for the lower payroll teams, incentivizing them to keep their homegrown stars. You might have more teams butting up against the cap, but fewer teams going over the cap.
It’s never been clear to me why there *needs* to be a replacement level. Any arbitrary point would serve the same purpose for comparing two players, which is almost invariably what people are actually interested in. If one player has -4 WARP and another player has -8 WARP, player A is +4 compared to player B. But we could just as easily make player have 4 WARP and player B have 0 WARP, and the relative difference is the same.
In my calculation of WARP, I set replacement level to -3.33, simply so the total number of WARP for a team comes closest to fitting actual team wins. This is “convenient”, but it isn’t absolutely necessary.
But we like to multiply WAR to get a $ value for salary purposes, so getting the correct 0 level is important there.
That’s only if you think people with negative WAR should be paid negative money. If we set the zero point to league average, obviously people below 0 would still get paid. They’d just make less than whatever our new baseline salary was.
No, it’s so that people with negative WAR are identified as being overall detrimental to your team.
This is only true if the value of a replacement level player is the same at each position. If that assumption is valid, then you are right – we can just look at Wins Above No Player At All (WANPAA), which uses a baseline of leaving a fielding position empty and assuming an automatic out at every plate appearance.
If, on the other hand, the WANPAA of a replacement level 1st baseman is different from the WANPAA of a replacement level shortstop, then defining replacement level is crucial to the WAR calculation.
No! This is only true if all players have the same playing time! Replacement level is basically the way of setting the value of a plate appearance.
For example, if we set MLB average as the baseline, then someone who is 0 WAA in 300 PA would look the same as someone who is 0 WAA in 600 PA, but the guy who did it in 600 PA is clearly more valuable.
Basically, if you want to compare the value of two players who had different playing time, then you need to use the right replacement level.
Was this in response to a different post? I was certainly not attempting to argue that WAA would be convertible to WAR with a simple linear transformation – I was trying to illustrate that replacement level matters a great deal.
Clearly if you are looking only at the batting side of the equation as in your example then position matters tremendously in setting replacement level. And in any case, using “no player” as a baseline rather than “replacement level” for a WA* calculation would still properly catch the case you are talking about: i.e. average production for 600PA is better than average production for 300PA. But that’s about the only way in which WANPAA would be useful.
Pete Palmer (The Hidden Game of Baseball) rated players on Wins Above average, with the consequence that over a career, a player who had some below average (but above replacement-level) seasons actually lost value relative to a player whose career ended suddenly at a peak. That’s simply wrong — an average ML player is not freely available, and has considerable value. Where you set the level certainly matters when evaluating a career, or in determining the market value of a free agent. “70% of average” seems to be a decent estimate of replacement level for MLB, but it is difficult to prove one level is clearly correct. Tom Tango insists that the correct approach is to rate everything vs. average, and then add a separate adjustment for replacement level, and that makes sense to me. It also makes it easy to change that level if you disagree with where it is set.
“An unloaded gun always loses a gun battle.”
….unless you get pistol-whipped by one of the old-schoolers.
The trouble with “replacement level” in baseball, it seems to me, is different fro Oliver’s point — it’s that, like OPS, WAR adds two numbers that aren’t really commensurate, because the “replacement levels” for hitting and fielding differ. Really good hitters who can’t field at all, say Jason Giambi, have long MLB careers, whereas really good fielders who can’t hit a lick don’t. Edgar Martinez was a very good hitter, but I wouldn’t say he was necessarily as good at hitting as Gary Pettis was at fielding.
The point is not that fielding isn’t a significant component of a player’s value —
it’s that fielding WAR and hitting WAR aren’t directly comparable.
Ozzie Smith and Barry Larkin do not like your evaluation of their careers.
Well that’s partly because MLB has gone to the trouble of creating a position which requires even less fielding ability than 1B for those who can’t. And it’s also because fielding ability is less enduring than hitting ability.
But fielding WAR and batting WAR are directly comparable: that’s one of the key elements of the whole WAR exercise.
Edgar’s best season with the bat was more than 60 runs above average. Gary’s best season with the leather was a little over 20 runs above CF average plus a little for that position. Babe Ruth had 7 seasons over 100 runs above average. Max about 120. Ozzie Smith had 1 season above 40 runs defense above average plus position above average. His next best seasons were around +30.
So, very good, (or the best) offensive seasons are worth a lot more than very good, (or the best) defensive seasons. It makes complete sense that a good hitting, awful fielding DH would have a job, while the reverse player, (say Raphy Belliard), would never be a regular.
This is a very interesting article and I’d love to attend this conference just because. I’m a Uof Mich/ MBA holder andI’m sure I could keep up…I love this kind of thinking, is this open to the public or invite only?
Thanks,
PJ
I’m guessing it’s open to the public, but check the website to be sure: http://www.sloansportsconference.com/