If Elias Used WAR
Yesterday, Buster Olney reported that Major League baseball and the Major League Baseball Players Association were nearing agreement on the new CBA. Olney notes that the new collective bargaining agreement will likely address free agent compensation by getting rid of compensation picks in the first round. As I noted last week, there are many fundamental problems with free agent compensation, as we currently know it. My focus last week was on how the system unintentionally provided incentives for rich teams to not just sign one type A free agent, but sign multiple. In addition to this unintended byproduct of the system, I briefly mentioned the archaic stats used to rank type A and B players. This led me to wonder what would happen if MLB instructed the Elias Sports Bureau to use more advanced metrics.
The rankings are based off of the past two seasons’ worth of statistics. With that in mind, I began thinking about what an idealized ranking system would look like. It didn’t take long to figure that Wins Above Replacement would be much better than the current equation used by the Bureau. Under the current system all players are divided into five different groups:
| Groups | Stats (From MLBtraderumors.com) |
| 1B, OF and DH | PA, AVG, OBP, HR, RBI |
| 2B, 3B and SS | PA, AVG, OBP, HR, RBI, Fielding%, TCs |
| C | PA, AVG, OBP, HR, RBI, Fielding%, Assists |
| SP | Total games (total starts + 0.5 * total relief appearances), IP, Wins, W-L Percentage, ERA, Strikeouts |
| RP | Total games (total relief appearances + 2 * total starts), IP (weighted slightly less than other categories), Wins + Saves, IP/H ratio, K/BB, ERA |
Once the players have been divided, they are ranked against their peers. The ranking works as follows: within each statistic, the player in question is ranked against his positional peers. If there are 30 players in a group, then the player the most HRs gets credited with 30 points, the player with the second most HRs would be credited with 29 points and so on until you reach the player with the fewest HRs, who will be credited with 1 point. All points are then scaled so that the max score for each stat is 100 (divide by the number of players in the given group and them multiply by 100). The scores for each stat are averaged to give the player his final score. If a player led his group in all stats, his final score would be 100. Once each player has been given a score, all the groups are aggregated. The top 20% of the aggregated list is deemed a type A player and 21%-40% is deemed a type B player.
Thanks to MLB Trade Rumors I was able to grab the rankings of all players for the 2010-11 seasons and take a look at their respective WARs. The first thing I did was divide the players into their respective groups. After the players were in the right groups I looked at the composition of each group. The average two-year WAR for type A Catchers was 6.2. The average type A player in Group 1 (1B, OF, and DH) had 8.1 Wins Above Replacement over the past two years. Similarly, Group 2 (2B, 3B and SS) had 8.1 Wins Above Replacement over the same time span. This seemed to be an interesting coincidence, but then I looked at Group 4 (SP). The average type A Starting Pitcher had 8.2 Wins Above Replacement over the last two years. It is surprising given their respective makeups, that groups 1, 2 and 4, would have nearly identical WAR. Finally, I looked at type A Relief Pitchers, and found that the averaged a measly 2-year WAR of 2.5. By looking at the average WAR for each group we can appreciate how type A players in certain groups are relatively valued. It seems as though all positions are valued approximately the same under the current system (catchers are slightly over valued) with the exception being relief pitchers. A type A relief pitcher nets the same compensation as any other position, but the value of the type A relief pitcher is on average a third of the value of other type A players (as measured by WAR). This overvaluation of Relief Pitchers is clearly an issue that needs to be addressed in the upcoming CBA.
Finally, I compared the current rankings as calculated by Elias and the idealized rankings using WAR where the top 20% of players according to WAR would be considered type A. The Venn Diagram below shows two sets and their intersection with each other. The red circle is the number of type A players according to the Elias ranking, and the blue circle is the number of type A players according to my WAR ranking. The intersection of the two groups, as seen in the purple shading, is the number of players that belonged to both sets, i.e. the number of type A players unchanged by the new ranking. At Bradley Woodrum’s behest, I’ve slapped in some festive Marios to make these impressive diagrams even more digestible.
If you look at the Venn diagrams above, you’ll see that a high percentage of the Elias type A players would continue to by type A players under the WAR ranking. If you exclude RPs, then 92% of Elias type A players would be WAR type A players. This is a much higher number than I would have expected. That being said, you can see from the numbers that lie only in the blue areas that the Elias ranking misses a lot of good players – has a high type II error. Finally, the Relief Pitcher Venn diagram illustrates the huge incongruence of the WAR rankings and the Elias rankings. Of the 52 type A RPs, only one of them would be type A according to WAR. I think the WAR ranking would be a great improvement, but the Elias rankings was not as awful as I had initially thought.
I suggested this in the comments to your piece last week, but it’s more on-topic here. If we’re going to use some system to rank free agents for compensation purposes [separate debate], why do we even need to use statistics? Why not just use his actual market value?
That is, something like: a guy signs a contract over $30M and he becomes a Type A, between $10M and $30M and he becomes a Type B, and between $5M and $10M and he becomes a Type C.
Or alternatively, come opening day, rank all of the offseason’s free agent signings according to the size of the contract, and make the top 15 guys Type A’s, the next 15 Type B’s, and the next 15 Type C’s.
I arbitrarily picked the numbers, and you can adjust them so that the appropriate number of players fall into each box. Or come up with any number of variations.
But such a system would eliminate the disconnect between a player’s ranking and his actual market value. The result would be that teams couldn’t game the system to collect picks, and individual players wouldn’t have their salary artificially deflated because of their ranking.
Where does the AAV vs total amount trade-off come into play with this system? Do we just average out the contract total and ignore what the player is actually getting paid that year? What about team/player/vesting options?
I like this idea in theory, but there are going to be ways to pay players but game the system (just like with taxes!).
I say total guaranteed contract. You could do total guaranteed value divided by total guaranteed years (i.e., AAV), but I think using AAV would overvalue certain older players who are good but seen as too risky to give long-term contracts to (like Oswalt and Ortiz this year).
As far as options and incentives, it’d be a minor problem if teams tried to use them to game the system, but I think that issue infinitely pales in comparison to the problem of using the Elias system or WAR or any other statistical-based system.
Disagree. Suddenly we’d see guaranteed contracts at league minimum and $15m incentives that kick in at 3 PA or 1 IP.
Gnomez,
The commissioners office approves all contracts. They would not approve something so blatantly designed to get around the rules.
I suppose the system will never be able to perfectly evaluate players (if so, give us that stat Elias!), but the money system is just too easy to game. At least with some measure of past player performance, you are not affecting a player’s value based on… well… his value. The system you proposed will dampen the high end salaries while raising the low end ones. That is NOT a direction you want a league to go (see: NBA).
Really I just don’t think that teams should be compensated for losing free agents, but that’s another story.
“$15m incentives that kick in at 3 PA or 1 IP”
Here’s a way around this problem. Look at the last N years of the player’s performance and count the fraction where the player has reached those incentives as a part of the AAV of the contract.
Let’s say a team signs Erik Bedard to an incentive laden contract which pays a $1M bonus at 75 IP and $1M for every 25 additional IPs up to 200. Let’s arbitrarily pick N = 3. In the last three season, Bedard has pitched 83, 0, and 129 innings. This system would give him a 2/3 chance to reach 75 IP, a 1/3 chance to reach 100 IPs and 125 IPs, and no chance to reach 150 or more IPs. So add $666,667, $333,333, and $333,333 to the guaranteed AAV of the contract.
I am the Yankees. I want to tank another team. I pay some scrub $30 million because I can afford it. They are treated as Type A. I sit them on the bench all season to play someone else that I bought for less.
This is why such a system is kind of dumb.
Wait, what? First of all, if the Yankees did this, they’d be giving the other team a draft pick. How does that qualify as “tanking?” Second of all, even the Yankees don’t have unlimited money. They’re not going to pay a scrub some absurd amount of money just to spite another team.
There are ways to cheat the proposed system. This isn’t one of them.
Total max contract value, including ALL incentives + options would be the best way to go if this system was used.
But if you do this then there would be tons of contracts given out to players just below whatever threshold is established for each level. And those thresholds would then artificially impact the market rate for players, which is what were trying to avoid in the first place.
No matter how you compensate/punish teams for signing FA’s, there will always be drawbacks and ways to manipulate the system
I have been preaching that Elias should use money as the measuring stick to determine compensation. Actually, you wouldn’t need Elias, or much of any formula. Use the average annual value of the contract that the free agent player signs with his new club. After all, that IS his value, and that is the reason that his former club should be compensated for losing the player. They’ve lost a player worth X amount of money, so they should receive a corresponding amount in compensation, though not full value in raw dollar terms.
If a Type A player is defined as one in the top 20%, then take the top 20% of MLB salaries, and any player making above that amount is a Type A free agent. If his club offers arbitration and he signs elsewhere, the club gets a draft pick in Round A- or two picks, however the formula works. Position doesn’t matter. If the market values players at certain positions higher than others, that will be reflected in the rankings.
I think the idea of using a player’s market value to determine compensation is a very good idea, so long as the compensation is not taken from another team – the comp should be in some sort of sandwich round, where no team is directly penalized. Also, I think that all ordinal ranking are inherently flawed. It assumes that all players over a certain threshold are of equal value. When you rank players, you don’t take into account the margin between each player, only that playeri >playeri+1. I think this also needs to be addressed.
Thank you Mario! But our Type A Free Agents are in Another Castle!
Out of curiosity, which 2 C, 4 1B/OF/DH, 2 2B/3B/SS, and 2 SP are Type A per Elias but not per your WAR criteria?
C:
Ramon Hernandez
AJ Pierzynski
1B/DH/OF:
Michael Morse
Michael Cuddyer
Ryan Howard
Delmon Young
2B/3B/SS:
Asdrubal Cabrera
Jeff Keppinger
SP:
Trevor Cahill
Randy Wolf
In other words, a list of the poor bastards who are going to have a hard time finding a suitor willing to give up a draft pick for them.
Or the guys who will not even be offered arbitration because they could never resign with an A around their necks. Who is going to give up two draft picks for Delmon Young?
Jeff Keppinger? Are you sure that’s not a misprint. The Giants would have to be idiots to re-sign him.
Oops!
Note: The ranking is for ALL players, not just free agents – only ranked free agents bring compensation, but all players are included in the calculations.
Jeff Kepp is not a free agent until next year. He’ll likely be a type B next year, which may be meaningless after this new round of collective bargaining.
Yirmiyahu almost read my mind. I was thinking more along the lines of which poor bastards, if free agents, would likely only be signed by a team picking in the first half of the first round, since said club would part with a 2nd round pick rather than a 1st round pick (at least until the free agent draft compensation pick system gets blown up). I guess Cuddyer and Ramon Hernandez fit that description this offseason.
I third the call to rank players according to the value of their contracts.
There is no reason to replace one flawed statistical evaluation with another flawed statistical evaluation. WAR obviously overrates defense and underrates many relief pitchers relative to the market. The fact that according to WAR many teams overpay relief pitchers does not change the fact that your team lost a relief pitcher to a team willing to pay him a lot of money.
Holy #%&@ those Venn Diagrams are awesome!
I think Paul Brunton said it best, “The artist must raise the cup of his vision aloft to the gods in the high hope that they will pour into it the sweet mellow wine of inspiration.”
Thank you for filling my raised cup.
I am loving the use of Mario sprites on this site recently.
Who was the 1 relief pitcher?
Sean Marshall… bet you didn’t see that one coming.
“I think the WAR ranking would be a great improvement, but the Elias rankings was not as awful as I had initially thought.”
This understates the degree that the current system overvalues relievers. None of them should ever be part of a pick compensation system.
The Rays will never get another comp pick again.
I don’t understand why the total ranked players is different between Elias and FanGraphs. For example, why are there 26 more SP’s in the exclusively blue section than in the exclusively red section. Shouldn’t the total number of players ranked be the same?
The only explanation I can think of is that 26 more FA SP’s are ranked in the top 20% of all SP’s, but that seems like a large enough number to question whether or not the qualifications for the two lists are the same.
There “should” be the same number of players in the red circles as there are in the blue circles. If you add up the number of players in red you get 176 and if you add up all the players in blue, you get 169. This discrepancy is due to rounding errors, but keep in mind there are over 900 players in my sample so that both groups have close to 20% of the sample.
So did you just take the top 20% of all players regardless of their positions to find which players should qualify for Type A under WAR? Elias takes the top 20% based on positional grouping I believe. So wouldn’t that be the reason why the blue circles don’t equal the red circles in each Venn Diagram? Your essentially just kicking the relief pitchers out of type A and adding more players at every other position to Type A. Is that correct?
The first round rankings are done by position (as explained in the post), then the positions are aggregated and the top 20% of the entire league is labeled type A. That is not the same as taking the top 20% of each position. And yes, I just took the top 20% of WAR for all the players eligible to create my WAR ranking. If you did it by position where the top 20% of each group by WAR was a type A you would encounter the same problems as Elias – RPs would be way overvalued, Catchers would be overvalued as well, but to a lessor degree.
I see now. Thanks.
Is it possible that good defensive catchers are undervalued by WAR? It seems like there’s more room for hidden value at that position than anywhere else, because their defensive role is so complicated and because of their effect on the pitching staff. But that’s not really a problem with the article, just something I thought about when he said catchers were overvalued.
I completely agree, I think a catcher’s true value is not completely captured by WAR (though most of it probably is). I think looking into catcher values would be an interesting future post. Thanks for the comment.
My biggest problem with WAR is that I have no idea how it’s computed, and I seriously doubt you can have one stat measure everything. It’s kinda like the NFL’s quarterback rating, but at least they show out that’s computed on the NFL website. And once you learn how to compute the quarterback rating, you realize you don’t need it. It basically mirrors touchdown-to-interception rate. What’s more, you can actually compare a team’s winning percentage with a certain player as opposed to without him, and figure out an actual “wins above replacement” rate without the complicated formula.
http://www.fangraphs.com/library/index.php/misc/war/
It’s a long read, but really interesting. This is exactly how WAR is calculated.