Be Cautious With Lineup Analysis Tool
When it comes to sabermetric studies, no single item sees more energy expended with less gain than the analysis of batting orders. The Book basically opened and shut the door on the issue: the best three hitters should bat first, second, and fourth, but even the most egregious of lineup errors won’t cost a team more than a win. It’s also more important to split lefties to avoid LOOGYs than it is to get that perfectly chained lineup.
That doesn’t mean that lineup construction isn’t fun, and I’ve certainly spent my share of time on largely fruitless but enjoyable studies on the batting order. There’s a tool available over at Baseball Musings that seems to make things easier for everybody, spitting out optimal lineups and even run totals for any lineup you can think of. Unfortunately, the numbers it spits out cannot be trusted and are no longer a reflection of reality.
The first tip to how obsolete the tool is comes from the two models the user is allowed to use. One is the 1998-2002 model, which will spit out horrendously large numbers in terms of runs/game due to using data from the offense-inflated days of the steroid era. This results in ridiculous numbers like the Blue Jays scoring nearly 5 runs per game when only two teams managed that number last season. The other option uses numbers from 1959-2004, which smooths things out much more but is still difficult to transplant into the context of 2011 baseball.
The other problem (a much smaller problem, given we’re already dealing with minutia) is the only inputs are on-base percentage and slugging percentage. In The Book, the inputs used are linear weights by each lineup slot. It’s important to realize that different batting slots tend to see different situations – leadoff sees bases empty often, third sees nobody on and two outs quite often, fourth sees most runners, etc. – and therefore a single from a hitter in one slot isn’t necessarily worth the same as a hitter in another. Compared to the issues with the run environment, this is minor, but it can still spit out some odd orders.
Use the lineup analysis tool if you must, but be aware that the run totals it spits out have nothing to do with the context of the current game. You’re much better off simply using The Book’s final conclusion. Take the best three hitters at #1, #2, and #4 (with power leaning towards #4 and OBP leaning toward #1). The next two go in #3 and #5. Then the worst four go in #6, #7, #8, and #9. This clean and relatively simple lineup analysis will rarely lead you astray.
Jack Moore's work can be seen at VICE Sports and anywhere else you're willing to pay him to write. Buy his e-book.
Wow. That was pretty harsh.
What you don’t note is that the tool does indeed work. If you take the composite batting order for a team at the end of the season and plug in the numbers for each slot, you get a very good estimate of the runs scored by the team. The reason the Blue Jays are projected to score so many runs is that the Marcels project them to be a good team.
Harsh, but oh so true.
The inputs are two of the 4 main factors when creating a lineup: Type/quality of hits produced by players, OBP, speed, handedness. And for the first factor, it muddies it somewhat by using slugging, when the actual distribution of the hits matter.
It’s an extremely limited “tool”.
Harsh? He didn’t even get around to mentioning that the entire construction of the tool is completely misguided. It applies a fixed multiplier to the offensive production of each position. But those multipliers (even accepting the questionable method used to generate them) are based on traditional lineup constructions. So, for example, SLG from the cleanup slot is very valuable because the three prior hitters get on base a lot (typically). The tool, however, allows you to construct untraditional lineups, but continues to assign the exact same value to each “slot” even while the lineup being tested places much better or worse hitters in preceding and subsequent slots. The value of a cleanup hitter’s SLG is obviously not the same if you put your best three OBP hitters in the 5, 6 and 7 slots.
The lineup tool as constructed cannot work. It does not work. And it never has worked. I’ve never understood why David keeps it active.
“, but even the most egregious of lineup errors won’t cost a team more than a win.”
According to what? Last time I checked the lineup tool, the potential best and worst lineups for any given team were almost 5 wins apart.
I think that this is the point of the article. If you accept the findings of The Book on lineups, the lineup tool is making too much of a difference between lineups (less than 1 win vs almost 5 wins as you put it)
That’s why the author mentions caution about using the tool. The author goes on to explore possible reasons why it might be differing between The Book.
And I strongly disagree.
You’d get more than 1W just from switching the number of plate appearances of your best hitter and your #9 guy. (from best/worst lineup)
RC should read The Book. It’s difficult to refute when you see the logic that goes into it.
When he says “most egregious” he means assuming you’re using common sense (namely the best hitters at 1, 2, and 4). Obviously batting your pitcher cleanup and your masher 8th will severely hurt your offensive output…
‘Common sense’ unfortunately has your best hitter hitting 3rd, and a speed guy 1st. Last year’s MLB averages per lineup spot:
1st: .264/.329/.382
2nd: .270/.334/.398
3rd: .281/.360/.460
4th: .269/.346/.461
5th: .265/.336/.439
6th: .259/.324/.419
7th: .249/.312/.386
8th: .243/.313/.371
9th: .208/.260/.295
1st spot had 880 steals, almost a third of the total.
Ha, my Cubs hit their best hitter 8th most of last year, and when they didn’t it’s because he was sitting on the bench in favor of a guy that hit like a pitcher. They may have cost themselves a win just by limiting Soto’s PA (he amassed 3.5 WAR in 387 PA, and 500 PA wouldn’t be out of line for a starting catcher hitting up in the lineup — and his backup was actually below replacement-level). That doesn’t even account for shifting him into higher-leverage situations and inferior hitters out of them.
If thats what he means, he clearly doesn’t understand the word hes using.
You’re right, the most pessimal lineup order versus the most optimal one could equate to a difference of around 4 or 5 wins.
The author doesn’t word this in a way that we know exactly what his meaning is, but the phrase “lineup error” could be refering to only one egregious change, such as batting one of your best hitters 9th instead of near the top of the order, while leaving the other hitters in more appropriate spots.
One single change like that could lose one win. Arrange the whole order in the worst possible way, and it could cost the team a few wins.
agreed, kinda harsh, but i definately understand. A lot of my Red Sox fans hated what it spit out on ESPN (ie Drew leading off, Crawford 5th)-for what its worth David, I still enjoy the tool.
The “most egregious” I took to mean as “most egregious, but sane”.
Putting your pitcher in the cleanup slot, for example, by itself, costs 0.1 runs per game, or 16 runs per season, which is 1.6 wins. Putting in an all-insane team will likely put you at minus 40 runs or so, or minus 4 wins. This part is nothing really new in The Book, and echoes Palmer’s findings in The Hidden Game from 30 years ago.
***
I don’t think the issue is really the 1998-2002 or 1954-present or whatnot. I presume that simply is to set the balance for coefficients.
The issue is that you have the same nine guys playing for 162 games. There’s no bench players, no nothing in there. So, you can’t make the comparison against what happens with a typical league-leading lineup.
***
That said, I’m not a fan of the lineup tool that uses as inputs only OBP and SLG. It may work for most of the cases, but that’s what happens with any regression that has limited parameters: it can never work in the more unique/extreme cases. It’s safer to say that anything the Lineup Tool spits out as within 0.05 runs per game as being a candidate for an optimal order. And anything more than 0.10 runs per game from the leader as being unlikely to be optimal.
So, what I would like to see in the presentation is something like: of all the lineup orders that is within 0.05 runs per game of the leader, Bautista is leadoff in 15%, cleanup in 30% , etc. Something like that.
That should be possible. I’ll see what I can do when I clear my pre-season plate.
I’m excited.
I’m excited, too. Despite the criticism above, I feel like the tool is still a very good way to compare teams to one another. No matter how good a tool is, we must understand how to use it, and with the caveats that we’re not inputting bench players, linear weights, etc. it’s still an informative tool.
In short, I don’t have a problem with the tool, I have a problem with how many people are using it. People are getting a bit too excited about it in the same way they often overemphasize projections for individual players. If the tool can be improved along the lines Tango suggests, great, but we’ll still need to use it smartly.
The lineup tool is pretty cool for what it is. Of course it is imperfect, but does anything better exist?
Yes: There’s http://lineupsimulator.com/ for one. Also http://www.sportsquant.com/baseballapp.htm. The downside is that you can’t test all possible lineups at once. (Which is a pretty big downside.)
Very cool. Thanks!
Loved the “lineupsimulator” link; the other required plug-ins so I ignored it. I entered the Reds 2010 stats, first with Arroyo as pitcher and then a composite of the batting stats of seven starting pitchers and Miguel Cairo as pinch-hitter. Batting the pitcher 8th had little effect, and for some lineups it was better to bat the pitcher 9th. All of the best lineups had MVP Joey Votto batting 1st, 2nd or 3rd, never 4th. The best lineup I found was Votto, Hannigan, Bruce, Rolen, Stubbs, Phillips, Gomes, Pitcher, Cabrera — 719 runs. Almost as good was Votto batting second or third and the pitcher batting 9th (718 runs.) Batting Votto second, third or fourth and batting the pitcher 8th dropped the run total to 715.
At the first one, is there anyway to get the data into the squares or cells quickly? Or do you have to type in each stat into each cell?
Also, is it a Markov model or does it actually play games and report the results?
I have the Star Simulator which was free but does not seem to be on the internet anymore. It actually plays the games and it is pretty easy to get the data into the model.
I think people are taking the tool too seriously. It can be useful if used right.
The problem with Baseball Musing’s implementation is that the optimal lineup typically has the pitcher batting 8th, which only reflects reality in St. Louis. If there was an additional function that allowed you to check the RS for any particular lineup, it would be greatly improved.
What I’ve done is go to the source materials underpinning his implementation, which provides the actual co-efficients for each slot in the lineup, then calculate the expected scoring with that lineup (which you can manipulate to match what you expect your team’s manager to do).
The complaint that the high offense years from which the original regression was done is understandable, but not entirely valid, I believe. Scoring is scoring. As the era of offense moves to defense, OBP and SLG has and will go down likewise. It is not like players will continue hitting like they used to but suddenly score less runs. Their OBP and SLG has gone down, and scoring with it.
It would be better if the regression was redone with a better set, more recent set of data, but really the offensive era ended maybe 1-2 years ago, so the new data isn’t really here yet. While the previous dataset has more hitters on-base and such, and thus scoring may be inflated to some degree, as Pinto noted, if you check the composite and aggregate data, it is still a good estimate for the team, and thus still useable; I’ve checked that too, it has been pretty good in recent years.
And that’s what I use it for, comparing last year’s lineup with this year’s lineup, to get a feel for how improved it is (or worse it is). You can then plug in that with pitching projections to pythagorean to get a feel for the range of how many wins your team might get for the particular season.
It is not perfect, nor exact, but really, a lot of people seem to act like it is, and write their articles as if they are, but all this is still an art to a large degree. I understand and applaud the push to be more of a science, but at the same time, maybe people should get off the high horse and maybe they will have better discourse.
And I count myself among these, as it is hard to separate the fact that there are a lot of formulas and seemingly precise science involved, and yet we can argue until we are blue in the face, but nobody can project 100% yet, and that is the threshold for science, repeatability.
Still, making it a science would end all this fun back and forth, and who would want that?
The absolute accuracy of any model when being used to compare options is not important, relative accuracy is. When you compare various lineups, the absolute run total is not important if the model accurately predicts which lineup produces more runs.
Agree, I use it to compare teams to one another and to compare various lineups to one another on the same team. I know the same lineup is not going to bat every game, and I don’t care. I’m not trying to project the exact run total of the team.
Has Prince Fielder leading off for the Brewers…
“One is the 1998-2002 model….this results in ridiculous numbers like the Blue Jays scoring nearly 5 runs per game when only two teams managed that number last season.”
The 2010 run environment was not exactly predictable, was it?. I bet even linear weights coefficients were off in 2010. In fact in The Book (published 2006), the coefficients used were derived from data from 1999-2002. So which range of years are you proposing should be used to accurately predict the 2011 run environment?.
“The other problem (a much smaller problem, given we’re already dealing with minutia) is the only inputs are on-base percentage and slugging percentage.”
OBP and SLG are very convenient. More importantly, they are real numbers (not estimates or adjusted). wOBA which is based on linear weights can be expressed as roughly (OBPx2 + SLG)/3.
The coefficients used in LWTS, like those used for OBP and SLG in the Tool, vary with run environment. As such, they are slightly different for league, team and park, but for convenience, these differences are ignored, and a league average team and park is assumed, with no difference between leagues.
“Take the best three hitters at #1, #2, and #4 (with power leaning towards #4 and OBP leaning toward #1). The next two go in #3 and #5. Then the worst four go in #6, #7, #8, and #9”
Amen. I guess that means the science is settled and we can all go home.
“This clean and relatively simple lineup analysis will rarely lead you astray.”
But who calculates the number?.
“In The Book, the inputs used are linear weights by each lineup slot….and therefore a single from a hitter in one slot isn’t necessarily worth the same as a hitter in another.”
Thats great. Have you tested it against the Blue Jays or other AL teams for 2010? Predicted vs Actual? How much more accurate was this than Daves lineup tool?.
Also, Daves lineup tool has different coefficieints for OBP and SLG for each lineup slot. So while the inputs are only OBP and SLG, the coefficients for OBP and SLG used by the Tool are different for each position in the order
The tool isn’t an actual simulator or Markov model–it’s just a linear regression, to my understanding, that uses fixed weights for OBP and SLG in each lineup slot, regardless of what the other slots are doing. So the more different your lineup is from the lineups used in the data from which the regression was derived, the less accurate the equation’s forecast will be. You’re much better off using a real simulator or Markov, which will properly account for the interdependence of each lineup slot on all the others.
This. Which causes me great confusion why this tool exists when there are plenty of off-the-shelf markov models available that can be run. I guess this is more efficient, but I still quite don’t get why you’d use a screwdriver to hammer in a nail, I guess.
For the same reason you use a nail instead of a 100 dollar screw made to aerospace tolerances.
Where can I get one of these off-the-shelf markov models?
I didn’t see it in a few casual breakdowns of the information in “The Book,” which I just ordered on Amazon and hope it answers the question I’m preparing to ask in-depth. Here, if anybody knows what The Book says in quick form, please do tell:
The Book states, roughly, that the #1 and #4 hitters should be the two best players on the team, with the leadoff spot skewing towards OBP and the cleanup spot skewing towards SLG. However, the question I have is, What if one guy leads the team in both? Is it better to bat that guy leadoff or in the cleanup spot?
Hey guys,
So, playing around this week I’ve put together a little tool that simulates seasons for a specified lineup. Would like to get your take on the approach. The tool is actually a simulator, to take into account the things Guy mentioned in this thread. Because of this, I obviously can’t calculate every permutation of a given lineup as it would take too long, but you can tweak the lineup and run the simulations again to see the difference.
This is very much in the development stage but I thought I’d post it here initially to get some feedback on my approach.
Basically, all players are loaded with their 2010 stats and teams. This is just a default but all stats can be adjusted up or down to get to what you want. So, if you want to simulate Pujols as hitting 60 homeruns or 20 homeruns, you can just type it into his homerun column. Once you’ve entered the stats as you want them, drag the players name into the box on the right.
Once you have nine players, you can re-arrange at will or you can run the simulation. The slider at the bottom controls how many seasons you want to simulate. Obviously, the more seasons you do the longer it will take.
If you’re interested, please have a look and tell me what you think about the approach. You can comment here or email me by clicking on my name.
Here is the link
Some quick thoughts:
1) While it works in IE, it functions a bit better in Firefox right now (the dragging is a bit screwed in ie)
2) Works best with minimum resolution of 1280×1024
Thanks in advance.
Ryan
Gah, link got filtered out…
Baseball Lineup Tool
oh my god, so sorry. this site needs an edit.
Finally, the link:
Baseball Lineup Tool
This better work or I give up.
Thanks for your useful article. One other problem is that mesothelioma cancer is generally due to the inhalation of materials from mesothelioma, which is a very toxic material. It’s commonly found among individuals in the engineering industry with long exposure to asbestos. It could be caused by moving into asbestos insulated buildings for a long time of time, Genes plays a huge role, and some persons are more vulnerable on the risk when compared with others.
Furthermore, i believe that mesothelioma cancer is a scarce form of most cancers that is usually found in those people previously familiar with asbestos. Cancerous cellular material form while in the mesothelium, which is a safety lining which covers a lot of the body’s areas. These cells ordinarily form inside lining from the lungs, belly, or the sac which actually encircles one’s heart. Thanks for expressing your ideas.