The Trouble With Stats (or the people who pretend to use them)

My father also used to enjoy telling me "figures don't lie, but liars figure." He was essentially trying to explain why people often don't trust those who use numbers to explain things that don't naturally appeal to our (highly flawed) instincts. There is a Simpsons quote along similar lines, but for those of you who are fans, you already know it, and for anyone else, it would be a waste of time.

There is another, larger problem, however, that also damages the reputation of statistical analysis and the things it can tell us about our world. It's when people know/do just enough to be dangerous, and without basic rigour produce findings that are unsupportable. When they spread those findings, people who don't have a strong understanding of the analytical process tend to take them at face value, and then end up being let down later. Rather than blaming the particular culprit who practiced the bad stats, they just blame stats in general. These bad actors aren't always guilty of malice, but it doesn't absolve them of crimes against science, and there is a perfect example on ESPN.com today.

Obligatory picture of Nomar, 'cause baseball!

Check out the article here (it's an Insider article, so if you aren't already behind the pay wall, you won't be able to read it). To summarize, Mr. Keating is trumpeting the value of a simple composite stat, runs per hit (R/H from now on), as some sort of gem in terms offensive value, in both real and fantasy terms. He begins, ironically enough, by pointing out that "many sabermetricians [statisticians who study baseball] barely glance at stats like runs and RBIs, which depend heavily on a player's offensive context" (which is true), and then immediately goes on to make a vague declaration about other skills he must have (not true).

Keating proceeds to provide a handful of historical examples of other players with a high R/H, in fun anecdotal fashion. Fully three of the seven paragraphs in the article don't talk about the particular player that it focuses on, Brian Dozier of the Twins, and even the ones that mention him hardly focus on him. The examples provided vary widely across run environments, from the height of the power-infused steroid era, to pre-WWII baseball, with no accounting for what the differences in offensive numbers looked like at those times.

Finally he cherry-picks a few contemporary data points that support his assertion, none of which are particularly relevant to the point that he is making (and in some cases, contradictory).

At no point does he cover anything like the methodology he used to arrive at his conclusions, or even imply that there was any methodology, which might be more disturbing. Here is one thing I know about people who appropriately use statistics: they love to share the gory details.

Now, I will admit, I had a little prior experience with this particular stat, because I had looked into it a few years ago, and ultimately rejected it as useful. However, context always matters, so it was worth another look. One of the first things that Keating does in the article is subtly undermine sabermatricians and complex statistics, which is curious for a guy who's bio at the bottom of the page says that he covers statistical subjects as a senior reporter. Basically, what he says is that R/H is a stat "...that is so simple you can calculate it in your head and it'll tip you off to hidden fantasy values" (I challenge both parts of that statement).

He mentions BABIP later on without any explanation, so he is assuming a certain amount of sophistication on the part of his readers, which is understandable since most fantasy enthusiasts have more exposure to statistical analysis than an average baseball fan. Certainly, anyone interested in "new" composite stats would also be familiar with OPS, and quite possibly wOBA and wRC as well, so the assumption has to be that R/H offers something that these other stats don't.

This is where the whole thing falls apart under scrutiny. Keating says that R/H is a good proxy for identifying other skills like speed, walk rate, and power, but all of those other stats I just mentioned do the same thing, but the difference is, they actually work. Depending on your analytical preference (or which site you get your baseball fix from), you might choose to believe in OPS, wOBA, or wRC (or base runs, etc.), but the reality is, they are all pretty good, and so they correlate very highly to one another (r values of above .95 between all of them). R/H, on the other hand, correlates with none of them, not even close.

Of course, those are all stats meant to capture real-world offensive value in a context neutral way, and maybe R/H only works for fantasy value. So the obvious thing to do would be look at R/H against ESPN fantasy player values. Unfortunately, once again, R/H has almost no correlation to the player rater values (whereas the fantasy rater correlates really well with those other stats).

"Wait," you might (and should) say, "the fantasy player rater is going to be heavily weighted towards counting stats, and thus might not be appropriate when comparing against a weight stat like R/H." Sadly, you, like Mr. Keating, would be wrong again, as Dozier was 7th in the majors in plate appearances (PA) among qualified batters, so he actually has an unfair advantage in this case.

The truth is, of players who had enough plate appearances to qualify for the batting title last year, 4 out of the top 5 by R/H were not in the top 30 fantasy players, and only 4 of the top 15 made the cut. Most of whom were not even guys with an balanced speed/power/discipline profile, like Dozier (which Keating says the stat should identify), but power hitters like Donaldson, Rizzo, and Bautista. If you look at wRC+, every single one of the top-15 players was in the top 40 by fantasy value, and has an r value twice as high as R/H (.71 vs .31).

A common way to look at how much noise might be in a particular stat is to look at year-over-year correlation, and I will say that for players with at least 350 PA in both 2013 and 2014 there was a slight correlation. At just below an r value of .5, however, it wasn't particularly strong, highlighting just how much luck goes into this formula by including runs, and frankly, this number would probably change by looking at more years (I admit to not going deeper here, but I will point out the failing if someone wants to look into it).

Bottom line: runs per hit is not a great stat over all, and it is not a great stat in the context of fantasy baseball value. I won't venture to say that Keating did this analysis and decided to hide the results from his readers, nor will I say that he was too lazy to do any analysis, but I do know that it took me all of thirty seconds to start finding problems with the assertion. As someone who plays fantasy baseball, I have no interest in giving my competition an advantage, but I also can't figure out how this kind of misleading "analysis" does any good for the field.

The issue isn't just that the number doesn't say what Keating claims it does, because it will, by the nature of its component parts and some of the reasons he gives, have some relationship with the skills he is trying to identify. The issues are that it does so very inefficiently due to flaws in the components used, that we already have much better stats for highlighting the same information. The last, biggest issue is that despite being at least somewhat aware of these flaws, Keating is touting this approach nonetheless, under the guise of statistical analysis, without providing any evidence in support.

Dozier is a good player, and has fantasy value, but anyone using R/H to evaluate players is going to be disappointed in their fantasy season. It is neither predictive nor descriptive of a player's skill, and shouldn't be used that way. The problem is that by placing this article behind the pay wall of an authority like ESPN, the false premise is given a credence that it doesn't warrant, and undermines statistical analysis as a whole.

Don't Fight the Data, Marketing People

Why do people find it so hard to change their behavior, despite evidence that they have been operating under assumptions that turned out to be false?  Maybe I have simply been lucky enough to spend much of my life in a world where data flows in a nearly unlimited stream, to be recorded and calculated by computers.  Answers to questions, and more numbers and information than we will ever have time to analyze are never more than a few clicks away at this point.  Yet still, once an idea has become entrenched, it is surprisingly hard to pry it loose.

An example which immediately shows my bias is the difficulty in getting companies to recognize the value, or even the necessity of devoting a portion of their marketing dollars to paid search, or even general digital advertising.  Plenty of studies have been done demonstrating the beneficial effects of having both online and offline media, or how search captures the interest generated by television, etc..  Nielsen itself, the king of measuring TV advertising reach and effectiveness, has produced these studies, even specifically calling for TV dollars to be re-allocated to online media for a more efficient marketing mix.  Yet still, in the face of overwhelming statistical evidence, marketing professionals who are used to traditional media don’t acknowledge what’s right in front of them.

The best parallel that I can draw comes from baseball, which in many ways reminds me of search marketing.  In both cases, compared to either other sports or other media channels, baseball/search is much more easily measured and quantifiable.  Between engine data and onsite tracking, the consumer journey through search can be quantified as a series of discrete actions, just like a game of baseball.  Looking at football or TV commercials, there are just too many variables to accurately attribute the effect of any one moving part.  With baseball however, like search, every play is recorded, and has been for a hundred years.

Where am I going with this?  Bunting.  I have made this speech a million times, but I will make it again for the one person who reads this who hasn’t heard it before.  If you watch enough baseball, especially in the NL, chances are you have seen a fair few sacrifice bunts in your time.  It has been a respected tool in the managers toolbox for a long time, and is celebrated by baseball purists everywhere.  The problem is that it is a bad decision, and it turns out that it always has been.

With so much available data, it turns out that you can really learn a lot about a subject.  By using linear weights based on every single play over a given period of time, you can create a very accurate run expectancy matrix (it turns out that you want a sample size of at least 3 years to get a model that follows the real results closely).  What this matrix does is tell you what the expected number of runs a team will score on average in any given game state (i.e. ‘a man on second base with two outs’).

Brass tacks: With a man on second and no one out (the most common, and conventionally ‘best,’ time for a sacrifice bunt) the run expectancy is around 1.166, which means that a team in that situation will on average score 1.166 runs in the inning.  With a runner on third and one out, which would be the result of a successful sacrifice bunt, the run expectancy is 0.976.  So if the sac bunt is executed perfectly and all goes according to the manager’s plan, he has created a situation in which his team’s run expectancy has dropped by 16.3%.  Keep in mind, that’s about the most defensible situation in which to bunt.  When starting with one out, or a man on first, the decline becomes much sharper.

Now, I am not saying that no one should ever under any circumstances order a bunt.  There are a variety of factors that come into play and they all need to be considered.   My point is simply that most of the time when you see a sacrifice bunt it is the product of a mindset that simply refuses to acknowledge the value of the data.  When no one had done the math it was an interesting argument, and now it just seems quaint and a little backwards, like chewing tobacco or fights where no one gets punched.

This is what I think of when companies believe that they are doing fine because they run some TV commercials and newspaper inserts.  A study found that about 90% of the CPG (consumer packaged goods) target audience watches TV, and about 81% go online.  Yet about 80% of CPG companies’ marketing dollars go to TV ads, compared to about 3% spent on OLA (online advertising).  

Why do people and industries insist on being behind the curve?  Not wanting to fall for a fad is one thing, but this here internet isn’t exactly new, or a flash in the pan.  In 1915, you could make a case for building horse carriages, in case those darn automobiles didn’t catch on.  In 1940, not so much.  What about the guys who swore that the talking pictures would never replace radio dramas?  Or dinosaurs that figured mammals were never going to be a big deal?

Not knowing something because the information wasn’t available to you is one thing, but willfully ignoring the data that is right in front of you is another.  Consumers are doing everything online, so be there when they are. 

And quit bunting so much.