Defending Nate Silver
posted by Howard Wasserman
Franita Tolson, guesting at Prawfs, wrote about the recent conservative attacks on Nate Silver: Deadsin offers (in the typical language of Deadspin, so be warned) a largely non-political explanation: Silver is being criticized (or at least questioned) by the political class (both activists and the mainstream media) for being a nerd relying on statistics, numbers, and math, rather than the “gut feelings” and “knowhow” and “real-world” experience that they have brought to the table for all these years. In other words, the political world is experiencing the same dynamic that the sports world (especially baseball) has been going through for about 15 years, since the rise of Moneyball and advanced metrics. Silver, of course, got his start writing for Baseball Prospectus. And as with many in baseball, the current guard in the political world either does not get it or does not want to get it. And as the math gets better, this will only intensify.
Thus, Chris Chilliza of WaPo could move Ohio into the “toss-up” category, despite the showing of fourteen polls for the past two weeks, in part because of the “absolute necessity for Romney to win the state if he wants to be president.” So because Romney really wants/needs it, the state must be a toss-up. This does not sound much different from baseball announcers who insist that average-but-”scrappy” players are better than superstars who produce big statistics because they “want it more” and “will do whatever it takes to win.”
By the way, for those of you who can’t get enough of this poll aggregation stuff, check out the Princeton Election Consortium, run by Dr. Sam Wang, a neuroscientist at Princeton. He uses a different model than Silver (and actually has criticized Silver’s approach), but with similar accuracy.
November 2, 2012 at 11:23 am
Posted in: Uncategorized
Print This Post
Responses (6)
Dave Hoffman - November 2, 2012 at 7:35 pm
I think part of the problem that many have with Silver is that although his method is totally nonfalsifiable, he presents (sometimes) as if he’s doing something scientific.
Gerard Magliocca - November 2, 2012 at 10:59 pm
I like Silver’s work, but the only potential flaw is the pollsters could be guilty of group think. In other words, polls aren’t really independent data points that can be statistically analyzed with precision. But that doesn’t mean it’s true this year.
The other problem, of course, is that you can’t bet the spread in elections, so a probability forecast is no fun.
A.J. Sutter - November 2, 2012 at 11:22 pm
Dave, are you saying that he could be doing something differently that would be falsifiable? Or that predictions such as he is making — about probability distributions relating to a future event — are inherently not falsifiable? (More concretely: Silver doesn’t predict that Obama will win, which would be falsifiable if Romney in fact wins; rather his prediction that Obama has a 75%, or whatever, chance of winning the Electoral College, which would not be falsified by a Romney win.) If the latter, do you reject all Bayesian methods? See also the Santa Fe Institute approach to complexity, whose proponents (e.g., here and here) say that in the frequent cases where outright prediction of certain events may not be possible, a probability distribution based on repeated runs of simulations might be the best we can do. (I think this is in fact what Silver is doing.)
For that matter, don’t econometrics-based explanations, such as those used in ELS, have falsifiability problems as well? Even some practitioners of that art have noted this issue.
Brett Bellmore - November 3, 2012 at 1:47 pm
Replication is kind of the gold standard for science, and I doubt he’s re-run any of these elections multiple times. The basic problem here is that N is small, and every election has been different in multiple possibly significant ways.
The other problem, of course, is that response rates for the polls have been dropping steadily, and are now in the pathetic single digits. That brings the entire practice of polling into question, and certainly makes error bars calculated on sample sizes dubious. We simply don’t know that those who respond are a representative sample of the larger population.
That’s not a problem you can solve by just aggregating polls. They’re all subject to the same potential error source.
prometheefeu - November 4, 2012 at 12:03 am
@Brett Bellmore:
But nevertheless, you could repeat the method and validate the methodology across elections assuming he keeps his method constant. Basically, you should be able to regress the election outcomes on the outcome probabilities.
Brett Bellmore - November 4, 2012 at 8:23 am
That’s true, the issue being that, as I said, every election is different in potentially relevant ways, and there haven’t been a lot of Presidential elections. With that latter being the key point.
Bet his model has more free parameters than there have been modern Presidential elections… Unavoidably so, there are more states than Presidents have been elected.
This is not even to suggest that he’s not sincere, or that his model is actually wrong. Just that I don’t place a lot of confidence in it.
Leave a Reply