Meta-Analysis, the Election, and 538


NPR’s “On the Media” show today featured Brooke Gladstone interviewing Nate Silver, lately infamous as the analyst and writer at the New York Times 538 blog.  Ms. Gladstone asked a question about meta-analysis, the election, and 538.  Paraphrasing, her question was, “Since you use meta-analysis and include a number of different polls, shouldn’t your analysis be more accurate?”

This is a common, incorrect belief.  It’s often called “regression to the mean.”  Statisticians will appeal to the central limit theorem.  Unfortunately, neither of those apply when it comes to election polling.  As a wise teacher once told me, “Errors don’t cancel, they accumulate.”  He was right then and is even more correct today.  Meta-analysis allows the errors in the various polling methodologies to accumulate.  It’s made even worse this year by the non-response rate to polls, hovering near 90 percent.  That means for every ten calls a pollster makes, they get one response.  So much for the “random sample” assumption.

I’ve made my predictions elsewhere.  I look forward to watching Mr. Silver eat a large portion of crow Wednesday.  For those interested, David Burge has written a clear, cogent description of what’s wrong with political polling.  Just skip past the math — the really good stuff starts about halfway down the article.