NPR’s “On the Media” show today featured Brooke Gladstone interviewing Nate Silver, lately infamous as the analyst and writer at the New York Times 538 blog. Ms. Gladstone asked a question about meta-analysis, the election, and 538. Paraphrasing, her question was, “Since you use meta-analysis and include a number of different polls, shouldn’t your analysis be more accurate?”
This is a common, incorrect belief. It’s often called “regression to the mean.” Statisticians will appeal to the central limit theorem. Unfortunately, neither of those apply when it comes to election polling. As a wise teacher once told me, “Errors don’t cancel, they accumulate.” He was right then and is even more correct today. Meta-analysis allows the errors in the various polling methodologies to accumulate. It’s made even worse this year by the non-response rate to polls, hovering near 90 percent. That means for every ten calls a pollster makes, they get one response. So much for the “random sample” assumption.
I’ve made my predictions elsewhere. I look forward to watching Mr. Silver eat a large portion of crow Wednesday. For those interested, David Burge has written a clear, cogent description of what’s wrong with political polling. Just skip past the math — the really good stuff starts about halfway down the article.