Deep Underground with Raul Groom

Wednesday, October 13, 2004

Right about now is when the entire nation becomes obsessed with polling. The way most people look at polling is that they have a certain fascination with it and use it as a general scorecard for a campaign, but they will tell you they basically think polls are "BS."

In fact, neither of these ideas has much merit. Polling is, when conducted properly (as almost all widely publicized polls are) an extremely accurate and surprisingly precise measurement. The problem is that people fail to understand what polls are a measurement of.

A poll tells you exactly the following - no more, no less:

How will a population of a given composition answer a particular question or set of questions?

Unfortunately this question tends to get shortened to:

How will the American people vote on November 2nd?

By assuming that it's this broad question we're answering, and not the more limited one, we run into problems that can't be reconciled. They key one is the question of voter turnout, that is, the correct composition of the population we're sampling. You can estimate this, but even if you estimate it, and you're correct, you can't necessarily make your numbers reflect that estimate.

Here's why - let's say you're polling a random sample of the population that includes 38% Republicans, 34% Democrats, and 28% "other," that is, independents and third-party folk. Now, you've estimated that turnout is going to be somewhere around the reverse of that, more like 39% Dems, 35% Republicans, and 26% other. So you can just weight the polling results you got to reflect that change, right?

Well, not exactly. If the numbers you have are that far off your estimates, there's only so much adjusting you can do without introducing a substantial amount of error. So you're really better off not adjusting, or adjusting only slightly, and letting people draw their own conclusions about what the poll means. The problem there, of course, is most people (including reporters) don't look at the population composition, they just see the result of the poll and trumptet it like the score of a baseball game.

That's why I just have to shake my head when I see Democrats on the web complaining about how "biased" some poll or another is. There are biased polls out there, don't get me wrong. But WaPo's pollsters are not out there trying to make Kerry look bad. They're sampling a population that happens to be overweighted with Republicans.

So when I see a WaPo poll that shows a 51-46 Bush lead, I don't freak out and start shouting about a pro-Bush bias. I see what the poll actually says, which is that based on current voter attitudes, if the election were held today and 38% of the people who showed up to vote were Republicans, Kerry would lose.

Which is not really news, except perhaps in that it's surprising that a race with such unprecedented Republican turnout (or lack of Dem turnout) would be so close.

So please, if you want to stay sane, stop using polls like a TV, and use them like a book. That is, you can't just look at it - you have to read it.


Post a Comment

<< Home