With an election looming, lots of figures from politicals are getting tossed around all the time. But what do those figures actually mean, if anything?
To understand that, it’s important to look at how polling works.
The sample is the people that the pollers gather opinions from. The goal is to get a sample that’s as representative as possible of the entire population. That means a mix of genders, income levels, education levels, race, age, etc, etc.
Geography matters too. If a sample is skewed urban or rural, that’s likely to influence the results. If it’s skewed towards richer or poorer areas, that can also have an impact.
With the American electoral college system, it’s not a one voter one vote system, so popular vote isn’t what wins elections, but it’s harder for polls to get electoral college numbers right than popular vote.
You might see poll results reported with the notation N= [the number of people in the sample]. Smaller samples are generally less likely to be representative than large samples. However, a large, poor quality sample isn’t accomplishing anything.
Sampling is the methods used to find the sample. The more randomness there is in sampling, the more likely it will be that the sample is representative of the population. Rigorous sampling methods are essentially for reliable results.
Telephone sampling used to be a key way of reaching people, but now some polls are conducted online. That can skew a sample younger for online polls and older for telephone.
If a certain set of people are less willing to answer poll questions, that can mean that whatever their views are won’t be adequately represented in the final results.
What’s the question?
If the question is ambiguous, or if the answer options that are given don’t adequately represent the range of possible opinions, then the results don’t mean much of anything.
Who’s doing the polling?
Just like poor research methodology yields results that are less meaningful, weak polling methodology yields lower quality results. Established polling firms that have been doing this for a long time are likely to do this better than some random internet site conducting a poll.
Identifying errors and areas of potential bias
Bias in polling doesn’t (usually) mean that the people conducting the poll are biased; it means that certain groups of people are over- and under-represented in the sample, and the results are biased because that group’s views are over- or under-represented.
Even with a large, truly random sample, there is some margin for error inherent in that. But getting a random sample is simply not possible, so that adds even more room for error. That extra error wiggle room may be relative smaller or larger depending on how well the pollsters have done their job.
Another area where there can be error is in people deliberately giving the wrong response. And of course, people can change their minds between the polling date and the election.
If pollsters are identifying areas where their sample isn’t representative, they can make statistical adjustments to help balance that out. That’s not about fudging numbers; it’s more like if a sample was 25% female and 75% male, the female responses would be weighted more to be more representative of the actual gender split in the population. And no, I don’t know the nuts and bolts of the statistical operations that are performed.
One of the concerns that arose out of 2016 polls was a failure to make statistical adjustments based on level of education. People who responded to polls were likely to have more education, and people with more education were more likely to vote Democrat. Without making adjustments to account for that factor, people intending to vote Democrat were over-represented in the poll results.
Interpreting the numbers
The numbers generated by polls provide a snapshot in time of the views of a certain set of people. That set of people will never match up with the people that vote on election day. More established polling companies will probably be a little closer than those using inferior methodology, but no one has a crystal ball.
So, they can be interesting source of information, but they’re still nothing more and nothing less than a snapshot of a segment of the population. What matters far, far more is what people have to say on election day. So use your vote!
For further information, Pew Research has a good article on election polling in the U.S.
The Science Corner has info on media & research literacy, fake news, public health, and debunking pseudoscience.