Science, Pseudoscience, & Media Literacy

What Political Polls Mean – And What They Don’t

the word vote displayed across a blue and red background

With an election looming, lots of figures from politicals are getting tossed around all the time. But what do those figures actually mean, if anything?

To understand that, it’s important to look at how polling works.

Sample

The sample is the people that the pollers gather opinions from. The goal is to get a sample that’s as representative as possible of the entire population. That means a mix of genders, income levels, education levels, race, age, etc, etc.

Geography matters too. If a sample is skewed urban or rural, that’s likely to influence the results. If it’s skewed towards richer or poorer areas, that can also have an impact.

With the American electoral college system, it’s not a one voter one vote system, so popular vote isn’t what wins elections, but it’s harder for polls to get electoral college numbers right than popular vote.

You might see poll results reported with the notation N= [the number of people in the sample]. Smaller samples are generally less likely to be representative than large samples. However, a large, poor quality sample isn’t accomplishing anything.

Sampling method

Sampling is the methods used to find the sample. The more randomness there is in sampling, the more likely it will be that the sample is representative of the population. Rigorous sampling methods are essentially for reliable results.

Telephone sampling used to be a key way of reaching people, but now some polls are conducted online. That can skew a sample younger for online polls and older for telephone.

If a certain set of people are less willing to answer poll questions, that can mean that whatever their views are won’t be adequately represented in the final results.

What’s the question?

If the question is ambiguous, or if the answer options that are given don’t adequately represent the range of possible opinions, then the results don’t mean much of anything.

Who’s doing the polling?

Just like poor research methodology yields results that are less meaningful, weak polling methodology yields lower quality results. Established polling firms that have been doing this for a long time are likely to do this better than some random internet site conducting a poll.

Identifying errors and areas of potential bias

Bias in polling doesn’t (usually) mean that the people conducting the poll are biased; it means that certain groups of people are over- and under-represented in the sample, and the results are biased because that group’s views are over- or under-represented.

Even with a large, truly random sample, there is some margin for error inherent in that. But getting a random sample is simply not possible, so that adds even more room for error. That extra error wiggle room may be relative smaller or larger depending on how well the pollsters have done their job.

Another area where there can be error is in people deliberately giving the wrong response. And of course, people can change their minds between the polling date and the election.

If pollsters are identifying areas where their sample isn’t representative, they can make statistical adjustments to help balance that out. That’s not about fudging numbers; it’s more like if a sample was 25% female and 75% male, the female responses would be weighted more to be more representative of the actual gender split in the population. And no, I don’t know the nuts and bolts of the statistical operations that are performed.

One of the concerns that arose out of 2016 polls was a failure to make statistical adjustments based on level of education. People who responded to polls were likely to have more education, and people with more education were more likely to vote Democrat. Without making adjustments to account for that factor, people intending to vote Democrat were over-represented in the poll results.

Interpreting the numbers

The numbers generated by polls provide a snapshot in time of the views of a certain set of people. That set of people will never match up with the people that vote on election day. More established polling companies will probably be a little closer than those using inferior methodology, but no one has a crystal ball.

So, they can be interesting source of information, but they’re still nothing more and nothing less than a snapshot of a segment of the population. What matters far, far more is what people have to say on election day. So use your vote!

For further information, Pew Research has a good article on election polling in the U.S.

The Science Corner: Pseudoscience, Public Health, and Media Literacy

The Science Corner has info on media & research literacy, fake news, public health, and debunking pseudoscience.

34 thoughts on “What Political Polls Mean – And What They Don’t”

  1. This is very informative. Thank you for sharing! I think a lot of people easily get confused. I’ve been trying to share where people can go to review their state ballots before they go vote. Many people didn’t know that was an option. I love how more people have been reaching out to educate others. Thank you for being one of them!!

  2. I’m very cautious about putting too much stock in the polls. They can be misleading, especially when you take into account the intricacies of the electoral college. I so much wish we would get rid of that archaic system and just go by the popular vote, which is actually democratic and would make everyone’s vote count equally!

      1. I suspect it’s because the system currently works to the benefit of the Republican Party, and the Democrats are not able to win the two-thirds majority they would need in Congress to change the constitution. Even if they did that, I’m not sure if enough states would ratify it either.

  3. Yeah!! It’s hard to tell from polls, and it seems to involve some predictive guesswork!! And the results could definitely be swayed if the interviewer wanted a certain outcome! The topic always makes me remember when I was in fifth grade and my older cousin Michele went to Kentucky Kingdom (now Six Flags) with me, and we took a soft drink taste test. I preferred cup #1, and the older teen girl said, “You chose Coke! YAY!” And then Michele preferred cup #2, and the teen girl deflated and muttered, “You chose Pepsi. Boo.” I don’t know why, but that still cracks me up.

    I have some concerns about mail-in ballots, and my dad and I plan to drive wherever we have to drive on election day so we can vote in person. I just hope that there’s no confusion or concern about late mailed ballots, or who knows what else, and it all seems stressful. Here’s hoping it all goes smoothly!

  4. I live in Germany and i can’t stand anymore!! This election campaign was too much to handle. I can’t even imagine how might the american people might feel right now!

      1. We will probably also always vote by mail. In 2016, we ran into T-1 in the polling place. Can you believe that? That is how close we must live to each other (we don’t know where T-1’s house is, and we don’t want to know). We haven’t run into each other during covid. We have kids at the same school but Younger Child might graduate in spring and then we probably won’t run into T-1, even living so close, since we don’t leave the house.

  5. There was a fantastic article on fivethirtyeight last week looking at how wrong polls have historically been and why we don’t notice it. Usually, it’s because they simply just pad the winner’s already expected lead. For instance, Reagan wiped the floor with Mondale in 1984, but that wasn’t fully expected. In fact, the polls were off by 14-17%. That’s huge! But because it was in the expected victor’s favor, it’s largely forgotten. Going back 60 years, the average amount polls are off when all is said and done is 2%. In 2016, it was 3.3% That’s not a big margin of error, but in the swing states, it’s all that was needed to make the difference. When people freak out about polls, usually a much closer look at the numbers, and history, will explain what is happening. One of the things this article stated — and if you like your politics with math, not partisanship, fivethirtyeight is a great site — is that if the election were held today and Trump one, it would be the second biggest polling error in 60 years. That wasn’t the case in 2016. It was much closer than people want to remember.

Leave a Reply