Science, pseudoscience, & media literacy

Can You Believe Statistics?

Mental Health @ Home Science Corner - Can you believe statistics? - image of different types of charts and graphs

There are all kinds of statistics that get thrown around, some of which appear rather dubious.  As a result, there seems to be a lot of suspicion about statistics in general.  But is that warranted?

Let’s start with what statistics are.  According to Wikipedia, statistics is “the discipline that concerns the collection, organization, displaying, analysis, interpretation and presentation of data.”  Descriptive statistics are concerned with describing relationships within sets of data, while inferential statistics examine phenomena that are subject to random variation.

Performing statistical operations is not inherently subjective, and steps are deliberately taken to reduce the risk of bias.  However, the people performing the statistics can make errors or poor choices.  Also, once statistical results are released out into the world, they’re often interpreted without taking the whole picture into consideration.

Polling

Let’s say I wanted to do a political poll.  I find some people to ask, collect my data, crunch my numbers, and find that Peanut the guinea pig is the favoured candidate to rule the world for 85% of respondents.

Go Peanut!  What more could anyone possibly need to know?  Well, a fair bit, actually, and that 85% figure doesn’t mean much without it.  You’ll want to know many people were polled, which may be written as N=100 if there were 100 people polled.  You’ll want to know how I found those people, and whether they represent a broad cross-section of the population of interest.  You’ll also want to know what question was asked and what the possible answers were.  What if the only options were Peanut or some dude who’s already dead?

You’ll also need some other details to accompany that 85% figure, which will look something like this line from Ipsos: “the results of the poll are considered accurate to within +/- 3.5 percentage points, 19 times out of 20.”  Huh?

Well, applying that to the Peanut poll,  that means you can say with 95% confidence (aka 19 times out of 20; determining this involves statistical calculations) that 85% of people, with a margin of error of 3.5% on either side to account for random chance in sampling, are pro-Peanut for leader of the world.

Go Peanut!

Clinical trials

Statistics are often used in research studies to interpret the data.  In clinical trials, typically a 99% confidence level would nornally be used rather than the 95% level often used for public opinion polls.  A “confidence interval” would be determined for each treatment arm based on the margin of error and a 99% confidence level.

To use random unrealistic numbers, let’s say 50% of people responded to drug A, with a confidence interval ranging from 40-60%.  Let’s say 31% of people responded to placebo, with the confidence interval ranging from 31-41%.  Even though 50% sounds much better than 31%, the statistics show no “significant” difference between drug A and placebo.  Significance in a statistical sense means the differences are larger than what would be expected to happen by chance.

Statistics can be very useful, but they’re only as good as the data they’re based on, and they’re only generalizable if a convincing reason is given as to why they should be.

So, what can you watch out for?  If you’re just given a figure without any context as to how it was arrived at, the number alone isn’t going to mean a whole heck of a lot.  Statistical calculations should result in a better understanding of the phenomenon in question; when used properly, the purpose should not be to obfuscate.

Also, pay attention to the source.  The sketchier the source is, the more likely they haven’t adhered to proper statistical principles.

In the end, statistics aren’t inherently good or bad; it’s just a way of managing numbers.  The more you understand about how the work, the more you’ll be aware of what kinds of questions to ask about any stats you’re presented with.

And as for Peanut, watch out, because he’ll continue his campaigning to be the next ruler of the world.

 

There’s more about separating science from BS on the science corner page.

Mental Health @ Home Store - My Health Portfolio cover and inside pages

 

 

The free Health Portfolio on the MH@H Store will help you to collect all of your important health information in one place so it’s ready whenever you might need it.

28 thoughts on “Can You Believe Statistics?”

  1. With numbers you can prove everything.
    I always like to read the discussion in articles, that can give you a lot of information too.
    I’ll vote for Peanut every day of the year! No matter what the polls say, I have statistically looked out of my window and the clouds formed a 99.86% chance of Peanut winning. I’ll take that proof over every poll 🙂

    1. Maybe I went a little overboard here. Statistics can be very useful but you need to be aware of many factors involved. You need to understand what you are reading, what the question and hypothesis is, who conducted the study and why and how the results relate to the findings that were expected. Who pays for the study is interesting. I don’t know for sure where you can find out the questions that were asked, when they are not being mentioned in the study itself. I know that they look at every question (like in a depression questionnaire) to determine if the question isn’t biased and if it relays to the topic at hand in a good way. I always ‘belief’ that the questions are ‘good enough’.

      1. Yes statistics are a useful tool but they’re only good as how the data is generated. The less information that’s made available, the more dubious the numbers.

  2. “There are three kinds of lies: lies, damned lies, and statistics.” (Attribution is a bit fuzzy on this, but it was popularized in the USA by Mark Twain et al)

  3. I agree with you in that they’re not inherently good or bad, it’s a way of managing and demonstrating numbers and findings. Definitely good to consider the source. I actually do online surveys with the likes of Ipsos and have done for years, and it’s made me all the more sceptical on findings from those sorts of things. I take it all with a healthy dose of caution but I do think they can be fantastic in giving you an idea of something or driving research and social change. Great post on this, it does pay to be mindful of what’s behind stats. xx

  4. I think of statistic as a very useful tool you can rely on, but it’s good to stay critical to some point. You’ve said it very well that statistics aren’t inherently good or bad.
    I have a good friend who is a statistician and you just gave us a topic to discuss about, so thanks for that 🙂

  5. “Lies, damned lies and statistics” as Disraeli said.

    One of my bugbears is not giving enough information. Every year, the crime statistics come out and the press says, “Oh, murder is up by 26% in a year!” Yes, but the murder rate in this country is low, so even a big percentage increase might not translate to a statistically significant increase in the actual number of crimes. Conversely, noting the trend but not the amount is equally problematic. When I read Paul Krugman’s The Return of Depression Economics he said that then-President Obama was increasing state spending in the recession, which he said was good, while in Britain the Conservative government was cutting it, which he said was bad. Something about this bothered me and I googled the exact figures and, despite austerity, government spending as a percentage of GDP was still something like 9% higher in the UK than in the US, which seemed to render his entire argument questionable. From a Nobel Prize winning economist, that bothered me enough that I can recall it years later.

    Then there’s showing individual statistics, but not a long series, making it hard to see trends, or starting or stopping a series at a random or overly-convenient time. For economic stories, not making clear if data is inflation-adjusted over time or not is a big issue… I could probably think of more, but I ought to go…

    1. Yes lack of relevant details can make figures essentially meaningless. Giving relative but not absolute risk is one that really gets on my nerves. Who cares if your risk of something doubles when the absolute risk is less than 1 in 1000…

  6. Ohh, you know Peanut always gets my vote! 😀 Best guinea ever, and he’ll be the greatest ruler of the modern world! Woo hoo!!

    This gives me the fun of being back in psych class. I took two semesters of research methods in psych, and one semester of statistics in math. The statistics was the hardest. I’m a math whiz, but more algebraically, I think? Stats and calculus are both harder for me. But I do remember the deviations from the norm and such. It is kind of hard to know, even in a broader sense, whether we can believe political polls. Because I was polled once, and the questions were obviously biased toward wanting to receive a certain response.

    But I have a great story about this. In fifth grade, my older cousin Michele and I went to Kentucky Kingdom. (It’s called Six Flags now, I think–a theme park.) We took a taste test for Coke versus Pepsi. I could tell the difference, and I chose Coke, which I greatly prefer to Pepsi. The woman got all excited and gushed, “You chose Coke! YAY!” So then Michele drank both cups and chose one. The woman deflated into a sad, sad person and said, rather flatly, “Oh. You chose Pepsi.” I don’t know why, but that cracks me up. I guess her employers manufactured Coke but not Pepsi?

  7. Having seen too many researchers cooking their statistics in real life, I’m forever sceptical. I now see it as an interesting read but no more reliable than horoscopes.

  8. We became interested and skeptical of statistics like 20 years ago when McDonald’s and Burger King both claimed to have America’s favorite fries based on surveys. Hmmm 🤔 🍟

  9. An elementary teacher taught me how to manipulate “facts” and how to make them say anything you want. Since then I’ve been intrigued with the concept and used its throughout college and even to this day. People say you can’t fight science or numbers, I disagree entirely. I enjoyed your writing!

  10. I am so skeptical about research. I now will check to see who is funding if I can find the information as so often these days even through previously reputable research facilities are needing funding so taking funding from companies. I would hope that the integrity of these research facilities would hold strong. I have doubts.

    You can also skew any research by how you ask your questions. For example in yes no questionnaires, when it is not always black and white. So forced to lead you in one direction only which may in fact not be fact.

    The 0-10; 0 being not at all, to 10 being always. again it can depend on how question is written, the mood of the individual taking the questionnaire on the day.
    Research is important but so is honesty integrity.

Leave a Reply