There are polls, and then there are “polls.”
Polls aren’t just a few questions you slap together and ask to anyone you feel like.
Polls are complicated bits of social science. They’re based on actual sets of scientific principles and industry best practices, like Canada’s MRIA or the British MRS. The pollsters and stats nerds behind polls use the best methods they can to try to get a random sample of a population so they can draw accurate conclusions about that population.
These pollsters and stats nerds are happy to tell you all about these methods. They’ll tell you things like:
- What they used as a sample frame (i.e., what’s our population? where did we get their contact information? does that source accurately reflect the overall population?)
- How they selected the sample (e.g., clustered? stratified? quota?)
- How they contacted participants (i.e., telephone? internet panel? in-person?)
- How they asked the questions (e.g., did you test the questions beforehand? are they standardized questions? have you and/or other pollsters used them before?) and
- How they weighted the final data to make it as accurate a representation of the population as possible
You can use this kind of boring, boring information to evaluate whether a poll’s drawn accurate conclusions about the population in question.
Of course, that doesn’t always happen. But when polls do go wrong, like they did in predicting the 2015 UK election outcome, this transparency allows different pollsters to look at what they did, understand what went wrong and how to make their polls more accurate in the future. You know, science.
“Polls,” however, are different.
A “poll” is when someone with an obvious agenda wants an impressive-sounding number to make a sexy headline.
The UK’s Daily Express just gave us one of these “polls.” Take a look at this headline:
“80% of Brits would be happy to quit UK for RUSSIA after Putin offers free land”¹
Really? 80% of all people in Britain? Really??
Of course not. This was a “poll,” not a poll.
This “poll” is just a question on their website that was answered by 22,000 people that read “Would you move to Russia in exchange for free land?” with the options being “Yes! Bargain” or “No way.”
Real polls don’t run on websites open to anyone and everyone. If you don’t know who you’re talking to in terms of basic demographics (age, gender, location, etc.), you don’t have a real poll.
Real polls don’t allow people to vote more than once. I did, twice, even though I’m a) not British and b) not in Britain. Try it. Delete your cookies and go again. And again. Maybe three times is enough.
Real polls don’t ask leading questions or have leading response options for yes/no questions that sound like they were written by some guy down at the pub.
And worst here, real polls don’t do all these things and then have the balls to say something as groundless as “80% of Brits” in the headline.
But “polls” like this make good headlines for propagandists. No wonder then that Russia’s RT and Komsomolskaya Pravda, to name two of the Kremlin’s finest examples, picked up on it.
“80% of Brits want to move to Russia after Duma considers giving out free land – poll,” raves RT.
“Almost 80% of Brits are ready to move to Russia for a plot of land in Siberia,” Komsomolskaya Pravda says (my translation from the Russian).
Blech. This put me into desk-flipping-over mode.
Propagandists know a few things, especially when it comes to polls and “polls.”
They know most people only read headlines and aren’t about to scroll through an article for details on how a poll/”poll” was done.
They know a lot of people don’t understand how polls are actually done and don’t have the luxury (or pain) of knowing a former stats nerd/pollster monkey to explain all this crap to them. And they know that numbers and percentages stand out, especially anything that reads ‘x% of y think that…’
Think about this the next time you see a headline proclaiming a poll result.
1. Do not trust headlines that feel the need to resort to CAPITAL LETTERS.
Also, the irony of a UKIP-backing, Brexit-loving paper using what I’d consider a Gallicism here has really, really made my day