Too few cell phones? Internet bias? Pollsters grope for answers.
In this week’s British elections, pollsters unanimously declared that no party would get a majority — until the Conservatives exceeded most predictions by a whopping 50 or more seats.
Two months ago in Israel, pollsters were broadly convinced that Prime Minister Benjamin Netanyahu’s coalition was in deep trouble — until it pulled off a sweeping victory.
Now, just 2½ years after most pollsters underestimated the extent of President Barack Obama’s reelection, the global polling establishment is again scrutinizing its data, while its American counterparts debated whether the methods of measuring political sentiment around the world are terminally flawed.
“For any polling company, there inevitably comes a time when you get something wrong,” YouGov pollster Anthony Wells wrote Friday in an early post-mortem on his firm’s website. “Every couple of decades a time comes along when all the companies get something wrong. Yesterday appears to have been one such day.”
YouGov, like most pollsters in Britain, significantly overestimated Labour’s support — showing the two parties basically tied for the three months leading up to Thursday’s elections.
The firms that surveyed voters across Britain and in specific constituencies — some of whom, like YouGov, also poll in the United States — said more time was needed to review exactly what went wrong. Up for debate are the differences between conducting polls via telephone and the Internet, how interviewers asked the ballot-test question and exploring the systemic reasons why British polls continue to underestimate Tory support.
“Seeing that really no country out there has really found the answer to this should be scary to people,” said Patrick Ruffini, the co-founder of the Republican digital analytics firm Echelon Insights. “It seems like things are going wrong a lot more than they’re going right, lately.”
Political surveys do more than measure just the horse race: They help explain the issues and personal factors that help voters reach their decisions. But when public pollsters botch the horse race, it undermines credibility in the industry as a whole.
And it would be difficult for the pollsters to perform any worse than they did before Thursday’s vote in the U.K.: Most surveys and the prediction models based on them showed the Tories and Labour as about even, with a better-than-even chance that Ed Milliband, Labour’s leader, could assemble a coalition with the Scottish National Party to oust Prime Minister David Cameron’s Tory-led government. Instead, Conservatives won an outright majority, 331 seats to 232 for Labour, and Milliband resigned as opposition leader on Friday.
The differences between using polling to predict election outcomes in a two-party race in the United States and multiparty, first-past-the-post parliamentary systems like Great Britain are numerous. But changes in communications are threatening the viability of public election polling in many developed countries where the landline phone was once a reliable medium for representative surveys.
Like their counterparts in the United States, Britons are abandoning their landlines. Telephone pollsters have shifted, contacting an increasing percentage of their respondents via mobile phone, but perhaps not enough, according to many pollsters. In the United States, polling mobile phones is far more expensive than relying on landlines, because survey researchers are required by law to enter cellphone numbers manually when dialing.
In Britain, many of the telephone polls conducted at the constituency level were commissioned by the billionaire Michael Ashcroft, who fielded surveys in more than 100 of the U.K.’s 650 districts in a seven-figure effort to help Britons better understand the election.
But Ashcroft’s final survey showed the two leading parties, Conservative and Labour, tied at 33 percent apiece — similar to almost every other national poll. At the constituency level, his results were also mixed. “Not a great night for any of the pollsters or forecasters,” Ashcroft said on Twitter Friday morning, saying a more complete post-mortem was needed.
London School of Economics associate professor Ben Lauderdale, writing at FiveThirtyEight, ESPN’s data journalism site, suggested that some analysts looked at the wrong data points in Ashcroft’s surveys: They focused on how voters said they planned to vote in their own local elections, rather than their broader preferences for a party.
Another leading pollster, Ipsos MORI, which polled by phone in Britain but had a mostly Web-polling presence in the United States, defended its British surveys but also vowed to reassess its methodology. The firm said in a statement that it was “within [the] margin of error for all the parties except for Labour, who[m] we overestimated. As we do after every election, we will be carrying out an internal review into our polling, how it can be improved and lessons that can be learnt, in order to become more accurate in the future.”
The majority of firms surveying the U.K. elections conducted their polls over the Internet. Nearly 90 percent of Britons have Internet access, according to World Bank data from 2013, even greater than the 84 percent of Americans who have access to the Web.
While there was little difference between the final Web and telephone surveys, there was a gap between the methodologies earlier in the campaign. Alberto Nardelli, data editor at The Guardian newspaper, noted last month that phone polls tended to favor the Tories, while Labour performed better in Internet polls. That phenomenon existed as late as a week ago, according to the New Statesman’s “May 2015” website.
“I would be concerned about what looks like systematic errors in the British polling — under-representation of Conservative voters and overestimation of support for the Labour Party,” said University of Michigan professor Michael Traugott. “I haven’t seen any polls that suggested that the Scottish National Party was going to do extremely well in Scotland at the expense of Labour primarily. So, systematic errors like that, patterned errors like that, suggest methodological problems.”
(Similarly, in Israel, some observers believe Web polling was biased against Netanyahu’s Likud Party and swayed the overall balance of pre-election forecasts.)
These failures, though not limited to Internet polling, have ramifications for an American survey industry in flux. Most political campaigns and larger media outlets use live-caller telephone polling for their pre-election surveys, but the costs associated with that research are skyrocketing.
That’s led a number of media outlets to experiment with Internet polling. The Associated Press, like POLITICO in 2014, used GfK’s probability-based KnowledgePanel, while Reuters commissioned polling of a self-selecting Ipsos panel. CBS News and The New York Times continued their live-caller polling relationship at a national level, but supplemented that work with a YouGov-produced Web panel that forecasted results of every Senate race and congressional election — a project for which they initially received criticism from other pollsters.
But if these recent elections are any indication, the polling industry might be stuck in limbo: Telephone polls are increasingly unreliable and too expensive, while Internet polling isn’t yet able to replace truly random surveys.
“I do think it remains a problem to the extent that people still rely on the phone surveys,” said Ruffini, of Echelon Insights. “I think an all-of-the-above strategy is what we’re going to have going into 2016.”
In the U.S., simply learning the lessons of the 2012 and 2014 elections and applying them to 2016 won’t work because Americans’ modes of communication are changing so quickly.
“One of the consequences of changing technology is that change becomes more rapid. It’s harder to keep up,” said Traugott, who led the Gallup Organization’s post-2012 review process. “So, we’re in an environment in which change is more prevalent and more rapid than it has been in the past, and we don’t understand all of the details of the consequences of mixing cell phones and landlines [for telephone polling] and the rate at which it should be done.”
If there’s some good news, pollsters said they doubted the U.S. could see a U.K.-style failure in the 2016 general election, given the unique systems in each country. But that doesn’t mean there aren’t significant issues with polling here, too.
“In any parliamentary system, it is tricky because it’s like you’re trying to project every House race in the country based on the generic ballot,” said Democratic pollster Nick Gourevitch of Global Strategy Group in New York. “And when you layer in multiple parties on top of that, it becomes more complex. I think, certainly here, that played a huge role — the complexity of the process.”
“I don’t think you could have that kind of surprise here,” added Republican pollster Jon McHenry. “We poll every state that has the potential to be a tossup. We’re going to see polls in some states that have competitive Senate races, even if they aren’t close [at the presidential level]. There will be lots of polls in Illinois because of [incumbent Republican] Mark Kirk, and they’ll all ask the presidential race right before that.”
McHenry instead compared the Tory landslide in Thursday’s election to now-Rep. Dave Brat’s upset victory last year over then-House Majority Leader Eric Cantor in a Virginia GOP primary — tens, if not hundreds, of Dave Brats running simultaneously that slip under the radar.
“Nobody’s really that involved in it, nobody really thinks there’s much of a chance for it to flip,” said McHenry. “And then, well, you know what happened.”