I wrote on Wednesday that this week was turning out to be eventful, and it’s now gone beyond eventful and become something of a psephological soap opera. We’ve now had 4 phone polls, 4 online polls, an ill-informed attack on the polling industry from a respected broadcaster and an outbreak of civil war within the YouGov extended family.
Last night ComRes published its latest phone poll for ITN and the Daily Mail, showing another 11-point lead. The changes in the tweet below are the optical changes from last month, but there’s actually been a methodological change – the headline figures now use Andy White’s likely voter model, and on an apples-to-apples basis, there has actually been a swing towards Leave. So in terms of the direction of travel, the evidence is pretty mixed, but simply having a number of phone polls has pushed the simple averages massively towards Remain and in doing so has led to a dramatic shift in sentiment in betting markets (and arguably the media too).
REMAIN 52 (+1)
LEAVE 41 (+1)
— NCP EU Referendum (@NCPoliticsEU) May 19, 2016
The forecast model was relatively little changed by the ComRes poll, as we’ve now had quite a few polls in the last few days – the main effect is further to dilute the noise from earlier in the week, so Remain now leads by about 5 points in the polling average and has an 81% chance of winning.
Then this morning came a grenade from recently-retired YouGov president Peter Kellner, who is now at liberty to say what he really thinks and as such has become even more interesting than he was already. His new site, the Politics Counter, launched this morning with a post on the great phone-online puzzle, which cites the NCP/Populus joint paper Polls Apart, and also draws on a number of issues discussed on this site recently. His conclusion is that phone polls are likely to be nearer the mark on this occasion.
— Tim Montgomerie ن (@montie) May 20, 2016
Unfortunately, the Kellner grenade landed first, and provoked a furious response from YouGov CEO Stephan Shakespeare, who appeared to blame Kellner personally for last year’s miss.
The bombshell turned out to be a new study, making YouGov the latest pollster to conduct parallel phone and online polls in an attempt to understand why the two modes have been producing consistently different results. The writeup is pretty short, and doesn’t contain a lot of numbers, but YouGov is releasing the microdata, so I’m going to avoid jumping to conclusions until I’ve been through that in detail.
It’s clear, though, that a lot hinges on the phone sample involved. As YouGov is an exclusively online pollster, the fieldwork was outsourced to a company called Feedback Market Research, who I don’t know much about besides them being headquartered on a farm in Humberside. The sample of 996 GB adults is rather different from other phone samples – it will be interesting to examine how.
And finally, Adam Boulton tweeted this last night:
Let me just say one thing. All the opinion polls in this referendum campaign are a waste of time. The polling industry is in crisis.
— Adam Boulton (@adamboultonSKY) May 19, 2016
“Crisis” is open to debate, but the idea that polls are a waste of time is completely wrong. Yes, we know polls can get it wrong and that some of the EU referendum polls are wrong (the debate is around which ones). But even so, they are not a waste of time. The best gauge we have of public opinion is to ask people what they think in the most scientific way possible. No method is perfect, and that’s why the public, whether directly or via the media, need to hear from independent experts to get the full picture, and why journalists need to be careful how they report individual polls.
But the alternative is to allow expectations and the narrative to be driven by guesswork, rumour, speculation, briefings, betting markets and the like, all of which are open either to bias or manipulation (and in some cases both). If you need a case study, look at the Oldham by-election. Both parties, the commentariat and the punters were all convinced that it was too close to call, only for Labour to win by 39 points, dwarfing any British polling miss I can think of. Just to put that into perspective, the general election errors in 2015, 1992 and 1970 put together would barely cover half of the Oldham “forecast” error. In the EU referendum, the difference between the biggest Leave lead (4 points with ICM online) and the biggest Remain lead (18 points with MORI) is 22 points. Even if the result were bang in line with one of these, the other would be wrong by, again, barely half the amount that the Oldham speculation missed by. At that is why we have opinion polls.