Where the polls went wrong

What can be done?

Clearly further work is needed in a number of areas, and I’ll be providing detailed suggestions in due course. A key issue is political engagement bias. Some pollsters have been asking people things like whether they watch Newsnight, for example, though as far as I’m aware none use this data for weighting.

NCP has under development a political engagement model, based on this analysis, which attempts to identify groups of voters that are undersampled. The priority for this will be to develop methodological improvements that solve the underlying problems, rather than simply retrofitting data to past election results.

However, an initial, unrefined version of the model, when applied to online polling data, corrects for around 4 points of the error or around 5 points using a basic adjustment to measure and account for differential likelihood to vote. This represents a substantial portion of the error at the general election, when opinion polls understated the Conservative lead over Labour by 6.5 points. With further refinements, even greater improvements in accuracy could be possible.

The working paper from Jon Mellon and Chris Prosser contains a number of important suggestions. One of these is to “gross up” non-voters in a sample to realistic levels before applying population-level demographic weights. This makes a lot of sense, notwithstanding the problem of people over-reporting their turnout, which remains a problem (though the vote validation exercise will provide further clarity).

This voters-verus-population mismatch underscores the case for a nationally representative exit poll. This was originally done instead of – and in 1992 alongside – the broadcasters’ seat forecast exit poll, and in those days the broadcasters each did their own. It’s without question that the cost of this exercise would be very high, as it would have to be in addition to the current exit poll, which is solely concerned with how people voted in marginal seats. But given the problem of voters and the electorate being two distinct populations with different demographics, the value of that information may very well exceed the cost of obtaining it.

It would have the added advantage of being useful in real time – given how slowly results seem to be counted these days, it would give the broadcasters something to talk about, besides giving airtime to politicians that want to make very bad predictions about the exit poll’s accuracy. It would also avoid almost entirely the problem of recall error.

Another important area is transparency. Thanks to the BPC, Britain is a world leader in this area, but there are some additional steps, like providing more readily reconcilable tables, that ought to be considered.

Ultimately the industry wants to restore confidence in its research. Public perceptions have gone from one extreme to another – before the election the assumption was that everything that went wrong in 1992 had been fixed in 1993 and that significant polling errors didn’t happen anymore. After the election, some took the view that polling was broken and no longer useful. Both of these attitudes are wrong and unhelpful.

Polling has its pitfalls. This year things went badly wrong, as they did in 1992 and in 1970, and as they probably will another generation from now, if not sooner. But most of the time, polls get it right, and provide extremely useful information. If public opinion matters, then understanding it matters too, and measuring it scientifically is the only way to do that. These two facts are equally important, regardless of whether the last major polling failure was six months ago or 23 years ago.

Correction and update (18th November): In the original version of this post, I wrote that the the BES cross-sectional survey was sampled using the electoral roll. It actually used the Royal Mail’s Postcode Address File. Thanks to Patrick Sturgis for this correction.

I also forgot to mention a very important piece of work on Labour to UKIP switchers – Revolt on the Left by the Fabian Society. Thanks to everyone that pointed this out.

About The Author

Related Posts