About that ORB poll (and NCP’s referendum forecast)

EU flag

Image: SXC/szymon szymon


It’s been quite a week for eyecatching poll numbers! On Monday, ICM’s monthly voting intention poll for the Guardian showed Labour and the Conservatives level on 36% – the first poll this parliament not to show a Tory lead. However it came with an abundance of caveats, following a series of previous warnings from ICM that they are still probably oversampling Labour voters. With characteristic honesty, Martin Boon essentially called his own poll a rogue.

But the ORB poll for Telegraph didn’t come with any pollster caveats. Many initially assumed that, like previous ORB polls, this one had been done online. But it was actually done by telephone, which is significant because phone polls have typically been showing comfortable remain leads, not a neck-and-neck race (it showed Remain 47 Leave 49). What is going on?

The first point to make is that ORB hasn’t published a poll using this methodology before, so we have nothing to compare it to on an apples-to-apples basis. Therefore we can’t tell whether these are its “normal” numbers, or an aberration.

The second point to make is to praise ORB’s transparency – they released a huge file of crosstabs, providing far more detail than is normally disclosed.

But… Those tables show some very strange patterns, which ought to raise serious questions. The most obvious is that are are too many homeowners in total and far too many homeowners in social grade DE. That suggests a very disproportionate number of home-owning DEs. There were also too many of this SEG among certain voters and 2015 voters. So it seems the sample isn’t particularly balanced.

Additionally, the weighting looks pretty severe. Obviously data needs to be weighted to the correct targets, but if as much as this is required, there must have been some groups that were extremely difficult to find. Finding it hard to get hold of people in general could explain the odd sample size of 823. And although that size isn’t disastrous, when combined with low weighting efficiency (a measure of the degree to which weighting effectively reduces the sample size) we’re looking at a big margin of error on top of sample bias.

Therefore I would treat this poll with an enormous amount of caution, while commending the level of detail in ORB’s disclosure.

Elsewhere we got a couple of “normal” online referendum polls from ICM and TNS. NCP will have a poll-of-polls up soon (one that doesn’t jump around depending of whether the last poll was phone or online). For the latest and EU referendum polls and updates, you can now follow @NCPoliticsEU:

And what about the NCP referendum forecast? Well, a considerable amount of work has been going on behind the scenes. As always, the priority is getting it right, even if that means delays (although last year I probably took that a bit to the extreme!)

Expect further details next week. In the meantime Steve Fisher and Alan Renwick at Elections Etc have published their initial prediction, suggesting an 87% probability of Remain winning, quite a bit higher than the (very consistent) 70% probability implied by Betfair.

Do I think one of them is right, or that both numbers are wildly wrong? You will find out here!

About The Author

Related Posts