Background
UK by-elections have been polled on and off at various points in history. Like national polling, polls of individual constituencies were traditionally done face-to-face, which was gradually replaced with telephone polling during the 1990s.
In the first half of the 2010s, Lord Ashcroft periodically delighted poll watchers by funding polls of individual constituencies, including at Westminster by-elections. But after he stopped, so did published polling of by-elections, with the result that no UK by-election has been publicly polled since 2016. Until now.
What we did
Number Cruncher uses a two-pronged approach to online polling, with respondents drawn both from panels carefully selected to avoid abnormally politically engaged samples, and also using a technique little used in the UK called river sampling.
River sampling involves recruiting respondents and polling them directly rather than first signing them up to a panel. It has various advantages even in national polling, because it allows us to reach the sorts of people who, for whatever reason, wouldn’t join a panel to take surveys regularly.
But it also allows us to poll locally, including for the Brecon by-election. This has not been done online before in the UK, because no panel is big enough to have sufficient membership in local areas.
The sampling is typically done through smartphone apps, but can also be applied to other parts of the internet in order to interview desktop users. Compared with using panels, it requires additional statistical and technological safeguards to catch duplicates and other sample quality issues – indeed several respondents had to be removed in Brecon and Radnorshire for this reason.
How accurate is it?
Naturally we wanted to test our sample against variables we weren’t weighting by (responses were weighted by age interlocked with gender and by age interlocked with education). The European elections less than two months ago provide one such data point.
The results were counted at local authority rather than constituency level, but based on the result in the county of Powys, and the differences in voting history between Brecon & Radnorshire and neighbouring Montgomeryshire (the two seats of which Powys is comprised), we can estimate the constituency result, and compare it to what respondents told us:
| Party | Powys result (%) | Estimated Brecon and Radnorshire result* (%) | Poll |
| Change UK | 2.0 | 2.0 | 1.1 |
| Conservative | 9.2 | 8.6 | 10.7 |
| Green | 7.2 | 6.7 | 3.6 |
| Labour | 7.5 | 10.7 | 10.9 |
| Liberal Democrat | 24.3 | 24.0 | 23.5 |
| Plaid Cymru | 12.5 | 10.9 | 11.5 |
| The Brexit Party | 36.1 | 34.2 | 36.5 |
| UKIP | 3.3 | 3.1 | 2.1 |
*Based on the 2019 Powys result and the 2015 and 2017 log ratio differentials between Brecon and Radnorshire and (Brecon and Radnorshire + Montgomeryshire).
Apart from being a bit low on the Greens, all of these differentials are well inside the ±4.4 point margin of error for a sample of this size.
We also asked how people voted at the 2017 election and the EU referendum. While these questions are subject to greater recall error because they probe behaviour 2-3 years ago, it is reassuring that the answers and results were very similar.
Our sample voted Tory over Lib Dem by 21 points in 2017 (actual margin 20 points) on an 81 per cent self-reported turnout (actual turnout 77 per cent) and voted 53-47 Leave in the EU referendum (Chris Hanretty’s estimate of the local 2016 result is 52-48).
We also looked at the geographic distribution of poll respondents, for which we did not set any quotas, compared with official estimates of the 18+ population in different parts of the constituency:
| Part of constituency* | 2017 ONS small area estimates (aged 18+, %) | Poll (%) |
| Llandrindod area | 38.0 | 38.9 |
| Central | 15.7 | 14.6 |
| Brecon area* | 31.4 | 29.9 |
| South | 7.8 | 7.1 |
*Where postcodes straddle the constituency boundary, only the part of the postcode that is in the scope of the poll is included in these numbers; Llandrindod area is LD1, LD6, LD7 and LD8; Central is LD2, LD4, LD5 and HR3; Brecon area is LD3; South is NP3 and SA9.
Next steps
Based on this analysis we are happy to put the poll results into the public domain. Further analysis of the microdata will follow, with a view to offering constituency-level and other local polling to clients. For further information, feel free to get in touch.
