Blog/Polling

We recently completed our internal review of our polling performance and methodological approaches for the 2024 UK general election, analysing both traditional and MRP polling. We are pleased to now be able to present the findings publicly.

While our performance in both forms of polling was strong, we continually strive to improve as a research organisation. We have already implemented several changes to enhance the accuracy of our polling estimates going forward.

Traditional voting intention

Our final regular voting intention poll, conducted 14–17 June (two to three weeks before the general election), showed Labour leading by 22 points — consistent with other pollsters during that period. Our MRP poll served as our final public release of the campaign.

To provide the most accurate assessment of our performance, this review will focus on our final (unreleased) internal poll of the general election, conducted in the days before the election.

Using our voting methodology as it existed last July, our final regular voting-intention poll would have indicated the following vote shares: Labour 40.1%, Conservative 23.9%, Reform UK 15.5%, Liberal Democrats 9.4%, and Green 5.8%.

When measuring accuracy using RMSE (root mean squared error) of the five largest parties' vote shares and the Labour-Conservative gap, our final survey would have ranked 7th among 20 polling companies.

What we did well

Several key methodological choices helped us achieve results in the upper tier of pollsters.

Turnout model

Rather than solely relying on respondents' self-declared likelihood to vote, we developed a turnout model using validated voter panels from the British Election Study across the previous three general elections, with voters cross-referenced against local electoral registers to determine whether they voted at each general election. This model predicted each respondent's turnout based on demographics and survey responses, including past voting behaviour and political interest levels.

The turnout model improved our accuracy compared to using self-assessed likelihood to vote. It adjusted the Conservative vote share from 23.1% to 23.9% and Labour's  from 41.1% to 40.1%, producing a 1.8-point net effect on Labour's margin.

We remain confident this turnout modelling approach is the right one, and we are pleased to see its adoption by other pollsters.

‍

Accounting for undecideds

To better understand undecided voters, we implemented a 'squeeze' question in our voting intention polls. Respondents who initially answered 'don't know' or 'I will not vote' were asked how they would vote if they had to choose, keeping 'I would not vote' as an option but excluding 'don’t know'. Those who selected 'I would not vote' again were excluded from final voting intention figures, with everyone else reallocated to their squeezed party choice.

Our pre-election research showed that most initially-undecided voters would reveal their preferences when pressed, and this squeeze method proved as effective as other reallocation approaches which carried greater risks.

Pollsters who incorporated undecided voters consistently showed lower Labour leads throughout the campaign, which the election results later vindicated.

These two methodological choices reduced Labour’s lead over the Conservatives from 19 percentage points (using demographics only) to 16 points. While this placed us well among our peers, we would not be comfortable continuing with a methodology that still missed the gap between the two largest parties by six points. The next section examines areas for improvement.

Potential sources of error

Past vote weighting

Due to concerns about false recall in an era of high vote switching, we opted not to weight by past vote at last year’s election. In hindsight, that looks to have been a mistake. If we simply used our 2024 method and also weighted by recalled 2019 vote, Labour would have fallen further to 39%, with the Lib Dems seeing a jump to 11%, much closer to their actual vote share.

Political engagement

A significant challenge in polling in Western democracies is the spectre of low-interest respondents, i.e. those who generally pay minimal attention to politics. Naturally, those who are less politically engaged are less likely to answer surveys about politics. On the contrary, those who follow politics obsessively are much more likely to join an online survey panel or say yes when a researcher call them up.

If political engagement bears no relationship with voting intention (or some other outcome we are trying to measure), this would not be a problem and we would not need to weight by political attention. However, the industry now faces the growing problem of political engagement being increasingly correlated with voting intention, leading to sometimes large polling misses.

In the UK, highly engaged respondents were more likely to support Labour (overestimated by every pollster in 2024), while in the US, less engaged respondents tended to support Donald Trump (who outperformed the polls in three consecutive presidential elections).

To try and mitigate this issue, we will be weighting by political interest levels going forward, using targets derived from the British Election Study, which have response rates which most pollsters can only dream of. However, we are also using a non-response bias adjustment method to account for the high likelihood that, despite relatively high response rates, even those who answer British Election Study surveys are on average more politically engaged than those who do not respond. We have designed a rigorous methodology to generate estimates of political interest for the whole population, rather than just those who participated in BES surveys. 

Weighting our data based on our 2024 method alongside past vote and our political interest estimates reduces error even further, improving the vote share estimate for every party. Labour falls from 39.3 to 38.2% and the Conservatives go from 23.2 to 23.7%, with Reform dropping from 15.3 to 14.5% and the Lib Dems and Greens climbing by 0.7 points and 0.3 points respectively.

This methodology would have placed us second out of 20 pollsters in terms of accuracy:

Late swing

Late swing – i.e. people changing their minds at the last minute – is very much the ‘get out of jail free’ card for a polling miss, but Labour’s vote share did fall over the course of the election campaign, and there is evidence that people voted differently to how they said they would.

Very helpfully, YouGov produced some analysis in their own post-mortem suggesting that Labour’s vote share was about 2.4 points lower in post-election vote recall than their final (unpublished) poll among the same set of respondents.

If we apply YouGov's differences in recalled vote for all parties to our methodology including past vote and political interest, we get very close to the final result, with every party within 1 percentage point of their election vote share.

Given the sampling error inherent to medium, this is more or less as accurate as one can expect an opinion poll to be without a good deal of luck. We are therefore confident that, given the amount of late swing that we saw, weighting by past vote and non-response-bias-adjusted political interest levels can solve most of our issues from 2024. These changes have now been implemented for our UK opinion polling, and our latest set of national voting intention figures using this new methodology will be published very soon.

MRP

Our final MRP model of the election campaign performed very well, predicting 87.3% of seats correctly, just one seat fewer than the on-the-night exit poll. In terms of total vote share error across all parties in all seats, our model was the second most accurate of nine public MRP polls.

One concern we raised during the election campaign was a function of MRP polling known as attenuation bias, or the tendency to pull regression values towards zero. As a result, many MRP polls had distributions of party vote shares across constituencies that were too ‘flat’ compared to a typical distribution. We predicted before the election that MRPs which did not account for this problem would probably end up being less accurate.

To fix this issue, we implemented a basic adjustment method while a different model, which we termed ‘regradienting’ internally – with more basis in the actual survey data – was developed. Some other pollsters also used an unwinding model, but we were somewhat cautious that these were mostly based on historic election patterns rather than the survey data, so were in some ways arbitrary. Historic patterns are of course valid, but there is no guarantee that they will hold. We wanted to design a methodology that would remove the arbitrary element of what 'looks' right, and instead use the survey data to determine the distribution of vote shares.

We are publicly releasing some of the data on this for the first time, showing the standard deviation in party vote shares (i.e. how flat the distributions were) in our initial MRP output, compared to our new ‘regradient’ model. Our regradient model produced more accurate results for five of the seven main parties, including getting the distribution of Conservative votes effectively bang on.

Had we fully implemented this new model, we would have seen a result which more accurately estimated the seat counts for the smaller parties, with the Lib Dems, Reform and Greens all coming closer to their actual results. This method has now been deployed across the Focaldata MRP platform, but we are continuously seeking out improvements where we can.

Our final MRP release also underestimated the number of votes cast for ‘other’ parties and independent candidates, in part due to the way our MRP voting intention question was asked. Our MRP and traditional voting intention questions have now been standardised, with explicit options for ‘an independent candidate’ and ‘another party (e.g. Workers Party / SDP / Yorkshire Party)’ to better capture the vote shares and distributions for minor parties.