latest forecast based on combining other forecasts shows a big increase in the probability that Remain will win the referendum, from 70.5% last week to 76.1% today. Intriguingly this has only been accompanied by a small increase in the forecast share of the vote for Remain, from 53.8% in the last forecast to 54.3%.
| Remain % share | Leave % share | Probability Remain wins | |
| Betting markets | 54.3 | 45.7 | 80.2 |
| Prediction markets | 79.1 | ||
| Citizen forecasts | 68.0 | ||
| Expert forecasts | 56.0 | 44.0 | |
| Volunteer forecasts | 54.3 | 45.7 | 76.3 |
| Polls | 53.7 | 46.3 | 72.7 |
| Poll based models | 55.8 | 44.2 | 80.7 |
| Non-poll based models | 51.8 | 48.2 | |
| Combined forecast (mean) | 54.3 | 45.7 | 76.1 |
(Individual forecasts collected in the evening of 24th May 2016.)
Some components of the forecast have changed barely or not at all due to limited or no new data. This suggests the combined forecast will inevitably be sluggish. On the other hand, the betting markets, prediction markets, volunteer forecasts and poll-based forecasts have all responded to the publication of opinion polls over the last week that are more favourable to Remain. To this extent recent fluctuations in opinion polls may be being amplified. Hopefully the combined forecast does a reasonable job of balancing older and more stable sources of information with more recent data.
While the polls help explain the movement in the betting and prediction markets it is interesting to consider whether those markets have over-reacted to the polls. The betting markets are now giving 80.2% probability for Remain winning, up from 73.5% last week. Similarly prediction markets have moved from 71.0% to 79.1%. Most striking was the movement in the currency markets following the publication of just one poll, by Ipsos MORI, showing 55% Remain 37% Leave.
That poll came out just after we finished data collection for last week’s combined forecast. Our poll of polls of polls has moved from 52.1% share for Remain last week to 53.7% this week. Not a huge movement.
Maybe the greater confidence in a Remain win reflects markets have over-reacted to particular polls or the fact that there have been relatively more telephone as opposed to online polls than usual over the last week and phone polls are more favourable to Remain. John Curtice and other commentators have sensibly downplayed down the suggestion of a big swing to Remain.
However, at the same time the Elections Etc poll-based model update yesterday showed how even a relatively small change in the polling average could lead to a substantial change in the probability of a Remain win now that there is less than a month to the vote. That raises further questions as to whether such models are estimating the level and distribution of uncertainty well.
METHODOLOGY
Details of methods and sources are below; there are no changes from the previous forecast. We do not endorse any of the component forecasts. As mentioned below, there are a couple of prediction markets with very low trading activity that we have not used. Otherwise these are all the useable forecasts that we know about. We have not excluded any forecasts based on our judgement of quality. Please do let us know if there are any other published forecasts that you think we may have missed and could be using. More generally, the methodology is still under development so comments very welcome.
Citizen forecasts
Citizen forecasts come from the results of representative surveys of voters, asking them what they think the outcome will be. Such voter expectation surveys have an excellent track record, arguably better than polls, prediction markets, quantitative models or expert judgment for US presidential elections. The percentage who think that Remain will win is taken as a collective estimate for the probability of a Remain win. Results of voter expectation surveys are listed here. For each pollster we take the average of the last two such polls within the last three months, and then average across pollsters.
Expert forecasts
At the moment in this category we just have the results of the Times Red Box sweepstake podcast contributors as our experts, plus some others we can identify as experts.
We considered taking the proportion of experts who gave predicted Remain shares above 50% as a pseudo probability of a Remain vote. However this figure would have been 95% and not fairly represent the uncertainty felt by many of the expert contributors. Nonetheless this unity here is striking. Moreover, there are various other published expert forecasts but they are not used here because they do not provide figures for either the share of the vote or the probability of one side winning. They too overwhelmingly point to Remain winning.
We hope that there will be more systematic surveys of academics, pollsters and journalists in due course.
Volunteer forecasts
Philip Tetlock’s Good Judgement project encourages people to forecast the outcomes of various social and political events and helps them learn and improve their forecasting skills. Tetlock claimsthat given some training, effort and practice, reasonably intelligent citizens can forecast better than experts, at least collectively. One of their forecasting challenges is the outcome of the Brexit referendum (here).
In addition we use the volunteered contributions to the Times Red Box sweepstake, i.e. the participants excluding the podcast contributors. For some academics and others that we know of we have reclassified their forecasts as expert. The proportion of these volunteered contributions giving predicted Remain shares above 50% is again taken to be the probability of a Remain vote. The median Remain vote share is used for the share forecast.
Prediction markets
These are websites which allow people to bet on the outcome directly with other participants, without a bookmaker setting the odds (explanation here). They are much lauded as a forecasting tool by many economists and business people because they draw on views from a wide range of people willing to risk their own money. For election forecasting they arguably have a better track record than polls, quantitative models and expert judgement.
For the Brexit referendum the prediction market websites only have markets for which side will win not the share of the vote. We use data from predictit and hypermind. We also use spread-betting markets from sportingindex and ig, taking the mid-point of the spread as the predicted probability or vote share.
Because of low trading rates we do not use ipredict.
Betting markets
These are traditional bookmakers. Even though the odds are formally set by the bookmakers, with enough people betting they are primarily driven by what the punters are willing to accept. We average across major bookmakers listed here after correcting for the over-round (whereby the sum of the implied probabilities from the published odds is more than 100).
Bookies allow people to bet on the share of the vote within particular bands (e.g. 45-50% Remain). To generate a combined vote share forecast we take the mid-points of the bands and weight them by the (corrected) implied probabilities. For large bands that extend to 0 or 100 we do not use the mid-points but figures five points from the interior bound. For example, if the band is 75% to 100%, we use 80% for the share calculation. Implied probabilities for these extreme bands are very small so the choice of mid-point makes little difference to the calculations.
Polls
For the share of the vote we use the average of the polling averages that are published by whatukthinks, Ben Stanley, Number Cruncher Politics, the FT, and ElectionsEtc. These polling averages typically aim to correct for differences between pollsters and so they should not fluctuate too much according to whether the most recent polls were online or telephone or from a particular company. So what we are calculating is a poll of polls of polls!
Since polling averages do not reflect the range of variance in the polls very well, we generate a pseudo probability for Remain winning from the proportion of polls that have Remain ahead. For this we take just the last two polls from each company-method combination within the last two months. So if a company has published two online polls and two phone polls in the last two months these are treated separately. Here we make no attempt to balance between online and telephone polls even though there are slightly fewer companies doing phone polls and they more clearly point more towards Remain.
Poll based forecasting models
Polls are a snap shot of opinion at the time they are taken. The historical relationship between polls and referendum outcomes tells us something about the direction and extent of any likely change in opinion, as well as the level of uncertainty we can expect of the outcome. Both ElectionsEtc.com and Number Cruncher Politicsprovide forecasts of this kind for both the probability of Remain winning and vote shares.
Non-poll based models
A traditional approach to election forecasting is to use historical data to develop a statistical model based on factors that are expected to influence the vote. These factors are often referred to as the “fundamentals”. See here for an introduction to these kinds of models for US elections.
