Dark Figures in Opinion Polls
In 2016 the British population voted to leave the EU in the so-called Brexit and a couple of months later the world was shocked even more by the election of Donald Trump as the president of the US. These events have some very serious implications: that classic opinion surveys underestimated large or at least very significant groups of the population up to the two elections. For many years analytics have known that to do a correct survey is a true work of art. This has not become easier with after “fake news” has been associated with classic opinion polls executed by traditional news media.
A very important reason for the bias of opinion polls is that the participation (response) rate has been in rapid decline the last decades123. In Denmark this is also the case. In the graph below the development of the participation rate in surveys regarding salaried employees in Denmark. These surveys were executed as a mix of telephone interviewing, filling out paper forms, and submitting answers via the internet. In only 20 years the rate of participation has dropped from 90 % to 50 %. The number of people that do not want to participate has been rising and their opinions can be very deviant from those who participate in the surveys.
Fig. 1 Development of participation (response) rate from 1990 to 2015.
Source: own calculations based on surveys regarding salaried employees in Denmark made by The National Research Center for the Working Environment. www.nfa.dk
The decline in the participation rate should worry all survey researchers. In general, there is an agreement that the reason for the decline is a combination of the following factors:
In general, the respondents feel overwhelmed by the number of surveys4.
Respondents think that the interviews are too long5.
It is often a bit difficult for the respondents to understand the questions quickly.
A part of the respondents think that the surveys represent “fake news” since it is often traditional
news media who are behind/have ordered the surveys.
No matter what the surveys promise some respondents will remain worried about their anonymity.
An illustrative example of the dark figures in Danish surveys is the number of respondents who acknowledges to voting for Dansk Folkeparti (DF) at the last election to the Folketing (the Danish parliament). For this example, we will use the data from European Social Survey (ESS) which is an international survey made by researchers that has been completed every other year in many European countries since 2002. ESS has the special quality that is has been executed consistently every time. This makes it so that it is possible to compare the results for each year without having to worry about changes in the design. Therefore, it is possible to follow the development of dark figures of the voters of Dansk Folkeparti.
In ESS they ask the question “What party did you vote for [in the last] election?”. Below is a comparison of how many respondents answered Dansk Folkeparti to this question and how many actually voted for DF at the national election.
Comparison of number of respondents that claim to have voted for DF and the actual percentage of the population that voted for DF at the last election for the Folketing.
These numbers are represented visually in the graph below.
Comparison between percentage who claimed to have voted for DF and percentage of votes of the population for DF.
In general DF has been strongly underrepresented in the ESS. Though, it has to be noted, that at the last round of ESS completed in the fall of 2014 there is barely a difference between the two percentages. It seems the voters of DF are have become more open to admitting to their support of political parties. A reason for this could be that DF became the biggest Danish party for the election of the European parliament in summer of 2014.
If one had compared the percentages of respondents who claim to have voted for another Danish political party in the Folketing, “De Radikale Venstre”, with the percentage they actually received at the last election the difference is actually reversed. The percentage of respondents who claim to have voted for “De Radikale Venstr” is higher than the percentage of votes that they actually receive.
The existence of dark figures in surveys poses a complex challenge and it complicated the ability to make prognoses about elections. At the election for the President of the USA in 2016 most surveys indicated that the Democratic candidate Hillary Clinton would win over Republican candidate Donald Trump. This was also the case in Florida which is one of the important swing-states. What happened with the opinion polls was most likely that the future Trump-voter did not participate in the traditional opinion polls. But both the Trump voters and Clinton voters alike use the internet. In the following Google Trend has been used to get a look at if the internet users searched for Hillary Clinton or Donald Trump up to the election November 8th 2016 in Florida. It is possible to look at what has been searched for but not for which reasons.
Source: own calculations based on Google Trend. The searches have been indexed.
Even though the traditional polls predicted that Trump would lose to Clinton in Florida, with a narrow margin67, a simple outline in Google Trend shows that more people searched for the name Donald Trump than for the name Hillary Clinton.
Something similar was the case at the last election for the national parliament in Denmark which was June 18th 2015. The current main party of the government Venstre had an unpredicted loss of voters while Dansk Folkeparti received again an unpredicted number of votes in South of Jutland. This could have been predicted to a certain if one had used Google Trend to look at which party name was more searched.
Searches on the party names Venstre and Dansk Folkeparti in South of Jutland up to the last election for the Folketing June 18th 2015
Source: own calculations based on Google Trend.
A preliminary conclusion based on the graph above would be that DF in South of Jutland would get more votes than Venstre at the national election.
Conclusion
Hans Bay. Vægtning af stikprøver (weight in surveys). Dansk Sociologi nr. 4 20. årgang. December 2009.
https://surveyanyplace.com/average-survey-response-rate
https://aspe.hhs.gov/system/files/pdf/255531/Decliningresponserates.pdf http://harvardpolitics.com/united-states/numbers-dont-lie-polling
Hans Bay
STATISTICIAN | VOLUNTEER
________
Being a statistician crunching numbers and the ability to handle vast amounts of date is part of Hans’ DNA but his real talent is to see beyond the numbers and understand what really motivates people. Behind every number in the world of consumer business there is a human action being taken. If we understand what motivates each action we are able to take better decisions.
More Publications
