Polls Trumped: The polls are wrong again…why should you trust your market research?
Another election, another failure of the polls. Last night we were all going to bed thinking we’d wake up to find the US had elected its first female president. The reality when we woke this morning was far different.
The polls had predicted a close race but were generally agreed on who the winner was likely to be. The election, however, has delivered what looks close to a landslide in the opposite direction. In 2015, the UK General Election, the polls were agreed the race was too close to call but neither party would have a clear majority…then the Conservatives won a majority. To 2016, in the UK again and the EU referendum: the polls said it would be close but Remain would win, yet on the day Brexit prevailed.
It all leaves the business owner wondering, if the pollsters can get it so wrong with all the resource and expertise they have, why should we believe the results of any survey? What is the point of spending money on market research when it can’t even answer a simple question of who people will vote for?
Polling is a complex business. There will be enough analysis and reflection on what went wrong again (for example these articles from Research Live, The Guardian and The Telegraph) and I’m not going to add to that here. Let’s instead try to understand some of the problems faced by the political pollsters and look at the parallels and differences between this and your customer research. From this we can understand what we can learn from last night’s events.
- The diversity and nature of the population being surveyed is generally far broader in political polling than it is in customer research. The polls need to represent the whole population, and that means everybody, not just those who have the internet and have time to click on a survey invitation on a particular day, or who will answer the phone on a particular evening. A high response rate is absolutely key to achieving a representative sample.
Door-to-door research performs best at achieving a truly representative sample, however this is costly and slow, and today’s media is hungry for instant reaction. Therefore quick-turnaround online and telephone polls dominate.
Reasons to feel reassured: Your customers are already engaged to at least a small degree with your brand, which will help you achieve the high response rates needed for confidence in the findings. You can also leave the survey open for longer – days, even weeks, to allow those who cannot respond immediately time to do so.
You may also be able to narrow down how you approach them without fear of excluding any of them, for example, if you are an online business you will know your customers are also online and thus have the means to respond to an online survey.
Lessons to learn: High response rates don’t just happen – you need an enticing survey invitation, an engaging questionnaire, and you need customers’ trust that something will be done with the results. A well-designed survey is your first step towards achieving this.
You also need to be aware of any bias that may have been introduced by your survey method, for example an online survey may exclude those who do not regularly access a computer. You may wish to consider reaching out to this group via a different method (e.g. telephone), or using statistical techniques such as weighting to correct for population skews in the final survey data.
- Political polls require a high degree of accuracy, particularly when the race is close. A swing of one or two percentage points in either direction could change the whole picture. The outcome being measured is often a binary and mutually exclusive choice: will it be outcome A or outcome B (Trump or Clinton? Leave or Remain?). There are no grey areas and no in-betweens.
Reasons to feel reassured: In customer research you are rarely looking at such a clear-cut outcome. Say our survey tells us 51% of people like your new product and 49% do not. If these figures are 5 percentage points out and in reality only 46% of people like your new product, that is still a lot of people and sales are still likely to be strong. The research findings have still been useful in helping you make your decision to launch the product. In elections however, if 51% of people say they will vote for one outcome and it turns out that only 46% do, that changes the whole outcome of the election and the poll’s prediction is 100% wrong.
Lessons to learn: Accuracy is still important when business decisions are at stake, so the survey must be well designed to keep the results within an acceptable margin of error.
- A big issue for pollsters is the one of the “shy Tories” (or in the US perhaps we should be calling them the “timid Trumpers” fnar fnar). If people are leaning towards the less-socially acceptable choice they may be reluctant to admit it during a poll, even if they believe that the results will be anonymous. This is known as “social desirability bias” and is to do with our innate need to paint a picture of ourselves as we would like others to see us. Left-leaning political choices are seen as the more compassionate “nicer” option over the pragmatism of the right, and has an often-subconscious influence over how we respond to polls.
Pollsters try to correct for this bias statistically, but it is difficult to estimate its size as it’s clearly not something you can directly ask.
Reasons to feel reassured: Purchase decisions rarely involve controversial decisions that people will be embarrassed about, unless you are trying to research some very niche “specialist” products!
Lessons to learn: Social desirability bias is a problem for all surveys. People who feel very engaged towards your brand and positive may subconsciously answer questions more favourably as they seek to give the answers they think you want to hear. “Yes I love that product you’ve spent such a long time designing, I’ll definitely buy it!” is lovely to hear, but no good if they’re not actually going to follow through. The way you introduce the survey, making it clear that you want all feedback even if it’s negative and thus giving them “permission” to say bad things about you without hurting your feelings, is important here.
- Polls are designed to measure future intention, and this is very hard for people to predict. They may have every intention of going out and voting for X, but on the day the baby is crying, and it’s raining, and that meeting over-ran, and suddenly going and voting is the bottom of the list of priorities. Polls try to adjust for this by looking at past voting behaviour to try to predict who will and will not turn out, but this is only a best-guess and it could be argued is as much an art as a science.
Reasons to feel reassured: Level of accuracy required again comes into play here. Imagine our survey says 9 in every 10 customers would use this service again. As long as half of these are telling the truth our sales still look strong – we just need to be aware when we are interpreting the question that we’re looking at the general sentiment towards the product and not an actual prediction of the future.
Lessons to learn: The usefulness of research on future intention is all in the interpretation. We cannot predict with certainty how many people will perform an action, but we can measure how many think they will and this will usually be enough to say whether a product has future potential or whether the general feeling is positive or negative. We need to be careful if we are using this sort of data to forecast how much stock to order in and be aware of its limitations.
- There are also a number of statistical considerations for pollsters that do not come into decisions about analysing customer research. As well has having to marry up voting preferences with likelihood to actually cast a vote on the day, the format of elections rarely mean it is as simple as how many people will vote for each outcome. In the UK you’d need an accurate poll in every constituency, but as the number of constituencies make this financially unviable certain projections are made based on demographics and past voter behaviour.
Reasons to feel reassured: These considerations are rarely relevant to a customer survey.
Lessons to learn: Never be complacent – always review whether your survey audience has any peculiarities that mean you need to reconsider your sampling method.
"Many of the issues the pollsters face are also faced by us to some degree when carrying out customer research."
The mistakes of the polls may have made the headlines again, but that does not mean it is time to write off all market research. Polling is a highly specialised branch of research with its own peculiarities and quirks that make the pollster’s job unique. However, that is not to say we should be complacent about our own research. Many of the issues the pollsters face are also faced by us to some degree when carrying out customer research.
The news today is a reminder to us all that we need to be considering these the potential for introducing bias in our survey design, and be aware of the limitations when we analyse the results and feed the information into the business decisions we make off the back of them. The balancing act between quick and cheap research and accurate robust research has never been more pertinent, and the importance of good survey design and fair analysis never greater.
If you are running (or planning to run) your own surveys in-house, consider speaking to us first. Our flexible service model means we can offer as much or as little input into your project as you choose, from consultation and advice on reducing bias in your chosen survey methods all the way through to designing and analysing your research for you. Look at the services section of our website for more information, and don’t forget to give us a follow on Twitter and LinkedIn!
If you have found this blog useful please consider sharing it with your network. I don’t want the headlines today to give people the wrong idea about market research – when done well it is a critical part of running a business and making those all-important business decisions. Please help me spread the word that research still rocks! Thank you.
photo credit: televisione Donald Trump e` il Nuovo Presidente degli Stati Uniti via photopin (license)