Ask the right questions, get the right answers: Week 2 - Why order matters and how to get it right

A series of easy-to-digest tips on questionnaire writing.

 

In the first blog of this series we looked at the initial steps in writing your survey questionnaire – putting pen to paper. This gave you the framework for setting out the questions you want to ask, but just as important to the success of your survey is the order in which those questions are asked.

As noted before, writing a good questionnaire is critical to the success of your survey and the order in which you present the questions to participants can affect both the number of people you have completing your survey and the accuracy of the data you collect.

If the first few questions of a survey are boring, are difficult to answer or take a long time to respond to, participants are less likely to continue with the survey. Instead they will just close the window and think nothing more of it, or worse, will be left with a bad impression of your brand and your business. Surveys like this look unprofessional and amateurish, and suggest you don’t respect the time of the people you are asking to complete them.

The ordering of questions can also have a direct impact on the answers given. For example, a satisfaction survey may have a measure of overall satisfaction with the company, and a series of questions asking about a more detailed list of attributes. Whether you ask that overall satisfaction question upfront or whether you ask it after you have been through the list or attributes can make a big difference to the overall satisfaction scores you get. In the former, you are gathering the overall impression of the business and the brand. Ask it after you ask about the detail means the answer is more thoroughly thought through, but also influenced by events and factors that may be of little importance overall or may have otherwise been forgotten. Someone might say they are happy overall at the start of the survey, but then once you remind them that they had a slight issue with the website a few months ago, or there was a delay on their delivery, their answer may change. The measure you want will depend on what your overall survey objectives are, so think carefully when structuring your questionnaire.

Think about the following guidelines when structuring your survey. Try to put yourself in your customers’ shoes as you read through. Test it out on friends and family to see if they can answer easily and that it makes sense to them. And remember to keep your survey objectives in mind at all stages of the process.

Putting things in order

You should be thinking about structure throughout the questionnaire writing process, starting when you have your broad list of topics to cover, and reviewing after every step in the process.

  • Order the topics you want to cover in a way that will appear logical to someone filling in your survey. For example, if you are talking about a transaction process work through the process chronologically: begin with the research and product decision, then the ordering process, then delivery, then satisfaction with the product and aftercare service. This will help participants think back to what they were doing, and help them remember more accurately what they thought at each stage.
  • Participants will find your survey more engaging if you draw them in gradually. Start with broad questions, and narrow down to the specifics. Asking too much detail upfront won’t provide enough of an introduction to a topic to “warm them up” and get them thinking about the topics you want to cover. It can also give the impression of a long questionnaire to follow.
  • Don’t start with questions that are difficult to answer. This includes open-ended questions that require effort on the part of the participant and questions that are confusing or difficult to understand. Wait until the participant has warmed up to the survey before asking them to invest this sort of energy or you risk losing them.
  • If you have a question that will affect whether other questions are asked, for example questions you will only ask to people who purchased online, make sure you get that information before you ask the questions that depend on it! This sounds obvious, but when the predictor question is something that you would normally put at the end, like age or gender, it can be easy to overlook.
  • Think about whether any of your questions will influence responses to later questions, like in the example discussed above about the positioning of overall satisfaction questions.
  • Try to put boring questions, such as demographic information (age, gender etc) towards the end of the survey. No one enjoys filling in those questions, but if they have already invested time in answering the interesting part they are more likely to stick with it and complete the survey than if you put them upfront.
  • The exception to the previous point is if you have quotas for your survey. This is where you set limits on the amount of people you want to respond to your survey based on certain characteristics, for example you want responses from 50 men and 50 women. ALWAYS put the questions that decide your quotas upfront. There is nothing worse for a participant than spending time filling in questions only to then be told they are not eligible for the survey – this is disrespectful and could be disastrous for your customer relationships.
  • Signpost your participants through the survey to help their understanding and enjoyment of the process. Mention when a topic area is changing so they are not surprised, and can prepare themselves for the new area of questioning. This can be done with text screens or by adding an introduction to the first question in each block.

Putting the effort into structuring your survey well and you will reap the rewards in higher response rates and better data accuracy. All this will mean better insights and solid evidence on which you can make your business decisions.

Next time we will be looking at the importance of question wording. Follow me on Twitter or add this blog to your RSS feed to make sure you don’t miss it! And if you have any questions or anything you’d like me to cover in this series please send me a message via the “Contact us” page of this website.

If you feel you need a bit of support I offer questionnaire writing services, or if you’d like to write your own I am happy to review it for you and give advice on any improvements. Please contact me for more information.