Are you struggling with writing questions for your online survey?

Our Social Research and Data Lead, Lucy Smith, and Angela Schlenkhoff-Hus, our colleague from Coalition for Efficiency, have teamed up to deliver an excellent guide to make surveys, delivered through the DataWise London scheme. The aim of this training series is to provide London’s civil society sector with the skills, resources, tools and support they need to develop their survey writing skills.

Our first guide goes through our top planning tips, and includes a data visualisation guide.

In this second guide, we go through the process of writing effective questions for your online survey, with what you should and shouldn’t include.

Online survey question writing: how it’s done

Writing good questions is key to delivering an effective survey.

There’s the simple matter of writing your questions. Questions should always feature clear, concise and uncomplicated language. Don’t write questions where the user has to read several lines before reaching the question mark.

You should avoid acronyms, technical terms or jargon that may confuse your respondents.

However, if you need to include tricky terms or concepts, be sure to provide definitions or examples.

Give a time frame

To help focus your questions, you should provide a time frame. For example, imagine you’re trying to discover how often a sample uses social media. When you ask the questions, you want to know whether they use social media on a “daily”, “weekly” or “monthly” basis.

Using a time frame makes it easier for respondents to come up with an answers. For example, if you’re interested in how much money people spend on groceries, you wouldn’t ask for a yearly figure, because few people keep records on such a wide data range. However, many people would be able to provide a good number for their weekly expenditure.

If you don’t have a specific time period in mind, begin questions with “in general”, “overall”, or “typically”.  Avoid absolutes, as strongly worded questions do not result in nuanced answers.

Question ordering

Always start and end with something fun or interesting!

You could begin with general questions before moving into more specific questions. It’s a good way to warm people up before you ask anything complex or sensitive.

Another option is to move from unprompted to prompted questions. Unprompted allows for a more detailed answer, whilst prompted involves ticking a box. This works because the respondent will have energy to provide detailed information at the start of the survey, where their engagement is at its peak.

Watch out for things that might influence later responses. If you are asking someone to consider a question from a parent’s perspective, the respondent might go on to answer all questions in this mindset. Ensure you tell the respondent to only answer the relevant question from a certain perspective – or you could capture very biased data.

Survey Question writing: what to avoid

Confused? Read on, and you’ll be a survey wizard in no time!

Leading Questions

Avoid leading questions. For example, “does our excellent new advice make your life easier?” is influencing the respondent towards a particular opinion. You want to know what the respondent really thinks.  Instead, you should use “how do you find our new advice line?”.  It’s a neutral question that opens the floor for the respondent to be honest.

Loaded Questions

You should steer clear of loaded questions. Take a question like “In the past week, how many hours did you waste on social media?”.  Loaded questions often cast aspersions on the respondent’s behaviours. A better – and more civil way – is to instead use “in the past week, how many hours did you spend on social media?”.

Double-barrelled Questions

Double-barrelled questions, meanwhile, are confusing. Avoid a classic blunder like “Do you eat heathy foods and exercise everyday?”. This is because the respondent could eat healthy foods on a daily basis, but may not necessarily exercise, and vice versa. Instead, split the question into two separate questions, and include a time frame.

Unbalanced Questions

Unbalanced questions are also problematic. Avoid phrasings like “do you like exercise”.  Human response is complex and shaded – one person might like exercise, but another person might like exercise even more so! It’s far preferable to ask “How do you feel about exercise?” and include a scale.  This way, you can capture more nuanced data.

Overly Broad Questions

Overly broad questions are also a problem. Remember, you want specific information, not generalised information. A question like “what do you think about our newsletter?” is not useful. “What do you think of the content of our newsletter?” encourages the respondent to provide a specific opinion.

Scaled Questions

When asking a scaled questions, you should keep a balanced set of values. For example, imagine you are trying to measure the respondent’s opinion on customer service worker’s performance. You could ask something like:

How helpful did you find our advisor?

• Extremely helpful
• Very helpful
• Helpful

If you did this, the resulting data would offer a very biased view of the advisor. The respondent does not have the option to say anything negative about the advisor’s performance. A better question would look like this:

How helpful did you find our advisor?

• Very helpful
• Helpful
• Neither helpful nor unhelpful
• Unhelpful
• Very unhelpful

How to write a scaled survey question

We have provided an example of a scale you might use when taking a sample. Asking scaled questions can create more nuanced answers than an absolute question. Take a look and see if the examples help you.

Satisfaction:  Very satisfied, Fairly satisfied, Neither satisfied nor dissatisfied, Fairly dissatisfied, Very dissatisfied + DON’T KNOW.

Agreement: Strongly agree, Tend to agree, Neither agree nor disagree, Tend to disagree, Strongly disagree  + DON’T KNOW.

Frequency: Always, Usually, Sometimes, Rarely, Never  + DON’T KNOW.

Importance usefulness, confidence: Very important, Fairly important, Not very important, Not at all important + DON’T KNOW.

Quantity: A great deal, A fair amount, Not very much, Not at all  + DON’T KNOW.

Recommend: 0 = Would not recommend, 10 = Would definitely + DON’T KNOW.

Mandatory Questions

Some questions are mandatory. This means the questions must be answered if you are to gather comparable data. It also allows for a before and after comparison of the respondents.

Optional questions

Respondents may not know the answers to some of the questions, or may feel uncomfortable providing an answer. Forcing respondents to answer questions may make them more likely to quit, so ensure you think carefully about whether or not a question is essential.

Types of Questions

You can ask a question in a variety of different ways, and in so doing you can get different types of results. Here are a few examples.

 Multiple Choice

• The respondent chooses one or more options from a list.
• Try to limit them to one selection (ensure the list contains mutually exclusive choices).
• Make it clear if people are allowed to select more than one option.
• Provide an option to add an alternative to those on your list.

Rating

The respondent is asked to select one point on a rating scale, e.g. from Strongly Agree to Strongly Disagree.

Ranking

Respondents are asked to place a number of options in their order of preference. We would discourage from using this particular question form, because it can confuse respondents and will make your survey more time intensive. However, there are certainly some situations where ranked questions can work – it’s just not advisable to overuse the form!

Here is an example of what a ranked question could look like:

Please put the following methods of communication used by the senior management team to disseminate performance information in order of effectiveness (1 = most effective; 5 = least effective):

Closed versus Open-ended questions

When you want results that are easy to quantify, use closed-ended questions.

When you want to give respondents the freedom to express themselves in their own words, use open-ended questions – these might require the respondent to fill a box up with text.

Asking sensitive questions

Sometimes, you will be called to ask sensitive and difficult questions during a data collection. It’s never easy, but here are some ways of ensuring you can collect the data without causing any undue discomfort.

• Ask personal or sensitive questions after establishing rapport with your respondent
• Start with questions that don’t make respondents feel vulnerable just like in- person conversations
• Make clear why you’re asking a personal or sensitive question
• Consider that asking for identifiable information might affect other answers on the survey