A key feature of Fulfilling Lives is that it is a learning programme; it aims to gather and share learning about what works, what doesn’t and what needs to change. A frequently used method to collect views and experiences from which to develop learning is the survey. In this blog, Kerry Dowding (Research and Evaluation Officer, Fulfilling Lives South East) shares her tricks and tips on how to get the most out of your survey design. Kerry has been creating surveys for the voluntary sector for the last eight years, and received a Masters of Research in 2016.
It’s 4.50pm, and you remember that you promised to send out a few survey questions by the end of the day. You scribble out some questions that roughly fit the bill, and send them off. The responses you get back are less than inspiring. So you end up going with the ideas you originally had for the project. We’ve all been there!
Why bother with survey methods?
A good survey can bring so much more to the table than this, if they are your method of choice (and they don’t suit every situation all of the time). A good survey can provide quality information to help us change systems and organisations, and to move forward in a way which is inclusive and co-produced. When done well, they are the gift that keeps on giving. Looking to get more from your surveys? Here are some hints and tips I’ve picked up along the way.
Before you start
- Why are you doing the survey in the first place? Write one sentence explaining the question you would like your survey to answer (and some smaller aims within that, which is fine – as long as they’re clear). Refer back to this and be strict – leave out anything that doesn’t help you to answer your question.
- How much time do you have to analyse this, really? If you have limited time, write fewer questions. Better to ask five questions that will be well used than twenty which you end up skimming over.
- Do you need to offer incentives? Some groups, like clients or volunteers, might be more likely to do your survey if there is an incentive. This is especially true if the survey is long, or they haven’t been in touch with the organisation for a while.
- How do you want to divide your results? You might like to split your data by area, hostel, gender or shoe size, to compare what different groups say in response to your questions. It’s important you think about this first so you include the questions you need. Just make sure these splits have a clear benefit to your work. For example, asking about shoe size is only really useful if it were clearly linked to something like NDT Which it’s not.
Question types and order
- Keep everyone in the loop. Tell them who you are, why you’re asking them questions, how long it will take, and what you’ll do with it all. It improves your response rate and is generally a good thing to do for people who are sharing their thoughts with you.
- Mix up open and closed questions. People can get bored with all open questions and provide poor quality answers. On the other hand, multi-choice ones don’t give you a lot of information to go on. Try putting open questions towards the end, so people get to them after they’ve been ‘warmed up’ by the easier ones.
- Throw in other types of question. Wake people up a bit by asking them to rank different options, use a slider to rate something from 0% to 100%, or use visual methods. There are lots of different types of question that are more fun to fill in (and analyse) than your regular.
- Don’t scare people off! Keep the sensitive questions for later in the questionnaire. Asking about someone’s criminal convictions or worst fears right off the bat can put people off.
A word on wording
- Be friendly and clear. Use clear, short words that everyone can understand. Think about trying to keep a warm, chatty tone (depending on your audience), and use plenty of examples for any tricky concepts – so there is no ambiguity about what you mean.
- Only one question per item! If you ask 3 things in one question, you don’t know which part of the question the person is answering. Asking ‘Do you feel happy, proud, confident and empowered through your work with Fulfilling Lives?’ in a yes/no question makes it unanswerable. A good rule of thumb is to banish the word ‘and’ from you questions as much as possible, so you can trust the answers you get.
- Keep an open mind. We often hope answers will be positive, but don’t let this optimism seep into your questions and make them look biased. Rather than asking if things are ‘better’ or ‘worse’ than before, just ask about what changes a person has noticed and see what they say.
- Make sure you cover all bases. There should always room for people to give an answer you didn’t expect, say they are unsure or surprise you. Sometimes the best answers refer to ideas you hadn’t even considered.
Getting it out there and making it useful
- When the results are in, get together with others who were involved in the survey to think about what to do next, and when. Set deadlines to act on what you’ve found, so it doesn’t fall down the list.
- Feedback to everyone. Make sure that those who took part know how it went – and any changes you’ve made as a result.
- Improve your method. Take five minutes to talk about how the survey was carried out and what you would do differently next time. This is important so that what you’ve learned isn’t lost.
- Share where you can. Think about if there is anyone else who might like to see the anonymised results/report that comes from the work. Data is an important product these days and being open (as long as it’s safe to do so) can improve relationships in the future.
It might not always be possible to do everything listed here all the time. However, finding ways to improve your survey work can have real and worthwhile benefits: reaching more people, getting better answers, and saving time by keeping focused. We’ve seen the results of this within our own organisation. As a project focused on learning and research, it’s important that we get the right information for ourselves, and to inform how we change systems. Try putting some of these ideas into your work locally, to banish the rushed survey questions forever!
References & further reading
Bryman, A. (2015). Social research methods. Oxford University Press. pp 231-267
Designing a Survey (January 19th, 2018). Retrieved from https://www.sciencebuddies.org/science-fair-projects/references/how-to-design-a-survey
Harrison, Chase. Harvard University Programme on Survey Research: Tip Sheet on Question Wording. (2007, November 17th) Retrieved from https://psr.iq.harvard.edu/files/psr/files/PSRQuestionnaireTipSheet_0.pdf
How to write in plain English. (2018, January 19th). Retrieved from http://www.plainenglish.co.uk/files/howto.pdf
Jha, Gaurav. Eight steps to designing an ideal survey. (2018, January 19th) Retrieved from https://tinyurl.com/y9a9z8nq
Rea, L. M., & Parker, R. A. (2014). Designing and conducting survey research: A comprehensive guide. John Wiley & Sons.
Thomas, G. (2013). How to do your research project: A guide for students in education and applied social sciences. Sage. pp 207-17