I love surveys!
I think the main reason why surveys are great is that they allow you to discover the answer to almost any burning question your organization might have about its donors/members. Furthermore, there’s never any shortages in people willing to take surveys and provide you with the answers you seek. You do have to be careful when crafting your questions though. If you ask poorly crafted questions, you get bad answers!
I want to briefly discuss a few different types of bad survey questions in this post. My hope is that you read them, learn why they aren’t great, and avoid them in future surveys.
Bad Question Types
Double Barrelled Questions
This refers to the practice of asking two questions and expecting only one answer. Have a look at the example below (I really wish this were a made up question, but I took it off someone else’s work!).
As you consider your giving and volunteering, is there a common thread or are there shared values among them?
- Yes
- No
- Somewhat
I won’t even ask if there’s a clear difference between ‘common thread’ and ‘shared values’ here, but when someone answers ‘Yes’ to this question, what are they answering? Yes, there’s a common thread? Yes, there are shared values? Yes to both? I have no clue.
Leading Questions
This is my most hated bad survey question type. When you want to ask a question and you have a desired answer in mind, you bake that answer in to the wording of the question itself! People generally will respond to questions with whatever the socially desirable answer is, and you’ll only ensure that happens by asking a leading question.
An infamous example of this dates back to 2016 when the federal Liberal party in Canada crafted a survey to gauge the appetite of Canadians for electoral reform. They had just won a sweeping electoral victory, winning a majority government through the very voting system (first past the post) that they had repeatedly promised to reform. I strongly believe that this gave them second thoughts about electoral reform. When they turned to their public consultations, these second thoughts would have figured pretty strongly in the questions they asked Canadians.
Have a look at the below trio of question examples from this survey (all had a likert scale response range, from ‘Strongly Disagree’, to ‘Strongly Agree’):
There should be parties in Parliament that represent the views of all Canadians, even if some are radical or extreme.
Voters should be able to express multiple preferences on the ballot, even if this means that it takes longer to count the ballots and announce the election result.
It is better for several parties to have to govern together than for one party to make all the decisions in government, even if it takes longer for government to get things done.
In each of these questions, the research agency working for the federal Liberals baked in a stern warning about proportional representation right after the thing they wanted to ask about. Instead of just a simple assessment of peoples’ thoughts, you also get 3 warnings: We’ll invite extremism into parliament, counting ballots will take longer, government will be more inefficient.
No wonder they concluded that it wasn’t the time for electoral reform!
Loaded Questions
This question type contains a hidden assumption that something must be true, even if that assumption just isn’t the case. Have a look at the below very innocent looking questions:
Which of our programs is most appealing to you?
What updates are you most excited to hear about from us?
It does sound quaint to be asking about which programs are the most ‘appealing’ and which updates the person’s ‘most excited’ by. What if the donor doesn’t find any programs appealing, and none of the updates excite her? In that case, the organization is going to create some sour grapes from the donor, either closing her up to more honest answers, or possibly even making her more likely to lash out at the organization.
After all, when was the last time you appreciated false assumptions people tossed your way?
Overstated Questions
You might feel really excited by all the progress your organization has made in addressing its cause, but don’t put too much excitement into the wording of your questions! An overstated question puts too much emphasis on some aspect of the question wording, robbing you of the opportunity to get a ‘shades of gray’ response. Have a look at the below question which I made up:
Do you feel that your gift to our org has made an enormous difference to the cause? (Strongly Disagree – Strongly Agree)
It’s all well and good to ask a donor if they feel their gift is making a difference, but as soon as I put the word ‘enormous’ in there, it creates a huge filter! Of the people who respond in the ‘Disagree’ range, you’re going to find people who think their gift is making somewhat of a difference, and maybe also a moderate difference, alongside the people who believe their gift is not making a difference.
I think in this case it would be better for the question to ask to what extent people believe their gift is making a difference. Then the answers won’t be so polarized!
Conclusion
I hope this has been fun and informative. Let me know if you’ve come across any questions that you’ve found particularly shocking or disappointing. If you want help crafting a donor/member survey for your organization, feel free to contact me and I’ll be happy to help.