Our Blog

What we're talking about

Survey Design: Creating the optimal panellist experience

by Adam Whibley.

After reading last week’s blog post by Pat Prunskus, the first thing I thought was, “I couldn’t agree more.” In my first few months here at AskingCanadians I was so adamant about making the surveys more engaging I asked one of the programmers to do the following mock-up of our profiler “pets” question:

At the time I neglected to recognize the fact that I’m the Online Panel Data Specialist, not a programmer, but I’m now getting the opportunity to voice my thoughts on the subject regardless. Since Pat has already done a great job of emphasizing the importance of appearances I’ll give up my dream of creating survey questions with cute little animal pictures, and instead cover a few other issues related to survey design that I have come across during my time here at AC.

Not too long ago AskingCanadians ran an omnibus study to assess customer satisfaction. One of the questions was, “What did you like the least about the survey experience?” The results were what you might expect.

The first matter brought to our attention was in regard to repetitive / redundant survey questions (12 per cent). One of our panellists put it quite eloquently, “I hate it when they are very long and tedious and ask dumb questions over and over.”

I can see where this individual is coming from. There are often massive grid questions that, while perhaps necessary, are not exactly enjoyable to fill out. For example: “How often (five point scale) do you do the following (25!) activities: on your home computer and on your mobile phone and on your tablet.” That’s 75 responses for essentially one question.

As I said, there may be a perfectly justifiable reason for asking these questions, but we should always evaluate (and re-evaluate) the reasons why we’re including each question when designing or reproducing a survey. Statements such as, “That’s how we’ve always done it…” should be a thing of the past.

The second biggest concern panellists brought to our attention were the lengthy screen-out and quota- full processes (17 per cent). While not everyone is the right fit for a particular study, we should do what we can to mitigate long screen-out practices. What’s the harm in moving the initial screening questions to the beginning of the survey? An argument against this is instances where demographics are tracked for those who drop out as well as those who participate. However, those designing the questionnaire should ask themselves if this is actually necessary for the study in question, prior to making someone answer 10 grid questions before getting the boot.

The final and most problematic matter was survey length (30 per cent). While there are valid reasons for producing long surveys, the costs may outweigh the benefits. A handful of issues I see arising are the number of people who don’t even start a survey, (I know when I see “Take this 45 minute survey!” I run the other way), dropout rates, and data integrity. As I am writing this I decided to run some data and assess dropout rates:

I think this illustrates my point nicely – the longer the survey the more likely a respondent will drop out, and frankly, shorter surveys are better for everyone. The cost-per-interview is reduced, the data has more integrity, and the panellist experience is better. As surveys get longer people start straight-lining and providing gibberish answers. (AC does monitor for these sorts of problems.)

As I said, I’m a huge advocate of visually-appealing, engaging surveys, but I feel the emphasis should be first placed on the questionnaire design stage. Removing redundant questions means the panellist experience for those who do fill out surveys is now that much better. Even those who are screened out win – a faster screening process alongside initiatives such as rerouting panellists to a new survey means they leave happy as well.

Anyway, regardless of any for or against arguments about the things I’ve said, take a moment and put yourself in a panellist’s shoes…do that aforementioned 45-minute survey. Then come back and we can sit down and discuss the merits of my case.