Excessive Survey Length

When it comes to optimal survey design, one of the common problems we observe at PeoplePulse is clients wanting to run excessively long surveys. We have had one client, for example, that could not be talked out of running a company-wide staff survey that was over 100 pages long (each page averaged 6 questions). Whilst this example may sound extreme, it is not uncommon for clients to send us draft questionnaires that are 35-45 pages in length. So let’s be clear up front about our attitude towards excessively long surveys …

They don’t work!

Why? Here’s a few reasons:

• Poor survey completion rates. We often see five poorly worded questions about a topic when one well worded question would have sufficed. When this occurs your survey can become too long to complete which in turn results in higher drop off rates.
• People are time poor. Who has 45 minutes free these days during working hours to complete your survey? Any longer than 10 minutes to fill out your survey and you’re heading towards the ‘non-completion’ danger zone.
• Suspect motivation to complete. If people do in fact complete excessively long surveys (say 100+ questions), it will usually be because you’ve enticed them with incentives for completion. However when the majority of your respondents have completed your survey for the primary purpose of obtaining a reward, this calls into question the validity of the data you’ve collected. Your ability to make sound strategic decisions based on the findings is limited as the results may not be truly representative of all members of your target audience.
• Questionable data quality. If you know you have to complete 35+ pages of questions, it is very tempting to quickly tick through each response without paying adequate attention to the question being asked. Ohio State University researcher Jon A Krosnick refers to a concept called ‘Satisficing’ in which survey respondents to long questionnaires:
  • Choose the first response alternative that seems to constitute a reasonable answer.
  • Select Don’t know instead of reporting an opinion.
  • Randomly choose among the response alternatives offered (especially prevalent when many of the survey’s questions are mandatory).
  • The longer your questionnaire is, the greater the likelihood that ‘Satisficing’ occurs as respondent’s attention spans begin to wane.
• Jeopardises future surveys. Building trust with your respondents is a huge issue when it comes to surveys (trust that you’ll treat my time with respect, trust that you’ll keep my responses confidential, trust that you’ll act on the results, etc). So if Jeopardiseyou abuse that trust once by running an excessively long survey and not respecting a respondent’s time, then those same respondents are unlikely to repay you by completing your surveys with quality responses in the future.

What can you do? Eight tips …

If we agree that excessively long surveys are to be avoided at all costs, what measures can you take to help avoid excessive questionnaire length issues for future survey projects?

1. Plan your survey objectives before you commence writing your questions.Make sure you keep it tight; 3 to 4 objectives are sufficient, but any more may mean that your questionnaire size starts to blow out.

2. Make sure each and every question fights to keep its place. If you want to probe the effectiveness of your reception area, do you really need to ask 5-6 separate questions – or could you ask one well thought out question – eg:

Thinking about your experience with our reception area upon your arrival to our office, how would you rate our performance? 5 = Exceptional …..

It may surprise you to know that it’s often harder to write one good question about a topic than writing 4-5 related questions about the same theme. But by sticking to a smaller group of ‘key’ questions, not only will your respondents appreciate the faster survey but when you start analysing your results it’s typically easier to decipher people’s opinions on a topic when one well worded question has been asked. So for each question you include ask yourself how knowing the answer will help you. If it wouldn’t, then remove the question.

3. Avoid questionnaire design by committee. We often see committees of 4-5 people that have been nominated to develop the questionnaire. Another common occurrence is one person writes the survey, and then sends it to 10-15 managers for comment. The dangers in this approach are twofold: (a) you compromise to ensure that everybody is pleased, and (b) the more people that see the draft survey, the more questions they forward for inclusion.

4. Preload ‘known’ respondent data. A tool such as PeoplePulse allows you to preload data about a respondent (such as name, company size, gender, location, division, etc). If you already have this data about your respondents, then why not preload it into your reports so the respondent doesn’t have to complete it within the survey? The benefit is consistent reporting of demographic and company data, and shorter questionnaire lengths.

5. Utilise skip logic / branching. Utilise PeoplePulse’s branching capabilities to ensure you are screening out respondents that you are not interested in hearing from and only displaying relevant questions to the people of interest to you. In other words, don’t make a respondent see sections 3 and 4 if those sections are not relevant to them.

6. Complete the survey as if you are the respondent before putting it live. Once you’ve completed writing your questions, take your ‘company’ hat off and put your ‘respondent’ hat on to ensure that each question makes sense and deserves to be there. If you suspect the survey is too long and find your attention waning, then in all likelihood your respondents will feel the same way too.

7. Be upfront about how much time is required. Time how long it takes to complete your survey at a sensible pace, and let people know up front in your e-mail invite and/or introduction page. If your survey takes 15 minutes to complete respondents would rather know and plan for that upfront than be blindsided a few minutes into the survey. While this tactic doesn’t shorten your survey, it sets accurate expectations before the survey is commenced which will help improve your completion rates.

8. One survey or two? If your survey is on the longer side, ask yourself if it could best be broken into two smaller ‘manageable’ questionnaires. Another popular approach is to launch a quarterly ‘pulse’ survey that simply focuses on 1 key theme per survey per quarter. Often organising your research into short succinct surveys achieves better response rates than running one large survey that many respondents fail to complete. It can also help you focus your post-survey action plans.

In short, when you run shorter, focused surveys everyone wins – your completion rates are higher, your respondents are happier and more likely to complete future surveys you run, and your reports are easier to decipher and act on.

Previous issues in this series:

• Part One: Poorly Produced Survey Invites
• Part Two: Invite Timing Tips – What Time of the Day and the Day of the Week Works!

Next issue in this series

• Part Four: Scant Consideration Given During Questionnaire Design to Reporting

Also ahead in this series:

• Part Five: Optimising Your Post Survey Follow up