What Are Ranking vs Rating Questions?

Ranking questions ask respondents to put the items of a list in their order of importance. For example:

Rating questions ask respondents to assign a value or score to a particular item. For example:

What Are the Benefits and Drawbacks of Ranking Questions?

Ranking questions are generally used when it is crucial that each item on your satisfaction or opinion survey has a unique value. Ranking questions can help you decide which features of a product to focus more on, which staff interventions to put more money or effort into, or which type of blog post you should be writing.

Ranking questions also help eliminate the tendency to straightline – to assign the same rating to a number of variables in a survey, simply because the respondent can’t be bothered reading each variable. You’ll see this issue as a row of numbers in a straight line – all 10s, all 1s or all 5s, usually. However, this tendency can be negated using other good question design practices too.

Ranking questions have quite a few drawbacks, though! Remember that:
•    People may actually regard some of the elements they are asked to rank as equivalent, creating a false hierarchy
•    They take a long time to answer because people must hold each variable in their head, make an overall comparison, then record it. As you add more choices, ranking questions become disproportionately complex.
•    People are more likely to rank options listed first more highly than to try to hold all options in their mind at once.
•    A limitation with statistical analysis for ranking questions is that mean values shouldn’t be used, since there may be a false impression of ‘distance’ between two options

You’ll need to word your ranking questions very clearly as well. Make it clear whether you want the most positive response to be numbered with the highest, or lowest number. Also make clear that you want people to use numbers for the ranking!

What Are the Benefits and Drawbacks of Rating Questions?

Rating questions are more widely used than ranking questions for a reason. They are easy to understand, and they also allow people to differentiate between items on a list … but with the added information on exactly how much ‘distance’ is perceived between different options.

However, rating questions aren’t infallible. Use them carefully, keeping in mind that:
•    If your lists are too long, they’ll encourage answer ‘straightlining’.
•    People consider the highest and lowest ranks differently. For example, some people never assign a 10/10 in a survey, and some people never assign a 1/10. This can create correlations where in fact none exist.

The visual layout of your rating questions will also be important – don’t put too many in a block, and make sure you leave sufficient space between lines so that respondents don’t need to strain their eyes to see where they should be marking their answers. Consider carefully whether you really need a 10-point scale, too – it is more difficult for people to trace across a longer line of possible ratings than a shorter line. A 5 point likert scale (eg Very Satisfied, Satisfied, Neutral, Dissatisfied, Very Dissatisfied) is considered typical. However, if you want to force respondents to either a positive or negative side and not allow for neutrality, consider using a 6 point scale which options on either side of neutral (eg Very Satisfied, Satisfied, Somewhat Satisfied, Somewhat Dissatisfied, Dissatisfied, Very Dissatisfied).

More importantly, don’t make every question on your survey a ranking question or a rating question! Open-ended questions are important too, and you’ll want to assign arbitrary categories for some questions. Variety is the spice of a survey!

Getting the right advice when designing your survey is crucial to its success, and a PeoplePulse project manager is able to guide you from the development of your survey content through to implementation and reporting. Contact us today to find out how we can help you with your survey needs.