Article Preview
TopAdvantages And Disadvantages Of Ranking Versus Rating
Inappropriately, the terms rate and rank are sometimes used interchangeably as if they were synonymous, disregarding a fundamental difference. That is, a rating requires one to assign a value to a stimulus using a common scale, whereas a ranking asks one to compare different objects directly to one another by arranging them in some order with respect to some attribute (such as importance, agreement, quality or preference, etc.). Paulhus (1991) identified three types of potential response biases with rating scales: social desirability bias, acquiescence bias, and extreme response bias (i.e., stringency and leniency). The chief virtue of ranking is that the procedure prevents the respondent from failing to differentiate between stimuli due to response styles bias such as acquiescence or extreme response (Baumgartner & Steenkamp, 2001; Berkowitz & Wolkon, 1964; Douceur, 2009; Harzing et al., 2009; Shuman & Presser, 1981; Toner, 1987), but the drawback is that it may force the respondent to artificially differentiate between items that may in fact be viewed as equivalent. Likewise, ranking does not allow for determination of the degree of difference between the objects being compared. Ranking is also a more time-consuming procedure; on average, it takes three times longer to answer a ranking than a rating question (Munson & McIntyre, 1979), although it is argued that the process thereby produces better quality data. According to a review by Krosnick (1999), the improvement in data quality occurs because ranking demands a greater degree of attention and respondents thereby make fewer mistakes when using this answer format.
Overall, Krosnick considers ranks to generally be more reliable and have higher validity with criterion measures in a variety of contexts. Comparisons of the merits of absolute performance appraisals (various rating formats) and relative (various ranking formats) have been the focus of much research in industrial psychology. Generally, relative formats are more valid measures of actual job performance when a “hard” criterion exists, such as sales volume (Goffin et al., 1996; Heneman, 1986; Nathan & Alexander, 1988). Moreover, Hartzig et al. (2009) found rankings to superior over ratings in cross-cultural studies. O'Mahony, Garske, and Klapman (1980) used a signal detection index of difference to determine whether rating or ranking is preferable for identifying differences in food flavors, and report that ranking is superior.