Current and prospective students (n =133) were surveyed about their preferences for a name for a new online series of courses to be launched by a university. Preferences for each of five names were solicited by means of analytical ratings, holistic ratings, and rankings. All three techniques were employed to assure that the most appropriate name for the program was selected, but this also afforded us the opportunity to study several theoretical issues: (a) Do the different methods lead to discrepant decisions at the aggregate level? (b) Is the holistic rating or the analytical rating approach more closely related to the rankings? (c) To what extent is lack of agreement between ratings and rankings due to lack of differentiation in ratings? The authors find that at the aggregate level all three methods suggest the same name for the program; the holistic rating is slightly more highly correlated with the ranking; and the lack of differentiation in ratings is one reason producing inconsistencies between ratings and rankings.
TopAdvantages And Disadvantages Of Ranking Versus Rating
Inappropriately, the terms rate and rank are sometimes used interchangeably as if they were synonymous, disregarding a fundamental difference. That is, a rating requires one to assign a value to a stimulus using a common scale, whereas a ranking asks one to compare different objects directly to one another by arranging them in some order with respect to some attribute (such as importance, agreement, quality or preference, etc.). Paulhus (1991) identified three types of potential response biases with rating scales: social desirability bias, acquiescence bias, and extreme response bias (i.e., stringency and leniency). The chief virtue of ranking is that the procedure prevents the respondent from failing to differentiate between stimuli due to response styles bias such as acquiescence or extreme response (Baumgartner & Steenkamp, 2001; Berkowitz & Wolkon, 1964; Douceur, 2009; Harzing et al., 2009; Shuman & Presser, 1981; Toner, 1987), but the drawback is that it may force the respondent to artificially differentiate between items that may in fact be viewed as equivalent. Likewise, ranking does not allow for determination of the degree of difference between the objects being compared. Ranking is also a more time-consuming procedure; on average, it takes three times longer to answer a ranking than a rating question (Munson & McIntyre, 1979), although it is argued that the process thereby produces better quality data. According to a review by Krosnick (1999), the improvement in data quality occurs because ranking demands a greater degree of attention and respondents thereby make fewer mistakes when using this answer format.
Overall, Krosnick considers ranks to generally be more reliable and have higher validity with criterion measures in a variety of contexts. Comparisons of the merits of absolute performance appraisals (various rating formats) and relative (various ranking formats) have been the focus of much research in industrial psychology. Generally, relative formats are more valid measures of actual job performance when a “hard” criterion exists, such as sales volume (Goffin et al., 1996; Heneman, 1986; Nathan & Alexander, 1988). Moreover, Hartzig et al. (2009) found rankings to superior over ratings in cross-cultural studies. O'Mahony, Garske, and Klapman (1980) used a signal detection index of difference to determine whether rating or ranking is preferable for identifying differences in food flavors, and report that ranking is superior.