Consumer decision support systems (CDSSs) help online users make purchasing decisions in e-commerce Web sites. To more effectively compare the usefulness of the various functionalities and interface features of such systems, we have developed a simulation environment for decision tasks of any scale and structure. Furthermore, we have identified three criteria in an evaluation framework for assessing the quality of such CDSSs: users’ cognitive effort, preference expression effort, and decision accuracy. A set of experiments carried out in such simulation environments showed that most CDSSs employed in current e-commerce Web sites are suboptimal. On the other hand, a hybrid decision strategy based on four existing ones was found to be more effective. The interface improvements based on the new strategy correspond to some of the advanced tools already developed in the research field. This result is therefore consistent with our earlier work on evaluating CDSSs with real users. That is, some advanced tools do produce more accurate decisions while requiring a comparable amount of user effort. However, the simulation environment will enable us to efficiently compare more advanced tools among themselves, and indicate further opportunities for functionality and interface improvements.
TopIntroduction
With the rising prosperity of the World Wide Web (WWW), consumers are dealing with an increasingly large amount of product and service information that is far beyond any individual’s cognitive effort to process. In early e-commerce practice, online intermediaries were created. With the help of these virtual storefronts, users were able to find product information on a single Web site that gathers product information from thousands of merchants and service suppliers. Examples include shopping.yahoo.com, froogle.com, shopping. com, cars.com, pricegrabber.com, and so forth. However, due to the increasing popularity of electronic commerce, the amount of online retailers proliferated. As a result, there are now easily millions (or 16-20 categories) of brand-name products available on a single online intermediary Web site. Finding something is once again difficult, even with the help of various commercially available search tools.1 Recently, much attention in e-commerce research has focused on designing and developing more advanced search and product recommender tools (Burke, Hammond, & Young, 1997; Pu & Faltings, 2000; Reilly, McCarthy, McGinty, & Smyth, 2004; Shearin & Lieberman, 2001; Shimazu, 2001; Stolze, 1999). However, they have been not employed in large scales in practicing e-commerce Web sites. Pu and Kumar (2004) gave some reasons as to why this is the case and when such advanced tools are expected to be adopted. This work was based on empirical studies of how users interact with product search tools, providing a good direction as to how to establish the true benefits of these advanced tools. However, insights gained from this work are limited. This is mainly due to the lack of a large amount of real users for the needed user studies and the high cost of user studies, even if real users were found. Each of the experiments reported in Pu and Kumar (2004) and Pu and Chen (2005) took more than 3 months of work, including the design and preparation of the study, the pilot study, and the empirical study itself. After the work was finished, it remains unclear whether a small amount of users recruited in an academic institution can forecast the behavior of the actual user population, which is highly diverse and complex.
Our main objective in this research is to use a simulation environment to evaluate various search tools in terms of interaction behaviors: what users’ effort would be to use these tools and what kind of benefits they are likely to receive from these tools. We base our work on some earlier work (Payne, Bettman, & Johnson, 1993) in terms of the design of the simulation environment. However, we have added important elements to adapt such environments to online e-commerce and consumer decision support scenarios. With this simulation environment, we hope to more accurately forecast the acceptance of research tools in the real world, and curtail the evaluation of each tool’s performance from months of user study to hours of simulation and a week of fine tuning the simulation results against a small but diverse amount of real users. This should allow us to evaluate more tools and, more importantly, discover design opportunities of new tools.
Our initial work of measuring the performance of various decision support strategies in e-commerce environments was reported in a conference paper (Zhang & Pu, 2005). The current article is an extended version of the conference paper. Besides adding significantly more details on the work already reported, there are a number of important and new contributions: