Article Preview
Top1. Introduction
As more and more people worldwide depend on the Internet to fulfill their information needs (Khatwani & Srivastava, 2017), and as the impact of Internet on people and societies have become increasingly profound (Teo, 2007; Lane et al., 2017), researchers throughout the world have studied factors maximizing successes of information technology implementations or global information management (Roztocki & Weistroffer, 2011, Lee et al., 2014; Caprio et al., 2015; Hung et al., 2016; Silic & Back, 2016; Soja, 2016; Chatterjee et al., 2017). One such technological implementation is the employment of search engines. Because of the critical role search engines play in bridging Internet information resources and information users, it is particularly important to evaluate effectiveness of search engines through effective measurements of their search results, as different search engines utilize different retrieval and ranking algorithms and therefore respond to search queries with different search results.
Average Internet searchers tend to take the search results presented by the search engines as a list of decreasing relevance, and they tend to browse only the first 20-30 items on a results list from a search engine. Moreover, business intelligence systems also seem to base many of their decisions on search results as returned by Internet search engines. If the most relevant results are not properly positioned on the result list, important information would be missed, and the decisions could be impaired. Therefore, precise relevance ranking of search result items as returned by search engines is extremely important.
However, because what resides on the Web is an ever-changing and extremely heterogeneous data collection (Jansen & Pooch, 2001), Web page ranking algorithms have become very complicated and dynamic (Dean 2016; Barysevich 2017). It is important to know that ranking algorithms of different search engines handle variables differently. Consequently, the degree of search result relevance varies from search engine to search engine. Ideally, if all returned items are ranked in terms of relevance to the search query, and the ranked data are captured in a two-dimensional chart where the X-axis represents the ranked items and the Y-axis represents the relevance score, then a decline curve appears. Understanding the downward curve is critical to evaluating the quality of search results because the downward curve serves as a yardstick in measuring relevance of search results of a search engine.
The primary purpose of this study is to explore effective measurement of search results from search engines through investigating relevance-decrease patterns of search results from two major search engines: Google and Bing. To accomplish the purpose, 4 domain categories were defined, and 24 search queries with 6 from each category were formulated and submitted to both Google and Bing. Retrieved results were then collected, and their relevance was judged by 32 subjects independently. A group of possible regression models were developed for regression analysis, and the performances of the regression models were tested. The best-fit regression model was identified through ANOVA analyses. The findings of this study help people better understand the relevance-decrease patterns of search results produced by search engines. The best-fit regression model identified in this study provides a way for people to evaluate search result relevance of search engines.