A Comparison of Retrieval Result Relevance Judgments Between American and Chinese Users

A Comparison of Retrieval Result Relevance Judgments Between American and Chinese Users

Jin Zhang (University of Wisconsin, Milwaukee, USA), Yuehua Zhao (Nanjing University, Nanjing, China), Xin Cai (University of Wisconsin, Milwaukee, USA), Taowen Le (Weber State University, Ogden, USA), Wei Fei (Suzhou Library, Suzhou, China) and Feicheng Ma (Wuhan University, Wuhan, China)
Copyright: © 2020 |Pages: 21
DOI: 10.4018/JGIM.2020070108
Article PDF Download
Open access articles are freely available for download


Relevance judgment plays an extremely significant role in information retrieval. This study investigates the differences between American users and Chinese users in relevance judgment during the information retrieval process. 384 sets of relevance scores with 50 scores in each set were collected from 16 American users and 16 Chinese users as they judged retrieval records from two major search engines based on 24 predefined search tasks from 4 domain categories. Statistical analyses reveal that there are significant differences between American assessors and Chinese assessors in relevance judgments. Significant gender differences also appear within both the American and the Chinese assessor groups. The study also revealed significant interactions among cultures, genders, and subject categories. These findings can enhance the understanding of cultural impact on information retrieval and can assist in the design of effective cross-language information retrieval systems.
Article Preview

1. Introduction

As people throughout the world continue their reliance on the Internet to fulfill their information needs (Khatwani & Srivastava, 2017) and as the Internet continues its profound impact on people and societies (Teo, 2007; Lane, et al., 2017), researchers have explored ways to maximize successes of Internet-based technology implementations or global information management (Roztocki & Weistroffer, 2011; Caprio, et al., 2015; Hung, et al., 2016; Silic & Back, 2016; Soja, 2016; Chatterjee, et al., 2017). Effective information retrieval from the Internet has remained an important topic in the field of global information management.

The key objective of information retrieval is to find relevant information. Therefore, the concept of relevance has been recognized as a fundamental issue to the evaluation of information retrieval systems as well as search engines (Borlund, 2003; Saracevic, 2007a; Zhang & Fei, 2010; Zhang et al., 2013). Relevance measures the effectiveness of a connection between a source and a target in a communication process (Saracevic, 1975).

Classic research on information retrieval concentrates on the techniques for the comparison of the representation of documents and search queries (Vakkari, 1999). Regarding the effectiveness of retrieval systems, traditional measures include precision and recall (Davidson, 1977). Typically, information retrieval systems treat all users the same way. However, more recent studies have regarded the information retrieval process as an interaction between users and the information retrieval system. From the user’s perspective, information retrieval systems should be designed to satisfy user information needs.

Some believe that information and culture cannot be partitioned since culture is composed of different and transmitted social information (Kim, 2013). Culture has been demonstrated as a key factor for the understanding of human behavior because culture could influence perceptions, opinions, and ultimately reactions to behavior. Cultural differences may shape information-processing procedures (Gutchess & Indeck, 2009), and cross-cultural differences may affect individuals’ information behaviors (Schwartz, et al, 2014). Some identified that information searching behavior differed between Chinese subjects and German subjects (Honold, 1999). Some explored differences in information seeking behaviors among software engineers from various countries and found that US and European subjects preferred non-social sources, while Indian and Pakistan participants favored social sources (Milewski, 2007). Therefore, understanding how cultural differences influence users’ relevance judgment is crucial to information-retrieval research.

The primary research objective of this study is to explore possible differences between American users and Chinese users in relevance judgment of retrieval results from search engines. To achieve this purpose, relevance judgments by two assessor groups of retrieval result in different domain categories were compared, interactions between the assessor groups and the domain categories were analyzed, and gender differences in relevance judgments were explored. The findings of this study can enhance understanding of cultural differences in information retrieval and can assist in the design of effective cross-language search engines.

Complete Article List

Search this Journal:
Volume 31: 1 Issue (2023): Forthcoming, Available for Pre-Order
Volume 30: 12 Issues (2022)
Volume 29: 6 Issues (2021)
Volume 28: 4 Issues (2020)
Volume 27: 4 Issues (2019)
Volume 26: 4 Issues (2018)
Volume 25: 4 Issues (2017)
Volume 24: 4 Issues (2016)
Volume 23: 4 Issues (2015)
Volume 22: 4 Issues (2014)
Volume 21: 4 Issues (2013)
Volume 20: 4 Issues (2012)
Volume 19: 4 Issues (2011)
Volume 18: 4 Issues (2010)
Volume 17: 4 Issues (2009)
Volume 16: 4 Issues (2008)
Volume 15: 4 Issues (2007)
Volume 14: 4 Issues (2006)
Volume 13: 4 Issues (2005)
Volume 12: 4 Issues (2004)
Volume 11: 4 Issues (2003)
Volume 10: 4 Issues (2002)
Volume 9: 4 Issues (2001)
Volume 8: 4 Issues (2000)
Volume 7: 4 Issues (1999)
Volume 6: 4 Issues (1998)
Volume 5: 4 Issues (1997)
Volume 4: 4 Issues (1996)
Volume 3: 4 Issues (1995)
Volume 2: 4 Issues (1994)
Volume 1: 4 Issues (1993)
View Complete Journal Contents Listing