A Characterisation and Framework for User-Centric Factors in Evaluation Methods for Recommender Systems

A Characterisation and Framework for User-Centric Factors in Evaluation Methods for Recommender Systems

Tatenda D. Kavu (University of Zimbabwe, Harare, Zimbabwe), Kuda Dube (Massey University, Auckland, New Zealand), Peter G. Raeth (University of Zimbabwe, Harare, Zimbabwe) and Gilford T. Hapanyengwi (University of Zimbabwe, Harare, Zimbabwe)
DOI: 10.4018/IJICTRAME.2017010101


Researchers have worked on-finding e-commerce recommender systems evaluation methods that contribute to an optimal solution. However, existing evaluations methods lack the assessment of user-centric factors such as buying decisions, user experience and user interactions resulting in less than optimum recommender systems. This paper investigates the problem of adequacy of recommender systems evaluation methods in relation to user-centric factors. Published work has revealed limitations of existing evaluation methods in terms of evaluating user satisfaction. This paper characterizes user-centric evaluation factors and then propose a user-centric evaluation conceptual framework to identify and expose a gap within literature. The researchers used an integrative review approach to formulate both the characterization and the conceptual framework for investigation. The results reveal a need to come up with a holistic evaluation framework that combines system-centric and user-centric evaluation methods as well as formulating computational user-centric evaluation methods. The conclusion reached is that, evaluation methods for e-commerce recommender systems lack full assessment of vital factors such as: user interaction, user experience and purchase decisions. A full consideration of these factors during evaluation will give birth to new types of recommender systems that predict user preferences using user decision-making process profiles, and that will enhance user experience and increase revenue in the long run.
Article Preview

1. Introduction

Evaluation of recommender systems reveal key issues that demand attention, especially when business enterprises target methods to stay relevant in business. Designers of recommenders have been mainly engaged in evaluating system-centric factors neglecting user-centric factors which keep customers around for more profit (Leino, 2014; Lin, Wang, Wang, & Lu, 2013; Knijnenburg, 2012). Inadequate evaluation of user-centric factors such as buying decisions, user experience and user interactions can lead into endorsement of inferior recommender systems. This can also create a routine of designing inferior recommenders which lacks the quality and focus required to increase revenue and user experience.

This paper investigates the adequacy, or lack of it, in recommender systems evaluation methods. Recommender systems evaluation methods mainly comprises of user-centric evaluation and system-centric evaluation. It is of paramount importance that these evaluation methods are adequately assessed when evaluating a recommender system. Failure to assess user-centric factors gives rise to neglecting important factors that cause rejection of a recommender system (Knijnenburg, 2012). Published work has shown that there is little correlation between algorithmic accuracy and user satisfaction. User-centric evaluation methods have not been employed to the same degree as system-centric approaches (Leino, 2014; Cremonesi, Garzotto, & Turrin, 2013; Konstan & Riedl, 2012; Knijnenburg, 2012). It is thus imperative to expand from recommender algorithms, to user interactions, decision-making processes and overall experience which are the essence of user-centricity (Tintarev et al., 2014). The main aim of the research presented is to use a proposed user-centric evaluation conceptual framework to prove through literature review that current evaluation methods of recommender systems lack the adequacy needed to fully assess user-centric factors, resulting in low levels of user interaction and experience. This will provide a basis for further research into the development of algorithms and models that can address this problem.

An integrative review approach was used which draws conclusions from related articles and report on current state of affairs on user-centricity through a characterization and a conceptual framework which makes it a primary research . Terms were elicited from the research problem a search was conducted from August 2014 to December 2015, specifically on articles published between January 2009 to December 2015. For inclusion criteria, all types of recommender systems ranging from eLearning, to news and social networking were included whereas for exclusion criteria, only evaluations of recommender systems were included. Results from this paper show that there are more system-centric than user-centric methods and the existing user-centric evaluation methods are not assessing buying decisions, user experience and user interactions. Even if new developed recommender algorithms have high precision in terms of prediction, they will face resistance if they do not address the above mentioned factors. User-centric methods lack practical approaches, which make them difficult to implement in a computational environment.

Complete Article List

Search this Journal:
Open Access Articles: Forthcoming
Volume 9: 2 Issues (2020): Forthcoming, Available for Pre-Order
Volume 8: 2 Issues (2019)
Volume 7: 2 Issues (2018)
Volume 6: 2 Issues (2017)
Volume 5: 2 Issues (2016)
View Complete Journal Contents Listing