The main research objective of the chapter is to provide an analysis of the technological quality evaluation models and make a proposal for a method suitable for the multiple criteria evaluation (decision making) and optimization of the components of e-learning systems (i.e. learning software), including Learning Objects, Learning Object Repositories, and Virtual Learning Environments. Both the learning software ‘internal quality’ and ‘quality in use’ technological evaluation criteria are analyzed in the chapter and are incorporated into comprehensive quality evaluation models. The learning software quality evaluation criteria are further investigated in terms of their optimal parameters, and an additive utility function based on experts’ judgements, including multicriteria evaluation, numerical ratings, and weights, is applied to optimize the learning software according to particular learners’ needs.
Top1. Introduction: The Problem Of Multiple Criteria Evaluation And Optimization Of Learning Software
The problem of evaluation and optimization of the technological quality of e-learning systems components, i.e. Learning Objects (LOs), Learning Object Repositories (LORs) and Virtual Learning Environments (VLEs), is high on the agenda of the European research and education system.
Evaluation can be characterized as the process by which people make judgements about value and worth. However, in the context of learning technologies, this judgement process is complex and often controversial. Although the notion of evaluation is rooted in a relatively simple concept, the process of judging the value of learning technology is complex and challenging (Oliver, 2000).
Different scientific methods are used for evaluating the quality of e-learning systems components (i.e. learning software). The chapter is aimed to consider the problems of expert evaluation of the technological quality of LOs, LORs and VLEs.
The basic notions, principles and methods applied in the Chapter are as follows: LO is referred to any digital resource that can be reused to support learning (Wiley, 2000). LORs are considered here as properly constituted systems (i.e. organized LOs collections) consisting of LOs, their metadata and tools / services to manage them (Kurilovas, 2007). Metadata is referred to structured data about data (Duval et al., 2002). VLEs are considered as specific information systems which provide a possibility to create and use different learning scenarios and methods.
Quality evaluation is defined as “the systematic examination of the extent to which an entity (part, product, service or organization) is capable of meeting specified requirements” (ISO/IEC 14598-1:1999).
Multiple criteria evaluation method is referred here as the experts’ additive utility function (e.g. Equation 3 in section “Experts’ Additive Utility Function”), including the learning software evaluation criteria, their ratings (values) and weights.
Expert evaluation is referred to a multiple criteria evaluation of the learning software that is aimed at the selection of the best alternative based on score-ranking results. According to Dzemyda & Saltenis (1994), if a set of decision alternatives is assumed to be predefined, fixed and finite, then the decision problem can be formulated as a task of finding the optimal alternative or ranking the various alternatives. In practice, usually experts (decision makers) have to deal with the problem of making the optimal decision in the multiple criteria situation where the objectives are often conflicting. In this case, according to Dzemyda and Saltenis (1994), “an optimal decision is the one that maximizes the decision maker’s utility”.
The authors of the Chapter apply here one the software engineering principles which claims that one should evaluate the software using two different groups/types of evaluation criteria: ‘internal quality’ and ‘quality in use’. According to Gasperovic and Caplinskas (2006), ‘internal quality’ is a descriptive characteristic that describes the quality of software irrespective of any particular context of use, and ‘quality in use’ is an evaluative characteristic of software obtained by making a judgment based on criteria that determine the worthiness of software for a particular project or user/group. According to Gasperovic and Caplinskas (2006), it is impossible to evaluate the quality in use without knowing the characteristics of internal quality.
The rest of the chapter is organized as follows. The next section presents the literature review and a short analysis of the existing technological evaluation models (i.e., sets of evaluation criteria) and methods for evaluation of LOs, LORs and VLEs. Then, multiple criteria evaluation and optimization of learning software for the particular learner needs are described. The fourth section offers further research trends whist conclusions are provided in the fifth section.
Top2. Literature Review: Existing Well-Known Technological Evaluation Models And Methods For Learning Software
First of all, let us review and briefly analyze the literature on existing well known evaluation models/tools (i.e. sets of evaluation criteria) and evaluation methods of LOs, LORs and VLEs.