MCEQLS Approach in Multi-Criteria Evaluation of Quality of Learning Repositories

MCEQLS Approach in Multi-Criteria Evaluation of Quality of Learning Repositories

Eugenijus Kurilovas (Vilnius University, Lithuania & Vilnius Gediminas Technical University, Lithuania)
DOI: 10.4018/978-1-4666-2669-0.ch006


This chapter analyzes the quality of XML learning object repositories. Special attention is paid to the models and methods to evaluate the quality of learning repositories. Multiple criteria decision analysis and optimization methods are explored to be applied for evaluating the quality of learning repositories. This chapter also presents the results of several large-scale projects co-funded by EU research programs that have been implemented in the area of learning repositories. Learning repositories’ technological quality model (system of criteria) and novel comprehensive model for evaluating the quality of user interfaces of learning repositories are presented in more detail. The general MCEQLS (Multiple Criteria Evaluation of Learning Software) approach is presented in this chapter. It is shown that the MCEQLS approach is suitable for evaluating the quality of learning repositories. The author believes that research results presented in the chapter will be useful for all educational stakeholder groups interested in developing learning repositories.
Chapter Preview

1. Introduction

LO Repositories (LORs), or learning repositories, are considered here as properly constituted systems (i.e. organised learning object collections) consisting of learning objects, their metadata and tools / services to manage them (Kurilovas, 2009c). Learning Object (LO) is referred here as “any digital resource that can be reused to support learning” (Wiley, 2000).

Authorities and/or companies in many countries have launched Web-based learning repositories in order to make it easier for teachers and students to find the best relevant LOs. The variety in the LORs across Europe reflects this situation: they are all Web based. However, some repositories are merely catalogues of LOs. They only contain descriptions of the learning content.

There has been a number of large scale LORs related EU-funded projects implemented in Europe during last few years. The largest of them are FP6 CALIBRATE, eContentplus ASPECT and eContentplus EdReNe projects. The author of this chapter acted as a manager of a Lithuanian team is all of those projects.

CALIBRATE and ASPECT projects were aimed at creation and development of European Learning Resource Exchange (LRE, 2011) system.

CALIBRATE (Calibrating eLearning in Schools) project (2005 – 2008) led to the design and implementation of an open source brokerage system that relies on open standards and open content to promote the exchange of LOs within a federation of e-learning systems. Its role is limited to carrying and routing messages exchanged by federation members rather than to facilitate semantic interoperability. With the system, semantic interoperability becomes the responsibility of the federation members that rely on “clients” to communicate with the brokerage system and to support the negotiation of common query languages and metadata formats. The system itself adopts a service-oriented architecture so that each service (e.g., LO discovery, digital rights management) can be used separately and combined with any (group) of the others. Current services include connection management, federated searching, and metadata harvesting.

CALIBRATE has produced tools to help developers to connect systems (e.g., repositories, search interfaces) to the federation. Tools include a test instance of the federation, a test repository, a test search interface, a “query watcher,” a tool for monitoring connections, and a connection toolkit. CALIBRATE developed an Agent-Based Search System (ABSS) to help ranking search results according to the profiles of users. This was implemented in the CALIBRATE portal/federation as follows: (1) the ABSS collects LRE metadata using the LRE harvesting service, (2) the CALIBRATE portal provides the ABSS with user profiles and Contextual Attention Metadata (CAM) using a SQI (Sequential Inquiry Interface) service, and (3) the ABSS is queried using the SQI service.

The ASPECT (Adopting Standards and Specifications for Educational Content) Best Practice Network (2008 – 2011) involved project partners and teachers using a version of the LRE service that enables schools to find open educational content from many different countries and providers. In ASPECT, a customised and password protected version of the LRE was developed for schools in the project that contained LOs from commercial providers and some additional search and retrieval features related to the exploration of the standards under investigation in the project.

In ASPECT, content providers from both the public and private sectors applied content standards to their LOs and made them available via the LRE. This represents a large-scale implementation of standards and specifications for content discovery and use.

As a result, the following elements, not directly linked to the quality of the specifications themselves, were identified as necessary for the successful adoption of the content standards:

  • Availability and quality of tools for producing compliant content and metadata.

  • Availability and quality of tools to test for compliance.

  • Availability of solutions adapted to the different needs of content providers from both the public and private sectors.

CALIBRATE and ASPECT research results are analysed in detail by Kurilovas (2012).

Complete Chapter List

Search this Book: