Evaluations of Online Learning Activities Based on LMS Logs

Introduction

Learning management system (LMS) is a broad term that is used for a wide range of systems that organize and provide access to online learning services for students, teachers and administrators. These services usually include access control, provision of learning content, communication tools, and organizations of user groups (Paulsen, 2002). Jovanovic et al. (2007) defined an LMS as “a software environment that enables interactive web-based teaching and supports administration of distance courses, allowing instructors to distribute information to students, producing course material, preparing assignments and tests, engaging in discussions, and managing courses and distance classes” (p. 46).

In 2005, 95% of all higher-education institutions in the UK were using an LMS (Browne, Jenkins, & Walker, 2006). At The Chinese University of Hong Kong (CUHK), two LMS (WebCT and Moodle) are centrally supported. Indeed, the majority of the University’s eLearning activities are supported by these central services; apart from one faculty (Engineering), there are relatively few non-centrally-hosted course websites.

Effective record-keeping, and extraction and interpretation of eLearning logs, can reveal valuable information on how these LMS are used to facilitate teaching and learning. As noted by Sen, Dacin and Pattichis (2006), the use of logs for tracking user activities is quite common in commercial settings where customer habits and trends are traced and monitored. Reading user logs also applies in educational settings – for example, the study by Black, Dawson and Priem (2008) on how to obtain measures of ‘community’ in online courses.

In universities with centralized web-based teaching and learning systems, the logs can be monitored through inbuilt mechanisms to track and record a certain amount of information about online activities. Colace and De Santo (2003) commented that monitoring an LMS can enable detailed and useful information on the LMS’s utilization and efficacy. This information can include trend data if the logs have been collected regularly over time. Such information can provide the basis for various decisions related to the implementation and promotion of eLearning.

However, these inbuilt web-log tracking systems do not normally provide institution-level data. The weblogs are reported in an interface designed for individual teachers to get a summary of activities recorded in individual courses (Mazza & Milani, 2005) rather than for analyses of online learning activities at higher levels (e.g. department, faculty or institution). Retrieval of data in WebCT (version 3.8) is even more challenging. As it does not employ a database structure, these records or logs of activities can be extracted using the provided display logs functions which are very limited in functionality. The locations where the record information is stored are not clear because of the lack of a database structure, and so time and effort are needed for: 1) testing through trial and error for the allocation of the desired information; 2) checking whether the data are accurate; and 3) developing software to extract the information for all courses in the University. After retrieval of information, additional work is needed to standardize and automate the integration, interpretation and reporting processes of the log data so that we have a common ground to compare and contrast eLearning uses over time.

Romero, Ventura and Garcia (2008) discussed using data-mining techniques to explore the raw log data of servers in order to understand various aspects of learner activities. However, such strategies, though allowing great flexibility in the topics of study, are technically complex. As noted by Black, Dawson and Priem (2008) “server logs are plagued with a low signal-to-noise ratio: simply preparing the data for modeling can consume 80% to 95% of a project’s resources” (p. 67). A system that is more powerful than the LMS inbuilt activity log systems, and can regularly retrieve and interpret the logs into a number of fixed representations for year-by-year comparison and contrast, seems to be what we need. Zhang et al. (2007) reported a similar system called Moodog which monitor students’ activities live on the Moodle LMS. Our system looks at the issue more from an institutional point of view. We may not need to monitor student activity every moment but retrieve and analyze the logs once every semester. Also, because of our particular context, our system reads and integrates as much as possible of the logs from both of the LMS we host – WebCT and Moodle.