As the acquisition of knowledge is often an expensive and time-consuming process, it is important to know whether it actually improves the student performance. The e-learning is a revolutionary paradigm that has lately been significantly evolving and it is closely related to the intelligent tutoring systems. Methodology for evaluating the educational influence of learning and teaching process, questions whether and in what amount, students learn effectively. Our contribution to this compulsive field of research is a meta-analysis of a series of experiments based on the same two-group methodology that reveals a more precise effect size of one particular e-learning system - eXtended Tutor-Expert System, a representative of web-based authoring shells for building intelligent tutoring systems.
TopIntroduction
Today, e-learning is commonly used in everyday life and it enables students to quality participate in learning and teaching process even when distance, schedule and similar circumstances makes that practically impossible. E-learning is enabled in e-learning systems which present a “personal teacher” for each student.
An evaluation offers information to make decision about using the product or not (Phillips & Gilding, 2003). One special form of e-learning system evaluation is evaluation of effectiveness designed to answer one specific research question: “What is the educational influence of an e-learning system on students?”. The main result gained through effectiveness evaluation is presented in a form of an effect size.
The most important effect size in literature comes from one of the most stimulating studies in the field of educational psychology. That is a Bloom's (Bloom, 1984) statement of the 2-sigma problem. Bloom reviewed the results of published meta-analyses and conducted one experiment, which showed that students who received individualized tutoring displayed an effect size of 2.0 compared with those who received normal instruction. In other words, students who received individualized tutoring scored an average of two standard deviations above others on achievement tests. Consequently, this gain of two standard deviations became the ideal toward which normal, group oriented instruction should strive. So, the 2-sigma problem became the quest for the ways how to get the quality of group instruction closer to individualized tutoring. Many researchers have taken up this search.
One possible solution for the 2-sigma problem is the usage of such e-learning systems which provide each student with a learning experience similar to the ideal one-to-one tutoring. The best representative of such e-learning systems are intelligent tutoring systems (ITS) that apply their learning and teaching process to each student individually. Intelligent tutoring systems are computer systems that support and improve the learning and teaching processes in a variety of knowledge domains, while respecting the individuality of the learner (Sleeman & Brown, 1982; Wenger, 1987; Burns & Capps, 1988). The major problem associated with ITS, is their expensive and time consuming development process. In order to overcome these problems we chose an approach of creating a particular ITS from flexible shells acting as program generators.
As scientists have known for a long time that only one experiment cannot solve a major issue (Hunter & Schmidt, 2004), in order to determine effect size of e-learning systems in general, as well as, different categories of e-learning systems, a different, more complex approach has to be used – a meta-analytic approach.
A meta-analysis (Glass, 1976) integrates the results of a set of experiments without losing relevant information and is used for assessing a quality of quantitative results. It is usually conducted to increase the internal and the external validity of the conclusions that can be drawn from those experimental studies, giving us far more definitive statistical conclusions. It is necessary to emphasize that meta-analysis can only be applied on researches that have empirical results, that is, only on those researches that use quantitative measures to produce quantitative results (Lipsey & Wilson, 2001).
The researchers should be familiar with meta-analysis process and should report their results in a way that is appropriate for meta-analyzing, in order to speed up a process of determining how close or distant we are from solving a 2-sigma problem with present state-of-the-art e-learning systems.
A paper providing a referent point for everyone who wants to evaluate the effectiveness of e-learning systems is Fletcher’s research (Fletcher, 2003) where he summarized some research findings for technology-based instruction: computer-based instruction 0.39 sigma (233 studies), interactive multimedia instruction 0.50 sigma (47 studies), “intelligent” tutoring systems 0.84 sigma (11 studies), recent intelligent tutoring systems 1.05 sigma (5 studies). These results suggest steady progress in learning outcomes.