Using Experimental Research to Investigate Students’ Satisfaction with Online Learning

Introduction

The first empirical studies untaken to investigate factors related to student satisfaction with online learning environments were primarily qualitative in nature (Garrison & Arbaugh, 2007) Transcripts from computer mediated conferencing were coded and themes were identified to describe the variables related to student satisfaction and their online learning experiences (e.g., Gunawardena, Lowe, & Anderson,1997). However, the enormous growth in students enrolling in online courses over the past five years has created greater opportunities for researchers to conduct large scale quantitative studies that offer more generalizable outcomes about online student satisfaction and learning. Results from empirically-based quantitative research designed to investigate the influence of student satisfaction with web-based learning contexts have been regularly reported in the literature (e.g., Arbaugh, 2008; Dziuban, Moskal, Brophy, & Shea, 2007). Although, these quantitative studies are considered empirical in nature, they use correlational research methods that provide results that can not be interpreted to verify cause and effect relationships. However, outcomes from correlational studies are important for identifying relationships that can be further investigated by the use of experiments to establish causal relationships.

A renewed emphasis on the use of experimental research outcomes to support the efficacy of K-12 instructional interventions and other school-based programs emerged from funding requirements specified by the No Child Left Behind Act (NCLB) in 2001 (U.S. Congress, 2001). Institutions are now required to evaluate the effectiveness of funded programs through the use of scientifically based evidence collected from experiment research studies (Feuer, Towne, & Shavelson, 2002). For example, the U. S. Department of Education’s Institute for Education Sciences (IES) (2008) clearly specify in their “What Works Clearing House: Procedures and Standards Handbook” that grant proposals must incorporate the use of randomized experiments to gain the highest ratings toward providing “strong evidence” of intervention or program effectiveness. The National Research Council (NRC) has for some time suggested that, as in the field of medicine, randomized field trials be used to evaluate the efficacy of intervention-based programs that are funded by the No Child Left Behind Act and other federal programs (NRC, 2002). The importance of using experimental research to evaluate program outcomes has greatly influenced the methods that higher education faculty use in collaboration with K-12 schools to test the effectiveness of instructional interventions and school-based programs funded by U.S. Department of Education.

Although, experimental methods have been commonly used to conduct research in higher education settings, their use for investigating technology-supported instructional innovations has been fairly limited (Ross, Morrison & Lowther, 2005). Experimental research has advantages over descriptive or correlational studies because of the controls used to reduce the influence of extraneous variables on dependent variables used to measure program outcomes. The rigor of experimental research designs is dependent on the procedures used to minimize threats to internal validity. Campbell & Cook (1966) have defined internal validity as “the approximate validity with which we can infer that a relationship is causal (p 37). That is, can we conclude that the intervention or independent variable caused a change or had an effect on a measurement or dependent variable of interest? For example, an experimental study could be designed to demonstrate improved student satisfaction and learning during online course discussions that incorporate frequent instructor interactions when compared minimal instructor interactions. If the internal validity for this experiment is sound, then significant increases in student satisfaction and learning for online courses that incorporate frequent instructor interactions is interpreted as being “caused” by instructor interactions rather other factors such course design, student interactions or type of instructor-student exchange. Experimental designs are classified according to the varying levels of control procedures they use to ensure internal validity. Although there are variations in the manner in which experimental designs are implemented, the most common configurations are discussed in the next section.