Determining the Consistency of Student Grading in a Hybrid Business Course using a LMS and Statistical Software

Determining the Consistency of Student Grading in a Hybrid Business Course using a LMS and Statistical Software

Kenneth David Strang (State University of New York, Plattsburgh, NY, USA & APPC Research, Australia)
DOI: 10.4018/jwltt.2013040103
OnDemand PDF Download:
$30.00
List Price: $37.50

Abstract

Extant literature asserted that peer assessments improved learning but only a few studies had addressed the student rating consistency issue and not had evaluated this within a Learning Management System (LMS). The researcher explored how to conduct peer assessments in Moodle LMS for a hybrid-mode business course (N=90 students) which required 270 (25-page) reports. Rater agreement statistical theory was applied to test the consistency of student peer assessments. The resulting coefficient was 0.79 and statistically significant. A pair-wise t-test and a nonparametric Mann Whitley u-test confirmed there was no significant difference between the student ratings and the scores given by the professor on the same assignments. Nonparametric Spearman correlation was very high between each student rating coefficient as compared to the score given in Moodle, which indicated the LMS was reliable in providing a grade for the quality of student peer assessment. A protocol was developed using SPSS to calculate a peer assessment inter-rater agreement consistency indicator since this was not available in Moodle.
Article Preview

Introduction

Researchers concurred that peer assessment is an effective teaching method to improve student learning (Gielen, Dochy & Onghena, 2011; Gielen, Dochy, Onghena, Struyven & Smeets, 2011; Li, 2011; KoÃ, 2011; Kritikos, Woulfe, Sukkar & Saini, 2011; Mok, 2011; Finn & Garner, 2011; Nicholson, 2011; Nulty, 2011; Wu, Davison & Sheehan, 2012). Delegating assessment work to experienced students is an efficient approach to help professors manage large classes and to expedite returning feedback (Dollisso & Koundinya, 2011; Johnson & Aragon, 2003; Kritikos, Woulfe, Sukkar & Saini, 2011; Li & Lei-na, 2012; Mok, 2011; Strang, 2013a; Thomas, Martin & Pleasants, 2011; Willey & Gardner, 2010; Zhi-Feng & Lee, 2013).

However, there were gaps in the literature addressed by this study. First, several researchers have found peer assessment was an effective pedagogy in an online course (Bayat & Naicker, 2012; Dollisso & Koundinya, 2011; Johnson & Aragon, 2003; Li & Lei-na, 2012; Lu & Law, 2012; Lu & Zhang, 2012; Wu, Hou & Hwang, 2012; Zhang & Blakey, 2012; Zhi-Feng & Lee, 2013). The problem with existing studies was that the reliability of student assessing was not being measured (Bedore & O'Sullivan, 2011; Falchikov & Goldfinch, 2000; Neus, 2011; Sadler, 2009). Other researchers questioned how to calculate the reliability of student assessment work (Fleiss, Nee & Landis, 1979; Heyman & Sailors, 2011; Speyer, Pilz, Van Der Kruis & Brunings, 2011).

Applying web-based technology such as a Learning Management System (LMS) for peer assessment is advocated especially for large online courses (Laurillard, 2007; McCabe, 2007; Li, Liu & Zhou, 2012; Shih, 2011). There are standalone software programs to automate peer assessment (e.g., iPeer; WebPA; SPARK) or subsystems (e.g., PeerMark within TurnItIn). Paradoxically, although Moodle is an open source LMS with a built-in module for peer assessment, there were no studies in the literature for conducting peer assessments and measuring their reliability. In fact, one scholar implied that faculty may refrain from using peer assessments due to the above concerns, despite their proven value as a learning tool (Bedore & O'Sullivan, 2011).

The researcher tried to address these problems here, namely, exploring how to conduct peer assessments in the Moodle LMS and then developing protocols to measure if students completed the work reliably. A secondary goal of this research was to determine if peer assessing was practical for a large online class with long essay type assignments. This study was conducted using Moodle across several sections of a hybrid business course at the State University of New York (N=90), where there were three 25-page written reports with oral presentations.

Complete Article List

Search this Journal:
Reset
Open Access Articles: Forthcoming
Volume 13: 4 Issues (2018): 1 Released, 3 Forthcoming
Volume 12: 4 Issues (2017)
Volume 11: 4 Issues (2016)
Volume 10: 4 Issues (2015)
Volume 9: 4 Issues (2014)
Volume 8: 4 Issues (2013)
Volume 7: 4 Issues (2012)
Volume 6: 4 Issues (2011)
Volume 5: 4 Issues (2010)
Volume 4: 4 Issues (2009)
Volume 3: 4 Issues (2008)
Volume 2: 4 Issues (2007)
Volume 1: 4 Issues (2006)
View Complete Journal Contents Listing