Educators gauge learning with CALL applications via achievement metrics such as point scores or level advancement. Overreliance on such metrics may limit validity of learner engagement measures. This chapter used learning analytic approaches (e.g., cluster and regression analyses) to investigate user engagement with a web-based CAPT program. Cluster analysis identified four types of users with effort-based attributes best distinguishing among types. Regression analysis found that lines recorded has the strongest association with point scores. Follow-up retrospective time-series analysis of cluster members showed distinct trends in learning behavior that indicate possible goal orientations per group. These results imply that one must deconstruct and identify the aspects of engagement that are actually being measured by application metrics. Additionally, significant differences in engagement patterns exist within high and low scoring groups that are opaque to analysis across the whole sample. Finally, activity logs provide data that suggest variability in motivation types.
TopBackground
This paper is positioned at the nexus of learning analytics (LA) and CALL, and demonstrates that data mining techniques, specifically cluster analysis, can be effectively utilized by individual course instructors to understand students’ trends of learning engagement. The CALL platform under investigation is English Central1 (EC), originally marketed as a computer-assisted pronunciation training (CAPT) platform, which provides pedagogically enhanced videos sourced from the Internet and partner media providers as material for pronunciation practice, listening practice and vocabulary learning. EC’s online CAPT program provides text-dependent, contextualized speaking practice with native speaker input, some of which is authentic media and some of which is pedagogically designed material.
CAPT programs are a readily available option for language learners and instructors in the 21st century, and they are promoted by developers because of potential advantages for foreign language/ second language (L2) learning (Neri, Cucchiarini, Strik & Boves, 2002; Neri, Cucchiarini Strik, 2006). CAPT programs provide language learners an option of independent study with corrective feedback from an automated system, thereby enhancing the learning experience due to mitigation of negative affect (Neri, A., Cucchiarini, C., Strik, H., & Boves, L., 2002) such as loss of face (Chiu, Liou & Yeh, 2007) which occurs in live classrooms.
To date few researchers have evaluated EC to gauge its effectiveness in yielding English learning gains, as well as to probe levels of satisfaction, perceived effectiveness, and attitudes among users. Doubtless, effectiveness of the program is a primary concern for EFL educators and EC designers. Yet, scrutiny must also be directed towards user characteristics to probe critical factors impacting program efficacy. Of particular concern is learner engagement over the long run. Therefore, the present study departs from the paradigm of variable analytic studies of effectiveness and turns towards exploration of user characteristics to identify variables of interest mined from activity log data, which can be useful for future material evaluation studies of EC and other language learning platforms. This investigative approach is adapted from the fields of educational data mining and LA, which emphasize analysis of learner engagement with large-scale, technology-enhanced learning apparatuses such as massively open online courses (MOOCs), learning management systems (LMSs) or online platforms in order to identify action plans for course modification or learner interventions. These research aims mirror the aims of CALL material evaluation studies which seek to examine the various facets of program effectiveness. Thus, adoption of LA data exploration methodologies may identify user types, and retrospective analyses of their respective learning engagement patterns will yield insight that proves informative when designing program efficacy studies.