Capturing Student Affect: Designing and Deploying Effective Microsurveys

Introduction

Despite decades of research on student outcomes, students continue to withdraw, fail, and dropout of higher education (in particular at two-year and other broad access institutions) at an alarming rate. According to the National Student Clearinghouse Research Center, of those students who started college in the fall of 2016, 26% of students did not persist at any institution the following year, and 38% were not retained by their initial institution. At two-year institutions, only 48% were retained (NSC Research Center, 2018). Research has shown that when students feel less motivated to attend a class or do not see the relevance of that class to their degree goals, they visit the learning environment less frequently (Simpson, 2013). Courses taken online have a 10-20% higher drop rate than face-to-face courses, with 40-80% of online students dropping depending upon the institution and course (Bawa, 2016). Therefore, online courses and degree programs are in particular need of systems that help to identify students who are at risk of dropping in time for instructors, advisors, and administrators to attempt to intervene. Traditionally, the main source of this data was end-of-course surveys, which by their very nature were too late (Hu, Lo, & Shih, 2014). Today, early warning systems provide the opportunity to identify students early enough to provide an intervention.

Early warning systems leverage student data to develop algorithms that can give instructors and administrators warnings about student behavior – in particular around failing to meet specific outcome metrics such as a course or term – in time for intervention and remediation (Howard, Meehan, & Parnell, 2018; National Forum on Education Statistics, 2018). Early warning systems can be used in any educational context including traditional face-to-face classes; hybrid, blended, or computer-enabled courses; or fully online courses. However, much interest has been directed to incorporating them into online courses, where student/instructor interaction may be more limited or where higher failure, drop, and withdrawal rates may exist (Bawa, 2016). Furthermore, online contexts typically provide more data, whether on student logins, time-on-task, behaviors, and performance, compared to other contexts. This data can be used to develop early warning systems by mining learner data, discovering trends that reveal at-risk students, and devising algorithms to prompt alerts. Many studies indicate that early warning systems can in fact enable successful interventions, if implemented with fidelity which, in turn, can drive learner outcomes in both blended and online courses (Jokhan, Sharma, & Singh, 2018; National Forum on Education Statistics, 2018; Sneyers & De Witte, 2018).

While early warning systems are an important mechanism for improving outcomes, many are based primarily on behavioral data (National Forum on Education Statistics, 2018). Surveys, however, can provide a meaningful compliment to the other datastreams typically used in early warning systems. Specifically, microsurveys—short surveys that are limited in both the number of survey items and the length of those items—create a unique opportunity to gather real time data on students’ affective states. This data can complement behavioral and cognitive data gleaned by traditional coursework or courseware platforms and, therefore, be a valuable data input into early warning systems.

Creating, deploying, interpreting, and improving microsurveys comes with a series of considerations that are unique from their longer-form counterparts. These considerations need to be addressed when conducting foundational research to guide initial conceptualizations; when developing and refining initial and iterative designs; when measuring the psychometrics properties of the survey; and when conducting formative evaluation efforts for both outcome and instrument improvement.