Human Systems Engineering and Educational Technology

Introduction

The design and development of educational technologies is a complex, interdisciplinary endeavor. For instance, learning science research can reveal principles of learning and instruction, such as comprehension processes (Chi, 2000; McNamara, 2004) and human tutoring (Chi, Siler, Jeong, Yamauchi, & Hausmann, 2001; Graesser, Person, & Magliano, 1995; VanLehn, Siler, Murray, Yamauchi, & Bagget, 2003). Advances in computer science then enable the implementation of these principles in innovative technologies, such as intelligent tutoring systems that teach self-explanation (McNamara, Levinstein, & Boonthum, 2004) or physics (Graesser et al., 2004; VanLehn et al., 2005). The most successful and impressive educational technologies tend to emerge from the integration of multiple approaches.

One question is whether such work addresses the full scope of human users’ (e.g., students and teachers) needs. A typical approach for educational technology research is to first develop a functional system and then evaluate it in lab or classroom studies. These tests include measures of learning or other growth, and may incorporate student perception or reaction data. This information is useful for pinpointing flaws to repair, and can sometimes explain mixed results (e.g., discovering that scaffolding hints were not grade-level appropriate). However, identifying problems may be possible earlier and less expensively via iterative usability testing. For instance, a handful of students might be asked to read and explain potential hints before the scripts are ever coded into the system. If students stumble or express confusion, revisions could make the text more readable. A “failed” study could be avoided.

In this review-style chapter, we promote a third research discipline that complements learning science and computer science, but which appears underrepresented in educational technology. This discipline—human systems engineering (or “user science”)—entails research and design that takes into account human users’ broad goals, needs, capabilities, and limitations. Notably, some “human factors” have been addressed across decades of learning science and computer science. For example, researchers have examined learners’ achievement goals (Harackiewicz, Barron, Pintrich, Elliot, & Thrash, 2002), feedback needs (Shute, 2008), prior knowledge (Shapiro, 2004), and misconceptions (Chi, Roscoe, Slotta, & Chase, 2012). Similarly, artificial intelligence and learning analytics advances (Baker & Yacef, 2009; Berland, Baker, & Blikstein, 2014; Desmarais & Baker, 2012) enable technologies that adapt to learners’ knowledge and performance (Aleven, McClaren, Sewall, & Koedinger, 2009; VanLehn, 2006), strategies (Winne & Hadwin, 2013), and emotions (Calvo & D’Mello, 2010; Woolf et al., 2009).

To the extent that users of these technologies are “learners” and “teachers,” any work that supports learning and teaching can be considered “user centered.” However, there are aspects of user needs that go beyond instruction. A central assumption is that systematic and iterative human systems engineering can contribute to educational technologies that are more functional, usable, and desirable, ultimately resulting in systems that are more effective. We first introduce human systems engineering along with key principles (e.g., usability; Nielsen & Budiu, 2013) and methods (e.g., knowledge elicitation; Cooke, 1994). To make these concepts more concrete, we then discuss examples from the development of automated writing evaluation systems.