Explore the Use of Handwriting Information and Machine Learning Techniques in Evaluating Mental Workload

Introduction

Mental workload is conceptualized as the processing costs incurred in task performance (Kramer, 1991). Understanding users’ mental workload during tasks has important implications for developing adaptive systems and optimizing interface design. For example, adaptive systems can provide adaptive aiding or adaptive task allocation based on their users’ current mental workload; the interaction design patterns or elements resulting in high mental workload can also be identified then get improved. However, mental workload added to human is different among individuals and their levels of expertise, which makes it difficult to evaluate (Kalyuga, Ayres, Chandler, & Sweller, 2003).

Current evaluation methods for mental workload can be generally divided into four main categories: subjective measures, performance measures, behavioral measures and physiological measures. These methods have been adopted with some success, but they still suffer from one or more of the following problems: post-hoc evaluation, obtrusiveness, inconvenience, insufficient granularity, and the need for additional equipment(see literature review for details). The current challenges researcher face in mental workload evaluation (Lin, Imamiya, & Mao, 2008; Vizer, Zhou, & Sears, 2009) is to develop a method that: (1) unobtrusively and continuously gathers data without extra equipment; (2) evaluates mental workload objectively, quantitatively and in real-time; and (3) is suitable for automatic evaluation and deployment in real-life scenarios.

Our research interest is to use hand handwriting features of free text to detect mental workload changes. Handwriting is a typical dynamic motor skill that requires the integration of cognitive and biomechanical systems to generate an output that is stable and reproducible. A few studies (G. Luria & Rosenblum, 2010; Gil Luria & Rosenblum, 2012; Tucha, Mecklinger, Walitza, & Lange, 2006; Werner, Rosenblum, Bar-On, Heinik, & Korczyn, 2006), mostly in the clinical and psychology field, have provided empirical evidence that measures of handwriting process can capture dis-automatization in handwriting as a result of mental stress with descriptive or linear statistical analyses and this dis-automatization may cause changes in handwriting behavior. These findings shed light on the way to detect mental workload changes. However, there remain two main challenges to develop an automated evaluation method for mental workload based on this empirical evidence. The first challenge is to determine which handwriting features are most predictive of mental workload. Although various handwriting features (e.g., temporal, spatial, pressure and dynamics measures) have been explored, there is not a clear and consistent answer to the question. We try to solve the problem using wrapper-based feature selection approach (Kohavi & John, 1997). This approach performs a combinatoric optimization to find the subset of possible features which produce the best accuracy. The second challenge is choosing a proper model to represent relationships between handwriting features and mental workload. Most previous studies (Gil Luria & Rosenblum, 2012; Ruiz, Taib, & Chen, 2011) focused on providing empirical evidence for the effects of mental workload on handwriting behavior using descriptive and linear statistical techniques. But these techniques seemed to have limitations due to the inability of capturing non-linearity of handwriting behavior changes(Gil Luria & Rosenblum, 2012). As a result, only a few handwriting features (e.g., the mean and standard deviation (SD) for segment duration) were reported to show significant differences across different three mental workload conditions and change with mental workload manipulation in a linear and synchronized way (Gil Luria & Rosenblum, 2012). Luria and Rosenblum have also emphasized that not all handwriting measures relate linearly to mental workload, and further suggested that non-linear models and extensive handwriting features should be explored to reveal the relationships between handwriting behavior and mental workload(Gil Luria & Rosenblum, 2012). By using machine learning algorithms, we built up automated classifiers to explore the linear and non-linear relationship between handwriting measures and mental workload levels. To the best of our knowledge, little effort has been made to develop an automated evaluation method for mental workload leveraging handwriting information.