Cognitive Diagnosis of Students' Test Performance Based on Probability Inference

Introduction

Educational researchers have investigated Intelligent Tutoring Systems (ITS) as a means of providing cost effective yet personalized tuition for many years. ITS implementations have demonstrated student learning improvements comparable to the assistance of a human tutor with intermediate expertise (Graesser et al., 2000). In an ITS cognitive diagnosis is often used synonymously with 'student modeling' that builds a representation of the student’s knowledge from the evidence provided by student inputs to solve problems (Self, 1993). A widely accepted definition of cognitive diagnosis has been given by Ohlsson (1986) “cognitive diagnosis is the process of inferring a person’s cognitive state from his or her performance.” It is widely accepted that cognitive diagnosis can help in personalized tuition an ITS must know, in particular what the student knows, what misconceptions he/she has, and thus help him/her improve learning outcomes.

Cognitive diagnostic model has become an exciting new field of psychometric research. Cognitive diagnostic models aim to diagnose students’ status of a group of discretely defined skills, thereby providing them with detailed information regarding their specific strengths and weaknesses (Huebner, 2010). To date, at least fourteen distinct cognitive diagnostic models have appeared in the literature (Xu, Chang, & Douglas, 2003). The NIDA model and the Fusion model are good examples of these models (Xu, et al., 2003); they can be thought of as a Q matrix, each row of which is a list of the cognitive attributes that an examinee needs to have mastered in order to give a correct response to the item. Rule-based models are also used to depict both desired students’ knowledge and common mistakes (McLaren, 2004), thereby student performance is monitored by comparing actual student problem solving sequences against the known production rules.

Although a number of methods for cognitive diagnosis have been developed, these methods have not been placed on a more rigorous rooting until John Self placed this problem in a general theoretical framework in 1993. He considered the problem of cognitive diagnosis as an instance of Model-Based Diagnosis (MBD), where many issues in cognitive diagnosis, previously discussed informally, are mapped onto formal diagnosis in Artificial Intelligence (AI) (Self, 1993). To illustrate the problem of diagnosing students’ attempts to solve problems, a three-column subtraction is modeled a subtraction problem-solving 'circuit' and a form of heuristic search is used in Ohlsson (1986) to find plausible sequences of actions that lead to a known solution. However, as an infinite number of ineffective behavior sequences are possible, such a cognitive diagnostic model is time consuming to develop and rarely complete.

As the precise cognitive modeling of a student is likely to be intractable, adaptive remediation methods have been used successfully by applying instructional techniques appropriate to the current student context rather than requiring complex cognitive student and domain modeling. This paper considers the problem of computerized cognitive diagnosis by combining model-based reasoning with a traditional testing system in order to provide the Computerized Adaptive Testing (CAT). Like Self, we put this problem in a Bayesian network framework to better define the nature of the cognitive diagnosis problem in CAT; we recast an adaptive course testing as a diagnosis system in Bayesian network; a course description is mapped onto the system description, knowledge items onto components, and observations are students’ inputs that indicate their test performance. As distinguished from Self’s work, we seek to use model-based cognitive diagnosis with hierarchical course modeling and employ adaptive remediation based upon the student’s current state and learning difficulties encountered revealed by his/her test behavior.