Student modeling and cognitively diagnostic assessment are important issues that need to be addressed for the development and successful application of intelligent tutoring systems (its). Its needs the construction of complex models to represent the skills that students are using and their knowledge states, and practitioners want cognitively diagnostic information at a finer grained level. This chapter reviews our effort on modeling student’s knowledge in the ASSISTment project. Intelligent tutors have been mainly used to teach students. In the ASSISTment project, we have emphasized using the intelligent tutoring system as an assessment system that provides instructional assistance during the test. Usually it is believed that assessment get harder if students are allowed to learn during the test, as its then like try to hit a moving target. So our results are surprising that by providing tutoring to students while they are assessed we actually prove the assessment of students’ knowledge. Additionally, in this article, we present encouraging results about a fine-grained skill model with that system that is able to predict state test scores. We conclude that using intelligent tutoring systems to do assessment seems like a reasonable way of dealing with the dilemma that every minute spent testing students takes time away from instruction.
In the United States there are concerns about poor student performance on new high-stakes standards based tests required by the No Child Left Behind Act (NCLB, 2002) legislation. To address this issue, educational technologies, like intelligent tutoring systems (ITS) have been developed and proven to be useful helping students learn. For instance, the Cognitive Tutors leads to large learning gains (Koedinger et al, 1997). Recently, President Obama made a commitment to increase investment on educational software, saying that, “[W]e will devote more than 3 percent of our GDP to research and development. … Just think what this will allow us to accomplish: solar cells as cheap as paint, …, learning software as effective as a personal tutor….” 1 So, how do we build ITS that are as effective as a personal tutor? To create an effective piece of learning software, we need a good model of student learning. Assessment of an examinee’s ability is the first step of student modeling in an ITS because student state is a prerequisite for creating a pedagogical strategy. The student model provides valuable information for the tutor to help build tutoring strategy (e.g., when to interrupt and what to say when interrupt), problem sequencing (e.g., what’s the next appropriate task to give to a student so his learning gain is maximized), performance prediction (how a student will respond to a step associated with certain rules), etc.
Recently, in an interview with U.S. News & World Report (Ramírez & Clark, 2009), U.S. Secretary of Education Arne Duncan weighed in on the NCLB Act and called for continuous assessment. He said that he is concerned about over-testing, and feels that fewer, better tests would be more effective. He wants to develop better data management systems that will help teachers track individual student’s progress in real-time, so teachers and parents can assess and monitor student strengths and weaknesses. To reflect on the “continuous assessment” idea, we think, one way that ITS research distinguishes itself from other educational software development is that it is concerned with modeling the knowledge of the learner in some computationally useful and inspectable way (McCalla & Greer, 1994). The modeling phase should involve understanding learner behavior in the rich context of the environment in which learning occurs, thus, obtaining a better understanding of each student’s pre-existing, or current knowledge status and how that knowledge is changing over time.
Student models in ITS are constructive, especially cognitive models. Cognitive modeling involves a great deal of detailed protocol collection and task analysis. The models are not easy to construct and are difficult to verify. Yet, cognitive modeling is very important in an ITS as it is the basis of cognitively diagnostic assessment and teacher reporting. Wiliam (2006) describes an assessment as formative only if information about what is being assessed results in change that would otherwise not occur. Here the definition of formative assessment becomes more detailed to the utility for teachers to understand if and how students are learning. By giving teachers cognitively diagnostic data in a timely fashion, teachers can change their teaching in response to the data they collect about student understanding. The US Department of Education (2003) stated, “Research shows that teachers who use student test performance to guide and improve teaching are more effective than teachers who do not use such information” (p. 2). Therefore, if assessments are to move from assessments of learning to assessments for learning (c.f., Stiggins 2005) then we must continue to focus on the box that encompasses diagnostic modeling data and teachers as the end-users.