In recognition of the importance of collaborative and problem-solving skills, educators are realizing the need for effective and scalable learning and assessment solutions to promote the skillset in educational systems. In the settings of a comprehensive collaborative problem-solving assessment, each student should be matched with various types of group members and must apply the skills in varied contexts and tasks. One solution to these assessment demands is to use computer-based (virtual) agents to serve as the collaborators in the interactions with students. The chapter presents the premises and challenges in the use of computer agents in the assessment of collaborative problem solving and describes how human and computer agent collaborative assessments are used in an international learning and assessment project, Animalia.
TopBackground
Collaborative problem solving is recognized as a core competency for college and career readiness. Students emerging from schools into the workforce and public life will be expected to work in teams, cooperate with others, and resolve conflicts in order to solve the kinds of problems required in modern economies. They will further need to be able to use these skills flexibly with various group compositions and environments (Care, & Griffin, 2014; Griffin, Care, & McGaw, 2012; O’Neil, & Chuang, 2008; Rosen, & Rimor, 2012). Educational programs have focused to a greater extent on the advancement of learning and the assessment of collaborative problem solving as a central construct in theoretical and technological developments in educational research (National Research Council, 2011, 2013; OECD, 2013a). Collaborative skills are included within the major practices in the 2014 U.S. National Assessment of Educational Progress (NAEP) Technology and Engineering Literacy (National Assessment Governing Board, 2013). In this assessment program, students are expected to show their ability in collaborating effectively with computer-based (virtual) peers and experts and to use appropriate information and communication technologies to collaborate with others on the creation and modification of knowledge products. Similarly, the Israeli national program of adopting the educational system to the 21st century illustrates a multi-year program with the goal of leading the implementation of innovative pedagogy and assessment in schools, including collaboration, communication, and problem solving (Israel Ministry of Education, 2011). Collaborative problem solving is one of the areas that the Organisation for Economic Co-operation and Development (OECD) emphasized for major development in the Programme for International Student Assessment (PISA) in addition to scientific, math, and reading literacy for the 2015 assessment. Collaborative problem solving refers to problem solving activities that involve collaboration among a group of individuals (O’Neil, Chuang, & Baker, 2010; Zhang, 1998). In the PISA 2015 Framework (OECD, 2013b), collaborative problem solving competency is defined as “the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills, and efforts to reach that solution” (p. 6). This definition; treats the competency as conjoint dimension collaboration skills and the skills needed to solve a problem. For the PISA assessment, the focus is on individual capacities within collaborative situations. Thus, the effectiveness of collaborative problem solving depends on the ability of group members to collaborate and to prioritize the success of the group over individual successes. At the same time, this ability is still a trait in each of the individual members of the group. Development of a standardized computer-based assessment of collaborative problem solving skills, specifically for large-scale assessment programs, remains challenging. Unlike some other skills, collaborative problem solving typically requires using complex performance tasks, grounded in varied educational domains, with interaction among students. These factors can affect the level of control that can be applied to ensure accurate assessment of students.