Assessing Collaborative Problem Solving through Computer Agent Technologies

Background

CPS is one of the two major areas that the Organisation for Economic Co-operation and Development (OECD) nominated in 2015 for major development in Programme for International Student Assessment (PISA) in addition to scientific literacy, math and reading literacy. In PISA 2015, CPS competency is defined as “the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills, and efforts to reach that solution” (OECD, 2013). An agent could be considered either a human agent or a computer agent that interacts with the student. The competency is assessed by evaluating how well the individual collaborates with agents during the problem-solving process. This includes establishing and maintaining shared understanding, taking appropriate actions to solve the problem, and establishing and maintaining group organization.

An operational definition of CPS in this chapter refers to “the capacity of an individual to effectively engage in a group process whereby two or more agents attempt to solve a problem by sharing knowledge and understanding, organizing the group work and monitoring the progress, taking actions to solve the problem, and providing constructive feedback to group members.” First, CPS requires students to be able to establish, monitor, and maintain the shared understanding throughout the problem-solving process by responding to requests for information, sending important information to agents about tasks completed, establishing or negotiating shared meanings, verifying what each other knows, and taking actions to repair insufficiencies in shared knowledge. Shared understanding can be observed as an effect, if the goal is that a group builds the common ground necessary to perform well together, or as a process by which peers perform conceptual change (Dillenbourg, 1999). An “optimal collaborative effort” is required of all of the participants in order to achieve adequate performance in a collaborative environment (Dillenbourg & Traum, 2006). Second, collaboration requires the capability to identify the type of activities that are needed to solve the problem and to follow the stages to achieve a solution. This process involves exploring and interacting with the problem situation. It includes understanding both the information originally presented in the problem and any information that is exposed during interactions with the problem. The accumulated information is selected, organized, and integrated in a way that is relevant and helpful to solving the particular problem and that is integrated with prior knowledge. Setting sub-goals, developing a plan to reach the goal state, and executing the plan that was created are also a part of this process. Overcoming the barriers of reaching the problem solution may involve not only cognition, but motivational and affective means (Funke, 2010; Mayer & Wittrock, 2006). Third, students must be able to help organize the group to solve the problem; consider the abilities and resources of group members; understand their own role and the roles of the other agents; follow the rules of engagement for their role; monitor the group organization; reflect on the success of the group organization, and help handle communication breakdowns, conflicts, and obstacles (Rosen & Rimor, 2012).