International assessments such as the trends in international math and science study (TIMSS), the program for international student assessment (PISA), and the international computer and information literacy study (ICILS) have traditionally relied on paper and pencil administration. These assessments are rapidly transforming into or have been developed as computer-based tests due to advances in information and communication technologies of the past decade. These computer-based assessments will eventually make traditional paper and pencil assessments obsolete. Specifically, international and other large-scale assessments can benefit from the use of automatic item generation (AIG) and/or computer adaptive testing (CAT) to enhance and strengthen test security and validity, as well as reduce costs over the course of multiple test administrations, encourage student engagement, and efficiently measure students' abilities.
TopIntroduction
International assessments, such as the Trends in International Math and Science Study (TIMSS) and the Program for International Student Assessment (PISA), have traditionally relied on paper-and-pencil administrations. These assessments have been redesigned as computer-based tests due to advances in information and communication technologies of the past decade. Computer-based assessments will eventually make traditional paper-and-pencil assessments obsolete. The International Computer and Information Literacy Study (ICILS), a new, computer-based assessment, could also benefit from various advanced digital technologies as described.
International assessments can benefit from the use of digital assessment technologies, such as automatic item generation (AIG) and/or computer adaptive testing (CAT), to enhance and strengthen test security and validity. Additional benefits include cost reduction over the course of multiple test administrations, increased student engagement, and efficient measurement of students’ cognitive abilities.
AIG is a process of generating unique items based on a template and item modeling approach. CAT is a process in which the assessment device chooses an individual’s assessment items based on their ability level and performance on previous test items. TIMSS is in the process of converting from a traditionally paper-and-pencil test to a computer-based assessment. PISA has already converted to a digital format and will be introducing CAT in the 2018 cycle.
Purpose
The purpose of this chapter is to demonstrate the various benefits of AIG and other assessment technologies on large-scale, international assessments. As previously mentioned, these benefits include, but are not limited to, cost effectiveness, increased validity, security, global content, advancement in cognitive assessment, and accessibility. Educational assessments must continue to advance at the same rate as the average student’s technological savvy. AIG and other technologies will keep students engaged with the assessment, giving educators and international leaders a clearer image of their respective country’s student population knowledge. This chapter contains background information for international assessments, AIG, and other technologies. Developments include an a priori, theoretical cost benefit analysis and sample items.
Significance Of The Chapter
These developing technologies have several important benefits relating to educators, students, and policy makers. As previously stated, the advance in assessment technology has the potential to keep students more fully engaged throughout the process, capturing an accurate account of the students’ cognitive abilities. Unlike the traditional paper-and-pencil assessment, an AIG computer adaptive assessment more efficiently and accurately determines a student’s ability by adapting questions based on a student’s response. This timesaving benefit could greatly reduce the overall amount of classroom time dedicated to assessment, which is one of the great burdens of contemporary education.
More broadly, AIG will change the overall practice of international assessment. Presently, development of thousands of new items is required for every new assessment. Following this process is an intensive series of adaptations and translations for each country participating in the assessment. The advancement and utilization of AIG lends itself to a streamlined process of assessment development, translation, and administration. Item models, the foundation of an AIG assessment, can produce thousands of unique instances, or items, and will not need updating as frequently as paper-and-pencil items (Choi, 2017b).
It is important to note that the initial cost of training master item model writers and item model developers would likely be higher than the cost of a traditional paper-and-pencil assessment or a basic digital assessment. However, the money savings and timesaving benefits of using AIG over multiple test administrations greatly outweigh the cost of the initial investment (Choi, 2017a).