Roles and Impacts of Automatic Item Generation on Assessment Research, Practice, and Policy

Introduction

The development of science and technology will have widespread impacts throughout society, and the 'assessment' will also be a sensitive area to the impact of scientific and technological developments. Especially information communication technology, or intelligent communication technology (ICT), which has developed rapidly in the 21st century, has had a sudden and even destructive effect on various studies and the field. The development of ICT, called the fourth science and technology revolution of the 21st century, differs from the previous scientific and technological revolutions. The key difference is the “smartness” or “intelligence” based digitalized data. Therefore, studies or industries that are highly associated with computerized data are significantly affected by the development of these ICTs, and the changes are expected to be sharp in the future. The assessment, which has the most central function of collecting, processing, and analyzing data, would be one of the most rapidly changing areas under the influence of this ICT. In this paper, I will deal with “Automatic Item Generation (AIG; Embretson & Yang, 2007; Gierl &Haladyna, 2013; Irvine & Kyllonen, 2002; Choi, 2017a)” science and/or technology, which has recently received a great deal of attention in the assessment field.

Modern researchers, psychometricians, item writers, and assessment service providers increasingly find themselves facing a new paradigm where the assessment item production process is no longer manual, but rather can be a massive production systematic and automatized by technology, that is, AIG. AIG is an emerging research and technology area where an innovative assessment tool where cognitive and psychometric theories are integrated together into a comprehensive assessment development framework for the purpose of generating assessment items using state-of-the-art technology, especially in ICT environments. With AIG technique, parent model or item templates once developed by experts (i.e., item model or item template writers) are utilized in producing (or generating) clone items (also called isomorphs) via computer-based applications. The number of content areas (e.g., K-12 education, licensure and certification), and the number of applications (e.g., workbook, homework, or exam, etc.) of AIG are exploding (Choi, 2017a). As such, this new reality raises important issues in effective and scientific item development with ICTs.

Choi (2017a) argue that, although AIG has been initiated and developed on the basis of automatic generation of assessment items, we should note that an interesting and important phenomenon that automation in AIG can affect the automation of other assessment components/processes (e.g., grading, testing, or reporting), rather than merely automating item generation. What's even more interesting is that AIG's automation role can have a profound impact on its upper realm, education and society as a whole (for example, education, training, teacher evaluation, and public policy, etc.). In this paper, I will deal with the following four topics about automation of AIG. First, I will introduce, in detail, the meaning/implication of automation in AIG via ICT development in. Second, analyze how automation of AIG affects the intelligence of the technology-based assessment system which is composed of various assessment components, that is, Assessment Engineering (AE) framework. Third, I introduce how AIG automation affects/interacts with other components of an AE system. In particular, I will introduce how AIG's automation will affect education, professional training, and/or public policy. Finally, I will discuss the future directions of AIG research and development.

Automatic Item Generation

Traditionally, item writing has been done manually by item writers and content experts. For these items to be administered to test-takers, they would have to go through item review and editing, field testing, item analysis, revision, editing, formatting, etc. During this process, many items are either discarded or revised and only a small number of them survive to be put into operational use. When the cycle for test administration is long (e.g., once a year), there seems to be little need for changing this approach.