Automating the Generation of Test Items

Introduction

With an ever increasing demand for more knowledge in our economy, organizations require methods of assessment to evaluate the skillset of their workers to produce new ideas, make new products, and provide new services. The ability to create these ideas, products, and services will be determined by the effectiveness of our educational programs. Education provide students with the knowledge and skills required to think, reason, communicate, and collaborate in a world that is shaped by knowledge services, information, and communication technologies (e.g., Binkley, Erstad, Herman, Raizen, Ripley, Miller-Ricci, & Rumble, 2012; Darling-Hammond, 2014). Educational testing has an important role to play in helping students acquire these foundational skills. Educational tests, once developed almost exclusively as a right-of-passage to satisfy demands for accountability and outcomes-based summative testing, are now expected to provide teachers and students with timely, detailed, formative feedback to directly support teaching and learning. With an increasing focus on formative learning principles being adopted to guide our educational testing practices, where assessment-related activities provide constant and specific feedback to modify and improve learning, assessments are being administered more frequently (Black & Wiliam, 1998, 2010). But when testing occurs more frequently, more test items are required. These additional test items must be created efficiently and economically while maintaining a high standard of quality. Fortunately, this requirement for frequent and timely educational testing coincides with the dramatic changes occurring in educational technology. Developers of local, national, and international educational tests are now implementing computerized tests at an extraordinary rate (Beller, 2013). Computerized testing offers many important benefits to support and promote key principles in formative assessment. Computers permit testing on-demand thereby allowing students to take the test at any time during instruction; items on computerized tests are scored immediately thereby providing students with instant feedback; computerized tests permit continuous administration thereby allowing students to have more choices about when they write their exams. In short, developments in computing technology enables the infusion of formative principles into our testing practices that would not have been previously possible.

Despite these important benefits, the advent of computerized testing has also raised formidable challenges, particularly in the area of test item development. Tests now require access to large numbers of diverse, high-quality test items to implement computerized testing as items are continuously administered to students. Hundreds of items are needed to develop the test item banks necessary for computerized testing. Unfortunately, creating test items is a time consuming and expensive process. Each individual item is written, initially, by a content specialist and, then, reviewed, edited, and revised by groups of content specialists (Gierl & Lai, 2016; Rudner, 2010). Hence, item development has been identified as one of the most important problems that must be solved before we can fully migrate to computerized testing because large numbers of high-quality, content-specific, test items are required (Haladyna & Rodriguez, 2013; Webb, Gibson, & Forkosh-Baruch, 2013).