Abstract
The assessment of real-world skills will often require complex and innovative types of computer-based test items to provide more authentic assessment. Having information about how students remain engaged with the various innovative elements during an assessment is useful in both assessing the utility of different types of innovative test items and assessing the validity of the inferences made about the test scores of individual students. This chapter introduces the Item Engagement Index (IEI) and the Student Engagement Index (SEI) and demonstrates their use with a variety of innovative items that were pilot tested for a nursing licensure exam. The IEI provided useful information about the amount of student effort each innovative item received, while the SEI was found useful in identifying disengaged test takers.
TopInnovative Items
One of the most exciting and promising aspects of the use of technology in assessment is the set of expanded possibilities afforded by innovative item formats. In general, innovative items are technology-based test items or tasks that include functionality and features that extend beyond traditional text-based multiple-choice items. These innovative items have also been referred to as “technology-enhanced items” (Zenisky & Sireci, 2013). Helpful discussions of the range of item types and functionalities that can be used in the assessment of real-world skills can be found in Parshall, Harmes, Davey, and Pashley (2010), Scalise (2012), Scalise and Gifford (2006), and Sireci and Zenisky (2006).
Key Terms in this Chapter
Response Time Effort (RTE): A response time-based index of the amount of effort exhibited by a test taker during a multiple-choice test event. Specifically, RTE equals the proportion of a student’s responses that were solution behaviors, as opposed to rapid guesses.
Response Time Fidelity (RTF): A response time-based index of the amount of effort received by a multiple-choice item. Specifically, RTF equals, across test events, the proportion of responses to an item that were solution behaviors as opposed to rapid guesses.
Innovative Items: Test items delivered on a computer (or some other digital device) that contain functions and features not possible with a traditional, text-based multiple-choice item.
Engagement: The degree to which a student taking a test directs attention and effort to providing correct answers to a test’s items.
Response Time: The elapsed time between when a computer-based item is displayed and when a test taker submits an answer.
Item Engagement Index (IEI): A behavior-based index of the amount of effort received by a test item, across a set of test events. IEI is based on the proportion of the expected response actions that were completed by students when responding to the item.
Student Engagement Index (SEI): A behavior-based index of the amount of effort exhibited by a student to a set of test items. SEI is based on the proportion of expected response actions that were completed by the student when responding to the items.