Rich-Media Interactive Simulations: Lessons Learned

Introduction

For over half a century, high-fidelity assessments have proven to be powerful tools for measuring a broad range of knowledge, skills, abilities, and competencies in both the workplace and educational settings. High-fidelity tools are measures that mirror or closely simulate a particular activity or group of activities. For example, these types of assessments include work samples such as driving a bus, running statistical analyses to answer a question, or taking photographs. They also include measures that do not necessarily replicate the exact activity but simulate it. If an important job activity is to analyze information about a project and then make recommendations on how to proceed, the assessment could create a fictitious project similar to one that would be completed on the job and the test taker could be asked to review the materials and make suggestions on next steps. As another example, a student may be asked to plan an approach to working with classmates to complete an assignment.

The validity of high-fidelity assessments is well documented (e.g., Tsacoumis, 2007; Arthur, Day, McNelly, & Edens, 2003; Schmidt & Hunter, 1998; Klimoski & Brickner, 1987; Moses, 1977; Bray & Grant, 1966), and they tend to be well received given their perceived relevance and face validity. However, they typically are resource and time intensive to implement since they involve live role players and because they require the evaluators to observe each test taker as he or she participates in the assessment. Given this, organizations often reserve the use of high-fidelity assessments only to evaluate candidates for their most senior or critical positions. In addition, businesses are continuing to search for more efficient ways to gather the information they need about one’s capabilities. Organizations have become increasingly reliant on technology-based solutions to evaluate students, teachers, and job candidates. In fact, the use of computers to administer traditional multiple-choice tests is now commonplace, and there is a growing trend to create and implement tests with multimedia components that use sound, video, animation, or some combination, along with text. The computer-based counterpart to live high-fidelity simulations are “rich media” assessments, which involve animation or video and allow the test taker to “interact” with the simulation and dictate how the assessment proceeds or unfolds. High-end interactive simulations have been used for training, such as pilot simulators, but in order for computer-based versions of high-fidelity simulations to gain traction in more traditional processes to assess job candidates and students, the technology must be easily accessible and the measurement challenges need to be addressed. In truth, the technological tools are available to develop and implement interactive computer-based simulations; however, the actual use of these tools for high-stakes processes, such as personnel selection, is still in its infancy.

The objective of this chapter is to describe the lessons learned from developing and implementing several rich media interactive simulations for promotional and developmental purposes in an organizational context, rather than an educational one. That said, the results generalize to any arena focused on creating accurate measures of a variety of skills and abilities. The ultimate goal is to figure out how to use the benefits offered by technology to help master the complexities associated with effectively measuring one’s competencies with enough precision and confidence to make personnel decisions—without the use of live evaluators.