In traditional software, hardware, and program development processes, the project evaluator has been relegated to a peripheral role at the end of the process. Today’s complex interactive media projects require a different evaluation model, one that situates the project evaluator firmly at the center. From this vantage point, the evaluator subtly guides a technologically sophisticated integration of people, institutions, technologies, and cultural assumptions. Instead of the summative role—verifying that milestones and objectives have been met—the 21st-century evaluator is cast in a formative role as a problem finder. From this vantage point he or she not only confirms that project goals and objectives have been met, but also evaluates collaborative processes and facilitates collaborations among the myriad stakeholders.
TopIntroduction
Scene
It’s early December. About 20 people have assembled in Orlando for the annual two-day meeting of the Water’s Journey through the Everglades (WJE) development team. WJE is a series of interactive learning experiences that, when complete, will help bring to life the biology, history, and geography of the Florida Everglades for visitors to a South Florida science center. The project is funded by the National Science Foundation.
Attendees
The development team is large and diverse. Twenty people are in the room at any given time; they include three co-principal investigators, the project’s operations manager, an audio expert, and five representatives of the science center that will house the themed experiences (the director, educational advisors, and youth coordinators). The graduate research assistants who design and build the interactive experiences drift in and out as their class schedules allow. The group also includes three project evaluators: the author, a university faculty member charged with formative evaluation; and two professional evaluators charged with summative evaluation. Although not physically present at the table, the thousands of target audience members—middle school students, their parents, their teachers, and other potential visitors to the museum—are on everyone’s minds. Counting these virtual stakeholders, it’s standing room only in the double conference room at the Media Convergence Lab on the campus of the University of Central Florida.
Agenda
Officially, the meeting was convened so the different groups could update one another on overall progress. Although information flows back and forth across the Internet throughout the year, this annual meeting brings the far-flung teams together for a face-to-face, two-day work session. The designers showcase the latest versions of the interactive kiosks and demonstrate new technologies that might be used. The museum personnel and principal investigators revisit the rhetorical situation—the audience, purpose, and context—for which the kiosks are being designed, as well as referring to the content requirements, examining the architectural plans and space constraints, and consulting the construction timetable. The evaluators cite relevant literature and interject with findings concerning middle school students’ expressed likes and dislikes to generate ideas for future assessment protocols.
The above description, while factually accurate, doesn’t begin to capture the energetic and collaborative spirit of the meeting. In particular, it glosses over those generative moments when team members’ different backgrounds and goals collide. This chapter examines several such moments of collision in two related realms—the technological and the interpersonal—to highlight the collaborative opportunities they open up for designers of complex virtual environments.
Context: Virtual Environments For Informal Learning
Two virtual environments are featured: WJE and an earlier interactive museum prototype, Journey with Sea Creatures (JwSC), designed in 2003 and 2004. Both environments employ mixed-reality technology (sometimes called augmented reality), in which virtual reality and real life come together through the medium of the computer screen to create an immersive experience for participants. (See Kitalong, Moody, Middlebrook, & Ancheta, 2009, for a full description of JwSC’s mixed reality elements.)
The completed JwSC prototype was pilot tested for about 10 days in October 2004 at the Orlando Science Center (OSC). Just before the prototype was installed at OSC, I was invited to design and carry out an on-site assessment, which I did. At the time, I knew very little about the assessment of informal learning, but my background in usability testing provided a starting point. Well-established methods exist for assessing both product usability and formal (school-based) learning; in contrast, few tried-and-true methods exist for evaluating museum-based technologies and the informal learning they facilitate (Falk & Dierking, 2000; Hooper-Greenhill, 1994b). Undaunted, I mobilized a team of undergraduate and graduate students, and we spent several days observing how museum visitors interacted with the JwSC prototype; the results of that study are published in Kitalong, Moody, Middlebrook, & Ancheta, 2009.