Abstract
This chapter introduces situated evaluation as an approach for evaluating socio-technical innovation and change. Many current evaluations simply identify the impacts of technology and deprecate alternate uses in their analysis. Situated evaluation instead calls for understanding how innovations emerge through use; this entails consideration of diverse uses, the contexts of use, and the reasons for the development of multiple realizations. The chapter presents a comparative study of different classroom uses of electronic Quill in order to demonstrate how this alternative evaluation can be conducted and to address the value of understanding and fostering diverse cultural appropriations of a socio-technical innovation.
What about the lay public as producers of technology and science? From the vernacular engineering of Latino car design to environmental analysis among rural women, groups outside the centers of scientific power persistently defy the notion that they are merely passive recipients of technological products and scientific knowledge. Rather, there are many instances in which they reinvent these products and rethink these knowledge systems, often in ways that embody critique, resistance, or outright revolt.
—Eglash, 2004, p.vii
TopQuestioning The Nature Of Standard Evaluation
Standard evaluation practice tends to emphasize either formative or summative approaches. Formative evaluation is typically done during the development or improvement of a program and is conducted iteratively. Results are often informal and lead to recommendations for change. Summative evaluation provides information on the program’s efficacy, such as improvement of student learning. In this chapter, we propose an alternative, which questions the basic assumption of “what” it is that is being evaluated.
In evaluating a new technology, researchers typically consider the innovation as a fixed object created by professional developers. They further assume that its benefits are somewhat fixed and known in advance with respect to social practice. For example, a program might be developed to help students learn a concept in science or to help a community engage in community building through better communication. Evaluation then becomes a way to improve that program or to assess its effectiveness. This is a reasonable approach, one that is fully in line with calls for reflective practice. But in its extreme form, the assumption that what the program actually is known prior to its integration into social practice becomes what Papert (1987) defines as technocentrism:
Egocentrism for Piaget does not, of course, mean “selfishness”—it means that the child has difficulty understanding anything independently of the self. Technocentrism refers to the tendency to give a similar centrality to a technical object—for example computers or Logo. This tendency shows up in questions like “What is THE effect of THE computer on cognitive development?” or “Does Logo work?” (p. 23)
The problem here is that a technocentric perspective limits the scope of the evaluation, often making it difficult to see unexpected uses of an innovation. But, as any developer knows, technical innovations often result in unplanned uses and diverse readings of the innovation. Often, the variation in use is greater than the variation in programs, so that the claim to be evaluating a particular program becomes convoluted with discussions about faithfulness of implementation or effectiveness of the program per se versus effectiveness of its introduction.
Key Terms in this Chapter
The innovation-in-use: Different ways in which the innovation is realized and thus created by diverse users. Situated evaluation, which is open to new variables and sensitive to alternate uses and interpretations, focuses on understanding innovation-in-use.
Appropriating technologies: Users actively rethinking the meaning and use of a technology and reinventing its practices within their situated, cultural contexts.
Idealization: The elements of the innovation as intended by developers.
Situated evaluation: An approach to uncovering or articulating the emergence of innovations through practice, assuming that innovations are mutually constituted by social practice and some external input.
Realization: The ways the innovation was used, modified, and re-created by users in situ.
Technocentrism: The tendency to focus on technological artifacts or mechanisms to the exclusion of social, cultural or historical perspectives.